Repository files navigation

Findoriginal and updated publication dates of any web page.It is often not possible to do it using just the URL or the server response.

On the command-line or with Python, all the steps needed from web pagedownload to HTML parsing, scraping, and text analysis are included.

The package is used in production on millions of documents and integrated intothousands of projects.

>>>fromhtmldateimportfind_date>>>find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')'2016-12-23'

$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html'2016-12-23'

• Flexible input: URLs, HTML files, or HTML trees can be used as input(including batch processing).
• Customizable output: Any date format (defaults toISO 8601YMD).
• Detection of both original and updated dates.
• Multilingual.
• Compatible with all recent versions of Python.

Htmldate operates by sifting through HTML markup and if necessary textelements. It features the following heuristics:

1. Markup in header: Common patterns are used to identify relevantelements (e.g.link andmeta elements) includingOpen Graphprotocol attributes.
2. HTML code: The whole document is searched for structural markerslikeabbr ortime elements and a series of attributes (e.g.postmetadata).
3. Bare HTML content: Heuristics are run on text and markup:
   • Infast mode the HTML page is cleaned and precise patterns aretargeted.
   • Inextensive mode all potential dates are collected and adisambiguation algorithm determines the best one.

Finally, the output is validated and converted to the chosen format.

1000 web pages containing identifiable dates (as of 2023-11-13 on Python 3.10)

For the complete results and explanations seeevaluationpage.

Htmldate is tested on Linux, macOS and Windows systems, it is compatiblewith Python 3.8 upwards. It can notably be installed withpip (pip3where applicable) from the PyPI package repository:

• pip install htmldate
• (optionally)pip install htmldate[speed]

The last version to support Python 3.6 and 3.7 ishtmldate==1.8.1.

For more details on installation, Python & CLI usage,please refer tothe documentation:htmldate.readthedocs.io

This package is distributed under theApache 2.0license.

Versions prior to v1.8.0 are under GPLv3+ license.

Initially launched to create text databases for research purposesat the Berlin-Brandenburg Academy of Sciences (DWDS and ZDL units),this project continues to be maintained but its future developmentdepends on community support.

If you value this software or depend on it for your product, considersponsoring it and contributing to its codebase. Your supportwill help maintain and enhance this package.Visit theContributing pagefor more information.

Reach out via the software repository or thecontact pagefor inquiries, collaborations, or feedback.

@article{barbaresi-2020-htmldate,  title = {{htmldate: A Python package to extract publication dates from web pages}},  author ="Barbaresi, Adrien",  journal ="Journal of Open Source Software",  volume = 5,  number = 51,  pages = 2439,  url = {https://doi.org/10.21105/joss.02439},  publisher = {The Open Journal},  year = 2020,}

• Barbaresi, A. "htmldate: A Python package to extract publicationdates from web pages",Journal of Open Source Software, 5(51), 2439, 2020. DOI:10.21105/joss.02439
• Barbaresi, A. "Generic Web Content Extraction with Open-SourceSoftware",Proceedings of KONVENS 2019, Kaleidoscope Abstracts, 2019.
• Barbaresi, A. "Efficient construction of metadata-enhanced webcorpora",Proceedings of the10th Web as Corpus Workshop(WAC-X), 2016.

Kudos to the following software libraries:

• lxml,dateparser
• A few patterns are derived from thepython-goose,metascraper,newspaper andarticleDateExtractorlibraries. This module extends their coverage and robustnesssignificantly.