Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up

Fast and robust date extraction from web pages, with Python or on the command-line

License

NotificationsYou must be signed in to change notification settings

adbar/htmldate

Repository files navigation

Python packagePython versionsDocumentation StatusCode CoverageDownloadsJOSS article reference DOI: 10.21105/joss.02439


Htmldate Logo


Findoriginal and updated publication dates of any web page.It is often not possible to do it using just the URL or the server response.

On the command-line or with Python, all the steps needed from web pagedownload to HTML parsing, scraping, and text analysis are included.

The package is used in production on millions of documents and integrated intothousands of projects.

In a nutshell


Demo as GIF image


With Python

>>>fromhtmldateimportfind_date>>>find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')'2016-12-23'

On the command-line

$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html'2016-12-23'

Features

  • Flexible input: URLs, HTML files, or HTML trees can be used as input(including batch processing).
  • Customizable output: Any date format (defaults toISO 8601YMD).
  • Detection of both original and updated dates.
  • Multilingual.
  • Compatible with all recent versions of Python.

How it works

Htmldate operates by sifting through HTML markup and if necessary textelements. It features the following heuristics:

  1. Markup in header: Common patterns are used to identify relevantelements (e.g.link andmeta elements) includingOpen Graphprotocol attributes.
  2. HTML code: The whole document is searched for structural markerslikeabbr ortime elements and a series of attributes (e.g.postmetadata).
  3. Bare HTML content: Heuristics are run on text and markup:
    • Infast mode the HTML page is cleaned and precise patterns aretargeted.
    • Inextensive mode all potential dates are collected and adisambiguation algorithm determines the best one.

Finally, the output is validated and converted to the chosen format.

Performance

1000 web pages containing identifiable dates (as of 2023-11-13 on Python 3.10)

Python PackagePrecisionRecallAccuracyF-ScoreTime
articleDateExtractor 0.200.8030.7340.6220.7675x
date_guesser 2.1.40.7810.6000.5140.67918x
goose3 3.1.170.8690.5320.4930.66015x
htmldate[all] 1.6.0 (fast)0.8830.9240.8230.9031x
htmldate[all] 1.6.0 (extensive)0.8700.9930.8650.9281.7x
newspaper3k 0.2.80.7690.6670.5560.71515x
news-please 1.5.350.8010.7680.6450.78434x

For the complete results and explanations seeevaluationpage.

Installation

Htmldate is tested on Linux, macOS and Windows systems, it is compatiblewith Python 3.8 upwards. It can notably be installed withpip (pip3where applicable) from the PyPI package repository:

  • pip install htmldate
  • (optionally)pip install htmldate[speed]

The last version to support Python 3.6 and 3.7 ishtmldate==1.8.1.

Documentation

For more details on installation, Python & CLI usage,please refer tothe documentation:htmldate.readthedocs.io

License

This package is distributed under theApache 2.0license.

Versions prior to v1.8.0 are under GPLv3+ license.

Context and contributions

Initially launched to create text databases for research purposesat the Berlin-Brandenburg Academy of Sciences (DWDS and ZDL units),this project continues to be maintained but its future developmentdepends on community support.

If you value this software or depend on it for your product, considersponsoring it and contributing to its codebase. Your supportwill help maintain and enhance this package.Visit theContributing pagefor more information.

Reach out via the software repository or thecontact pagefor inquiries, collaborations, or feedback.

JOSS article reference DOI: 10.21105/joss.02439Zenodo archive DOI: 10.5281/zenodo.3459599

@article{barbaresi-2020-htmldate,  title = {{htmldate: A Python package to extract publication dates from web pages}},  author ="Barbaresi, Adrien",  journal ="Journal of Open Source Software",  volume = 5,  number = 51,  pages = 2439,  url = {https://doi.org/10.21105/joss.02439},  publisher = {The Open Journal},  year = 2020,}

Acknowledgements

Kudos to the following software libraries:


[8]ページ先頭

©2009-2025 Movatter.jp