- Jon Dufresne

- Ville Skyttä

- Jonathan Vanasco

- Tom Most

* Cease supporting DATrie under PyPy.

* **Remove``PullDOM`` support, as this hasn't ever been properly

* **Remove PullDOM support, as this hasn't ever been properly

completely unused by anyone.**

to clarify their status as public.**

* **Get rid of the sanitizer package. Merge sanitizer.sanitize into the

code changes.**

html5lib Package

:mod:`html5lib` Package

..automodule::html5lib.__init__

..automodule::html5lib

:members: __version__

:mod:`constants` Module

:mod:`serializer` Module

..automodule::html5lib.serializer

treebuilders Package

:mod:`~html5lib.treeadapters` Package

..automodule::html5lib.treeadapters

..automodule::html5lib.treeadapters.genshi

..automodule::html5lib.treeadapters.sax

:mod:`base` Module

..automodule::html5lib.treewalkers.base

:mod:`etree_lxml` Module

..automodule::html5lib.treewalkers.etree_lxml

:mod:`genshi` Module

..automodule::html5lib.treewalkers.genshi

html5lib consists of a number of components, which are responsible for

handling its features.

Parsing uses a *tree builder* to generate a *tree*, the in-memory representation of the document.

Several tree representations are supported, as are translations to other formats via *tree adapters*.

The tree may be translated to a token stream with a *tree walker*, from which:class:`~html5lib.serializer.HTMLSerializer` produces a stream of bytes.

The token stream may also be transformed by use of *filters* to accomplish tasks like sanitization.

Tree builders

The parser reads HTML by tokenizing the content and building a tree that

the user can later access. There are three main types of trees that

html5lib can build:

the user can later access. html5lib can build three types of trees:

* ``etree`` - this is the default; builds a tree based on``xml.etree``,

* ``etree`` - this is the default; builds a tree based on:mod:`xml.etree`,

which can be found in the standard library. Whenever possible, the

accelerated ``ElementTree`` implementation (i.e.

``xml.etree.cElementTree`` on Python 2.x) is used.

* ``dom`` - builds a tree based on``xml.dom.minidom``.

* ``dom`` - builds a tree based on:mod:`xml.dom.minidom`.

* ``lxml.etree`` - uses lxml's implementation of the ``ElementTree``

* ``lxml`` - usesthe:mod:`lxml.etree` implementation of the ``ElementTree``

API. The performance gains are relatively small compared to using the

accelerated ``ElementTree`` module.

When instantiating a parser object, you have to pass a tree builder

class in the ``tree`` keyword attribute:

To get a builder class by name, use the:func:`~html5lib.treebuilders.getTreeBuilder` function.

..code-block::python

To get a builder class by name, use the ``getTreeBuilder`` function:

When instantiating a:class:`~html5lib.html5parser.HTMLParser` object, you must pass a tree builder class via the ``tree`` keyword attribute:

..code-block::python

The implementation of builders can be found in `html5lib/treebuilders/

Tree walkers

Once a tree is ready, you can work on it either manually, or using

a tree walker, which provides a streaming view of the tree. html5lib

provides walkers for all three supported types of trees (``etree``,

``dom`` and ``lxml``).

In addition to manipulating a tree directly, you can use a tree walker to generate a streaming view of it.

html5lib provides walkers for ``etree``, ``dom``, and ``lxml`` trees, as well as ``genshi`` `markup streams<https://genshi.edgewall.org/wiki/Documentation/streams.html>`_.

The implementation of walkers can be found in `html5lib/treewalkers/

<https://github.com/html5lib/html5lib-python/tree/master/html5lib/treewalkers>`_.

Walkers make consuming HTML easier. html5lib uses them to provide you

with has a couple of handy tools.

html5lib provides:class:`~html5lib.serializer.HTMLSerializer` for generating a stream of bytes from a token stream, and several filters which manipulate the stream.

HTMLSerializer

You can customize the serializer behaviour in a variety of ways, consult

the:class:`~html5lib.serializer.htmlserializer.HTMLSerializer`

documentation.

You can customize the serializer behaviour in a variety of ways. Consult

the:class:`~html5lib.serializer.HTMLSerializer` documentation.

Filters

You can alter the stream content withfilters provided by html5lib:

html5lib provides severalfilters:

*:class:`alphabeticalattributes.Filter

<html5lib.filters.alphabeticalattributes.Filter>` sorts attributes on

the document

*:class:`lint.Filter <html5lib.filters.lint.Filter>` raises

``LintError`` exceptions on invalid tag and attribute names, invalid

:exc:`AssertionError` exceptions on invalid tag and attribute names, invalid

PCDATA, etc.

*:class:`optionaltags.Filter <html5lib.filters.optionaltags.Filter>`

removes tags from the stream which are not necessary to produce valid

removes tags from thetokenstream which are not necessary to produce valid

HTML

*:class:`sanitizer.Filter <html5lib.filters.sanitizer.Filter>` removes

*:class:`whitespace.Filter <html5lib.filters.whitespace.Filter>`

collapses all whitespace characters to single spaces unless they're in

``<pre/>`` or ``textarea`` tags.

``<pre/>`` or ``<textarea/>`` tags.

To use a filter, simply wrap it around a stream:

To use a filter, simply wrap it around atokenstream:

..code-block::python

Tree adapters

Used to translate one type of treetoanother. More documentation

Tree adapters can be usedtotranslate between tree formats.

*:func:`html5lib.treeadapters.genshi.to_genshi()` generates a `Genshi markup stream<https://genshi.edgewall.org/wiki/Documentation/streams.html>`_.

*:func:`html5lib.treeadapters.sax.to_sax()` calls a SAX handler based on the tree.

Encoding discovery

* The encoding may be explicitly specified by passing the name of the

encoding as the encoding parameter to the

:meth:`~html5lib.html5parser.HTMLParser.parse` method on

``HTMLParser`` objects.

:class:`~html5lib.html5parser.HTMLParser` objects.

* If no encoding is specified, the parser will attempt to detect the

encoding from a ``<meta>`` element in the first 512 bytes of the

document (this is only a partial implementation of the current HTML

5specification).

specification).

* If no encoding can be found and the chardet library is available, an

* If no encoding can be found and the:mod:`chardet` library is available, an

attempt will be made to sniff the encoding from the byte pattern.

* If all else fails, the default encoding will be used. This is usually

`Windows-1252<http://en.wikipedia.org/wiki/Windows-1252>`_, which is

a common fallback used by Web browsers.

Tokenizers

The part of the parser responsible for translating a raw input stream

into meaningful tokens is the tokenizer. Currently html5lib provides

two.

To set up a tokenizer, simply pass it when instantiating

a:class:`~html5lib.html5parser.HTMLParser`:

..code-block::python

HTMLTokenizer

This is the default tokenizer, the heart of html5lib. The implementation

can be found in `html5lib/tokenizer.py

<https://github.com/html5lib/html5lib-python/blob/master/html5lib/tokenizer.py>`_.

HTMLSanitizer

This is a tokenizer that removes unsafe markup and CSS styles from the

input. Elements that are known to be safe are passed through and the

rest is converted to visible text. The default configuration of the

sanitizer follows the `WHATWG Sanitization Rules

<http://wiki.whatwg.org/wiki/Sanitization_rules>`_.

The implementation can be found in `html5lib/sanitizer.py

<https://github.com/html5lib/html5lib-python/blob/master/html5lib/sanitizer.py>`_.

from __future__importabsolute_import,division,unicode_literals

"getTreeWalker","serialize"]

base: webencodings

py26-base: ordereddict

optional: -r{toxinidir}/requirements-optional.txt

doc: Sphinx

commands =

{envbindir}/py.test {posargs}

{toxinidir}/flake8-run.sh

changedir = doc

commands = sphinx-build -b html . _build