* If all else fails, the default encoding will be used. This is usually

`Windows-1252<http://en.wikipedia.org/wiki/Windows-1252>`_, which is

a common fallback used by Web browsers.

Tokenizers

The part of the parser responsible for translating a raw input stream

into meaningful tokens is the tokenizer. Currently html5lib provides

two.

To set up a tokenizer, simply pass it when instantiating

a:class:`~html5lib.html5parser.HTMLParser`:

..code-block::python

HTMLTokenizer

This is the default tokenizer, the heart of html5lib. The implementation

can be found in `html5lib/tokenizer.py

<https://github.com/html5lib/html5lib-python/blob/master/html5lib/tokenizer.py>`_.

HTMLSanitizer

This is a tokenizer that removes unsafe markup and CSS styles from the

input. Elements that are known to be safe are passed through and the

rest is converted to visible text. The default configuration of the

sanitizer follows the `WHATWG Sanitization Rules

<http://wiki.whatwg.org/wiki/Sanitization_rules>`_.

The implementation can be found in `html5lib/sanitizer.py

<https://github.com/html5lib/html5lib-python/blob/master/html5lib/sanitizer.py>`_.