- Developing with lxml
The lxml.etree Tutorial
| Author: | Stefan Behnel |
|---|
This is a tutorial on XML processing withlxml.etree. It brieflyoverviews the main concepts of theElementTree API, and some simpleenhancements that make your life as a programmer easier.
For a complete reference of the API, see thegenerated APIdocumentation.
Contents
A common way to importlxml.etree is as follows:
>>>fromlxmlimportetree
If your code only uses the ElementTree API and does not rely on anyfunctionality that is specific tolxml.etree, you can also use the followingimport chain as a fall-back to ElementTree in the Python standard library:
try:fromlxmlimportetreeprint("running with lxml.etree")exceptImportError:importxml.etree.ElementTreeasetreeprint("running with Python's xml.etree.ElementTree")
To aid in writing portable code, this tutorial makes it clear in the exampleswhich part of the presented API is an extension oflxml.etree over theoriginalElementTree API.
The Element class
AnElement is the main container object for the ElementTree API. Most ofthe XML tree functionality is accessed through this class. Elements areeasily created through theElement factory:
>>>root=etree.Element("root")
The XML tag name of elements is accessed through thetag property:
>>>print(root.tag)root
Elements are organised in an XML tree structure. To create child elements andadd them to a parent element, you can use theappend() method:
>>>root.append(etree.Element("child1"))
However, this is so common that there is a shorter and much more efficient wayto do this: theSubElement factory. It accepts the same arguments as theElement factory, but additionally requires the parent as first argument:
>>>child2=etree.SubElement(root,"child2")>>>child3=etree.SubElement(root,"child3")
To see that this is really XML, you can serialise the tree you have created:
>>>etree.tostring(root)b'<root><child1/><child2/><child3/></root>'
We'll create a little helper function to pretty-print the XML for us:
>>> def prettyprint(element, **kwargs):... xml = etree.tostring(element, pretty_print=True, **kwargs)... print(xml.decode(), end='')>>> prettyprint(root)<root> <child1/> <child2/> <child3/></root>
Elements are lists
To make the access to these subelements easy and straight forward,elements mimic the behaviour of normal Python lists as closely aspossible:
>>>child=root[0]>>>print(child.tag)child1>>>print(len(root))3>>>root.index(root[1])# lxml.etree only!1>>>children=list(root)>>>forchildinroot:...print(child.tag)child1child2child3>>>root.insert(0,etree.Element("child0"))>>>start=root[:1]>>>end=root[-1:]>>>print(start[0].tag)child0>>>print(end[0].tag)child3
Prior to ElementTree 1.3 and lxml 2.0, you could also check the truth value ofan Element to see if it has children, i.e. if the list of children is empty:
ifroot:# this no longer works!print("The root element has children")
This is no longer supported as people tend to expect that a "something"evaluates to True and expect Elements to be "something", may they havechildren or not. So, many users find it surprising that any Elementwould evaluate to False in an if-statement like the above. Instead,uselen(element), which is both more explicit and less error prone.
>>>print(etree.iselement(root))# test if it's some kind of ElementTrue>>>iflen(root):# test if it has children...print("The root element has children")The root element has children
There is another important case where the behaviour of Elements in lxml(in 2.0 and later) deviates from that of lists and from that of theoriginal ElementTree (prior to version 1.3 or Python 2.7/3.2):
>>>forchildinroot:...print(child.tag)child0child1child2child3>>>root[0]=root[-1]# this moves the element in lxml.etree!>>>forchildinroot:...print(child.tag)child3child1child2
In this example, the last element ismoved to a different position,instead of being copied, i.e. it is automatically removed from itsprevious position when it is put in a different place. In lists,objects can appear in multiple positions at the same time, and theabove assignment would just copy the item reference into the firstposition, so that both contain the exact same item:
>>>l=[0,1,2,3]>>>l[0]=l[-1]>>>l[3, 1, 2, 3]
Note that in the original ElementTree, a single Element object can sitin any number of places in any number of trees, which allows for the samecopy operation as with lists. The obvious drawback is that modificationsto such an Element will apply to all places where it appears in a tree,which may or may not be intended.
The upside of this difference is that an Element inlxml.etree alwayshas exactly one parent, which can be queried through thegetparent()method. This is not supported in the original ElementTree.
>>>rootisroot[0].getparent()# lxml.etree only!True
If you want tocopy an element to a different position inlxml.etree,consider creating an independentdeep copy using thecopy modulefrom Python's standard library:
>>>fromcopyimportdeepcopy>>>element=etree.Element("neu")>>>element.append(deepcopy(root[1]))>>>print(element[0].tag)child1>>>print([c.tagforcinroot])['child3', 'child1', 'child2']
The siblings (or neighbours) of an element are accessed as next and previouselements:
>>>root[0]isroot[1].getprevious()# lxml.etree only!True>>>root[1]isroot[0].getnext()# lxml.etree only!True
Elements carry attributes as a dict
XML elements support attributes. You can create them directly in the Elementfactory:
>>>root=etree.Element("root",interesting="totally")>>>etree.tostring(root)b'<root interesting="totally"/>'
Attributes are just unordered name-value pairs, so a very convenient wayof dealing with them is through the dictionary-like interface of Elements:
>>>print(root.get("interesting"))totally>>>print(root.get("hello"))None>>>root.set("hello","Huhu")>>>print(root.get("hello"))Huhu>>>etree.tostring(root)b'<root interesting="totally" hello="Huhu"/>'>>>sorted(root.keys())['hello', 'interesting']>>>forname,valueinsorted(root.items()):...print('%s =%r'%(name,value))hello = 'Huhu'interesting = 'totally'
For the cases where you want to do item lookup or have other reasons forgetting a 'real' dictionary-like object, e.g. for passing it around,you can use theattrib property:
>>>attributes=root.attrib>>>print(attributes["interesting"])totally>>>print(attributes.get("no-such-attribute"))None>>>attributes["hello"]="Guten Tag">>>print(attributes["hello"])Guten Tag>>>print(root.get("hello"))Guten Tag
Note thatattrib is a dict-like object backed by the Element itself.This means that any changes to the Element are reflected inattriband vice versa. It also means that the XML tree stays alive in memoryas long as theattrib of one of its Elements is in use. To get anindependent snapshot of the attributes that does not depend on the XMLtree, copy it into a dict:
>>>d=dict(root.attrib)>>>sorted(d.items())[('hello', 'Guten Tag'), ('interesting', 'totally')]
Elements contain text
Elements can contain text:
>>>root=etree.Element("root")>>>root.text="TEXT">>>print(root.text)TEXT>>>etree.tostring(root)b'<root>TEXT</root>'
In many XML documents (data-centric documents), this is the only place wheretext can be found. It is encapsulated by a leaf tag at the very bottom of thetree hierarchy.
However, if XML is used for tagged text documents such as (X)HTML, text canalso appear between different elements, right in the middle of the tree:
<html><body>Hello<br/>World</body></html>
Here, the<br/> tag is surrounded by text. This is often referred to asdocument-style ormixed-content XML. Elements support this through theirtail property. It contains the text that directly follows the element, upto the next element in the XML tree:
>>>html=etree.Element("html")>>>body=etree.SubElement(html,"body")>>>body.text="TEXT">>>etree.tostring(html)b'<html><body>TEXT</body></html>'>>>br=etree.SubElement(body,"br")>>>etree.tostring(html)b'<html><body>TEXT<br/></body></html>'>>>br.tail="TAIL">>>etree.tostring(html)b'<html><body>TEXT<br/>TAIL</body></html>'
The two properties.text and.tail are enough to represent anytext content in an XML document. This way, the ElementTree API doesnot require anyspecial text nodes in addition to the Elementclass, that tend to get in the way fairly often (as you might knowfrom classicDOM APIs).
However, there are cases where the tail text also gets in the way.For example, when you serialise an Element from within the tree, youdo not always want its tail text in the result (although you wouldstill want the tail text of its children). For this purpose, thetostring() function accepts the keyword argumentwith_tail:
>>>etree.tostring(br)b'<br/>TAIL'>>>etree.tostring(br,with_tail=False)# lxml.etree only!b'<br/>'
If you want to readonly the text, i.e. without any intermediatetags, you have to recursively concatenate alltext andtailattributes in the correct order. Again, thetostring() functioncomes to the rescue, this time using themethod keyword:
>>>etree.tostring(html,method="text")b'TEXTTAIL'
Using XPath to find text
Another way to extract the text content of a tree isXPath, whichalso allows you to extract the separate text chunks into a list:
>>>print(html.xpath("string()"))# lxml.etree only!TEXTTAIL>>>print(html.xpath("//text()"))# lxml.etree only!['TEXT', 'TAIL']
If you want to use this more often, you can wrap it in a function:
>>>build_text_list=etree.XPath("//text()")# lxml.etree only!>>>print(build_text_list(html))['TEXT', 'TAIL']
Note that a string result returned by XPath is a special 'smart'object that knows about its origins. You can ask it where it camefrom through itsgetparent() method, just as you would withElements:
>>>texts=build_text_list(html)>>>print(texts[0])TEXT>>>parent=texts[0].getparent()>>>print(parent.tag)body>>>print(texts[1])TAIL>>>print(texts[1].getparent().tag)br
You can also find out if it's normal text content or tail text:
>>>print(texts[0].is_text)True>>>print(texts[1].is_text)False>>>print(texts[1].is_tail)True
While this works for the results of thetext() function, lxml willnot tell you the origin of a string value that was constructed by theXPath functionsstring() orconcat():
>>>stringify=etree.XPath("string()")>>>print(stringify(html))TEXTTAIL>>>print(stringify(html).getparent())None
Tree iteration
For problems like the above, where you want to recursively traverse the treeand do something with its elements, tree iteration is a very convenientsolution. Elements provide a tree iterator for this purpose. It yieldselements indocument order, i.e. in the order their tags would appear if youserialised the tree to XML:
>>>root=etree.Element("root")>>>etree.SubElement(root,"child").text="Child 1">>>etree.SubElement(root,"child").text="Child 2">>>etree.SubElement(root,"another").text="Child 3">>>prettyprint(root)<root> <child>Child 1</child> <child>Child 2</child> <another>Child 3</another></root>>>>forelementinroot.iter():...print(f"{element.tag} -{element.text}")root - Nonechild - Child 1child - Child 2another - Child 3
If you know you are only interested in a single tag, you can pass its name toiter() to have it filter for you. Starting with lxml 3.0, you can alsopass more than one tag to intercept on multiple tags during iteration.
>>>forelementinroot.iter("child"):...print(f"{element.tag} -{element.text}")child - Child 1child - Child 2>>>forelementinroot.iter("another","child"):...print(f"{element.tag} -{element.text}")child - Child 1child - Child 2another - Child 3
By default, iteration yields all nodes in the tree, includingProcessingInstructions, Comments and Entity instances. If you want tomake sure only Element objects are returned, you can pass theElement factory as tag parameter:
>>>root.append(etree.Entity("#234"))>>>root.append(etree.Comment("some comment"))>>>forelementinroot.iter():...ifisinstance(element.tag,str):...print(f"{element.tag} -{element.text}")...else:...print(f"SPECIAL:{element} -{element.text}")root - Nonechild - Child 1child - Child 2another - Child 3SPECIAL: ê - êSPECIAL: <!--some comment--> - some comment>>>forelementinroot.iter(tag=etree.Element):...print(f"{element.tag} -{element.text}")root - Nonechild - Child 1child - Child 2another - Child 3>>>forelementinroot.iter(tag=etree.Entity):...print(element.text)ê
Note that passing a wildcard"*" tag name will also yield allElement nodes (and only elements).
Inlxml.etree, elements providefurther iterators for all directions in thetree: children, parents (or rather ancestors) and siblings.
Serialisation
Serialisation commonly uses thetostring() function that returns astring, or theElementTree.write() method that writes to a file, afile-like object, or a URL (via FTP PUT or HTTP POST). Both calls acceptthe same keyword arguments likepretty_print for formatted outputorencoding to select a specific output encoding other than plainASCII:
>>>root=etree.XML('<root><a><b/></a></root>')>>>etree.tostring(root)b'<root><a><b/></a></root>'>>>xml_string=etree.tostring(root,xml_declaration=True)>>>print(xml_string.decode(),end='')<?xml version='1.0' encoding='ASCII'?><root><a><b/></a></root>>>>latin1_bytesstring=etree.tostring(root,encoding='iso8859-1')>>>print(latin1_bytesstring.decode('iso8859-1'),end='')<?xml version='1.0' encoding='iso8859-1'?><root><a><b/></a></root>>>>print(etree.tostring(root,pretty_print=True).decode(),end='')<root> <a> <b/> </a></root>
Note that pretty printing appends a newline at the end.We therefore use theend='' option here to prevent theprint()function from adding another line break.
For more fine-grained control over the pretty-printing, you can addwhitespace indentation to the tree before serialising it, using theindent() function (added in lxml 4.5):
>>>root=etree.XML('<root><a><b/>\n</a></root>')>>>print(etree.tostring(root).decode())<root><a><b/></a></root>>>>etree.indent(root)>>>print(etree.tostring(root).decode())<root> <a> <b/> </a></root>>>>root.text'\n '>>>root[0].text'\n '>>>etree.indent(root,space=" ")>>>print(etree.tostring(root).decode())<root> <a> <b/> </a></root>>>>etree.indent(root,space="\t")>>>etree.tostring(root)b'<root>\n\t<a>\n\t\t<b/>\n\t</a>\n</root>'
In lxml 2.0 and later, as well as inxml.etree, the serialisationfunctions can do more than XML serialisation. You can serialise toHTML or extract the text content by passing themethod keyword:
>>>root=etree.XML(...'<html><head/><body><p>Hello<br/>World</p></body></html>')>>>etree.tostring(root)# default: method = 'xml'b'<html><head/><body><p>Hello<br/>World</p></body></html>'>>>etree.tostring(root,method='xml')# same as aboveb'<html><head/><body><p>Hello<br/>World</p></body></html>'>>>etree.tostring(root,method='html')b'<html><head></head><body><p>Hello<br>World</p></body></html>'>>>prettyprint(root,method='html')<html><head></head><body><p>Hello<br>World</p></body></html>>>>etree.tostring(root,method='text')b'HelloWorld'
As for XML serialisation, the default encoding for plain textserialisation is ASCII:
>>>br=next(root.iter('br'))# get first result of iteration>>>br.tail='Wörld'>>>etree.tostring(root,method='text')# doctest: +ELLIPSISTraceback (most recent call last):...UnicodeEncodeError:'ascii' codec can't encode character '\xf6' ...>>>etree.tostring(root,method='text',encoding="UTF-8")b'HelloW\xc3\xb6rld'
Here, serialising to a Python text string instead of a byte stringmight become handy. Just pass the name'unicode' as encoding:
>>>etree.tostring(root,encoding='unicode',method='text')'HelloWörld'>>>etree.tostring(root,encoding='unicode')'<html><head/><body><p>Hello<br/>Wörld</p></body></html>'
The W3C has a good article about the Unicode character set and character encodings<http://www.w3.org/International/tutorials/tutorial-char-enc/>`_.
The ElementTree class
AnElementTree is mainly a document wrapper around a tree with aroot node. It provides a couple of methods for serialisation andgeneral document handling.
>>>root=etree.XML('''\...<?xml version="1.0"?>...<!DOCTYPE root SYSTEM "test" [ <!ENTITY tasty "parsnips"> ]>...<root>... <a>&tasty;</a>...</root>...''')>>>tree=etree.ElementTree(root)>>>print(tree.docinfo.xml_version)1.0>>>print(tree.docinfo.doctype)<!DOCTYPE root SYSTEM "test">>>>tree.docinfo.public_id='-//W3C//DTD XHTML 1.0 Transitional//EN'>>>tree.docinfo.system_url='file://local.dtd'>>>print(tree.docinfo.doctype)<!DOCTYPE root PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "file://local.dtd">
AnElementTree is also what you get back when you call theparse() function to parse files or file-like objects (see theparsing section below).
One of the important differences is that theElementTree classserialises as a complete document, as opposed to a singleElement.This includes top-level processing instructions and comments, as wellas a DOCTYPE and other DTD content in the document:
>>>prettyprint(tree)# lxml 1.3.4 and later<!DOCTYPE root PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "file://local.dtd" [<!ENTITY tasty "parsnips">]><root> <a>parsnips</a></root>
In the original xml.etree.ElementTree implementation and in lxmlup to 1.3.3, the output looks the same as when serialising onlythe root Element:
>>>prettyprint(tree.getroot())<root> <a>parsnips</a></root>
This serialisation behaviour has changed in lxml 1.3.4. Before,the tree was serialised without DTD content, which made lxmllose DTD information in an input-output cycle.
Parsing from strings and files
lxml.etree supports parsing XML in a number of ways and from allimportant sources, namely strings, files, URLs (http/ftp) andfile-like objects. The main parse functions arefromstring() andparse(), both called with the source as first argument. Bydefault, they use the standard parser, but you can always pass adifferent parser as second argument.
The fromstring() function
Thefromstring() function is the easiest way to parse a string:
>>>some_xml_data="<root>data</root>">>>root=etree.fromstring(some_xml_data)>>>print(root.tag)root>>>etree.tostring(root)b'<root>data</root>'
The XML() function
TheXML() function behaves like thefromstring() function, but iscommonly used to write XML literals right into the source:
>>>root=etree.XML("<root>data</root>")>>>print(root.tag)root>>>etree.tostring(root)b'<root>data</root>'
There is also a corresponding functionHTML() for HTML literals.
>>>root=etree.HTML("<p>data</p>")>>>etree.tostring(root)b'<html><body><p>data</p></body></html>'
The parse() function
Theparse() function is used to parse from files and file-like objects.
As an example of such a file-like object, the following code uses theBytesIO class for reading from a string instead of an external file.However, in real life, you would obviously avoid doing this and use thestring parsing functions likefromstring() above.
>>>fromioimportBytesIO>>>some_file_or_file_like_object=BytesIO(b"<root>data</root>")>>>tree=etree.parse(some_file_or_file_like_object)>>>etree.tostring(tree)b'<root>data</root>'
Note thatparse() returns an ElementTree object, not an Element object asthe string parser functions:
>>>root=tree.getroot()>>>print(root.tag)root>>>etree.tostring(root)b'<root>data</root>'
The reasoning behind this difference is thatparse() returns acomplete document from a file, while the string parsing functions arecommonly used to parse XML fragments.
Theparse() function supports any of the following sources:
- an open file object (make sure to open it in binary mode)
- a file-like object that has a.read(byte_count) method returninga byte string on each call
- a filename string
- an HTTP or FTP URL string
Note that passing a filename or URL is usually faster than passing anopen file or file-like object. However, the HTTP/FTP client in libxml2is rather simple, so things like HTTP authentication require a dedicatedURL request library, e.g.urllib2 orrequests. These librariesusually provide a file-like object for the result that you can parsefrom while the response is streaming in.
Parser objects
By default,lxml.etree uses a standard parser with a default setup. Ifyou want to configure the parser, you can create a new instance:
>>>parser=etree.XMLParser(remove_blank_text=True)# lxml.etree only!
This creates a parser that removes empty text between tags while parsing,which can reduce the size of the tree and avoid dangling tail text if you knowthat whitespace-only content is not meaningful for your data. An example:
>>>root=etree.XML("<root> <a/> <b> </b> </root>",parser)>>>etree.tostring(root)b'<root><a/><b> </b></root>'
Note that the whitespace content inside the<b> tag was not removed, ascontent at leaf elements tends to be data content (even if blank). You caneasily remove it in an additional step by traversing the tree:
>>>forelementinroot.iter("*"):...ifelement.textisnotNoneandnotelement.text.strip():...element.text=None>>>etree.tostring(root)b'<root><a/><b/></root>'
Seehelp(etree.XMLParser) to find out about the available parser options.
Incremental parsing
lxml.etree provides two ways for incremental step-by-step parsing. One isthrough file-like objects, where it calls theread() method repeatedly.This is best used where the data arrives from a source likeurllib or anyother file-like object that can provide data on request. Note that the parserwill block and wait until data becomes available in this case:
>>>classDataSource:...data=[b"<roo",b"t><",b"a/",b"><",b"/root>"]...defread(self,requested_size):...try:...returnself.data.pop(0)...exceptIndexError:...returnb''>>>tree=etree.parse(DataSource())>>>etree.tostring(tree)b'<root><a/></root>'
The second way is through a feed parser interface, given by thefeed(data)andclose() methods:
>>>parser=etree.XMLParser()>>>parser.feed("<roo")>>>parser.feed("t><")>>>parser.feed("a/")>>>parser.feed("><")>>>parser.feed("/root>")>>>root=parser.close()>>>etree.tostring(root)b'<root><a/></root>'
Here, you can interrupt the parsing process at any time and continue it lateron with another call to thefeed() method. This comes in handy if youwant to avoid blocking calls to the parser, e.g. in frameworks like Twisted,or whenever data comes in slowly or in chunks and you want to do other thingswhile waiting for the next chunk.
After calling theclose() method (or when an exception was raisedby the parser), you can reuse the parser by calling itsfeed()method again:
>>>parser.feed("<root/>")>>>root=parser.close()>>>etree.tostring(root)b'<root/>'
Event-driven parsing
Sometimes, all you need from a document is a small fraction somewhere deepinside the tree, so parsing the whole tree into memory, traversing it anddropping it can be too much overhead.lxml.etree supports this use casewith two event-driven parser interfaces, one that generates parser eventswhile building the tree (iterparse), and one that does not build the treeat all, and instead calls feedback methods on a target object in a SAX-likefashion.
Here is a simpleiterparse() example:
>>>some_file_like=BytesIO(b"<root><a>data</a></root>")>>>forevent,elementinetree.iterparse(some_file_like):...print(f"{event},{element.tag:>4},{element.text}")end, a, dataend, root, None
By default,iterparse() only generates an event when it is done parsing anelement, but you can control this through theevents keyword argument:
>>>some_file_like=BytesIO(b"<root><a>data</a></root>")>>>forevent,elementinetree.iterparse(some_file_like,...events=("start","end")):...print(f"{event:>5},{element.tag:>4},{element.text}")start, root, Nonestart, a, data end, a, data end, root, None
Note that the text, tail, and children of an Element are not necessarily presentyet when receiving thestart event. Only theend event guaranteesthat the Element has been parsed completely.
It also allows you to.clear() or modify the content of an Element tosave memory. So if you parse a large tree and you want to keep memoryusage small, you should clean up parts of the tree that you no longerneed. Thekeep_tail=True argument to.clear() makes sure that(tail) text content that follows the current element will not be touched.It is highly discouraged to modify any content that the parser may nothave completely read through yet.
>>>some_file_like=BytesIO(...b"<root><a><b>data</b></a><a><b/></a></root>")>>>forevent,elementinetree.iterparse(some_file_like):...ifelement.tag=='b':...print(element.text)...elifelement.tag=='a':...print("** cleaning up the subtree")...element.clear(keep_tail=True)data** cleaning up the subtreeNone** cleaning up the subtree
A very important use case foriterparse() is parsing largegenerated XML files, e.g. database dumps. Most often, these XMLformats only have one main data item element that hangs directly belowthe root node and that is repeated thousands of times. In this case,it is best practice to letlxml.etree do the tree building and only tointercept on exactly this one Element, using the normal tree APIfor data extraction.
>>>xml_file=BytesIO(b'''\...<root>... <a><b>ABC</b><c>abc</c></a>... <a><b>MORE DATA</b><c>more data</c></a>... <a><b>XYZ</b><c>xyz</c></a>...</root>''')>>>for_,elementinetree.iterparse(xml_file,tag='a'):...print('%s --%s'%(element.findtext('b'),element[1].text))...element.clear(keep_tail=True)ABC -- abcMORE DATA -- more dataXYZ -- xyz
If, for some reason, building the tree is not desired at all, thetarget parser interface oflxml.etree can be used. It createsSAX-like events by calling the methods of a target object. Byimplementing some or all of these methods, you can control whichevents are generated:
>>>classParserTarget:...events=[]...close_count=0...defstart(self,tag,attrib):...self.events.append(("start",tag,attrib))...defclose(self):...events,self.events=self.events,[]...self.close_count+=1...returnevents>>>parser_target=ParserTarget()>>>parser=etree.XMLParser(target=parser_target)>>>events=etree.fromstring('<root test="true"/>',parser)>>>print(parser_target.close_count)1>>>foreventinevents:...print(f'event:{event[0]} - tag:{event[1]}')...forattr,valueinevent[2].items():...print(f' *{attr} ={value}')event: start - tag: root * test = true
You can reuse the parser and its target as often as you like, so youshould take care that the.close() method really resets thetarget to a usable state (also in the case of an error!).
>>>events=etree.fromstring('<root test="true"/>',parser)>>>print(parser_target.close_count)2>>>events=etree.fromstring('<root test="true"/>',parser)>>>print(parser_target.close_count)3>>>events=etree.fromstring('<root test="true"/>',parser)>>>print(parser_target.close_count)4>>>foreventinevents:...print(f'event:{event[0]} - tag:{event[1]}')...forattr,valueinevent[2].items():...print(f' *{attr} ={value}')event: start - tag: root * test = true
Namespaces
The ElementTree API avoidsnamespace prefixeswherever possible and deploys the real namespace (the URI) instead:
>>>xhtml=etree.Element("{http://www.w3.org/1999/xhtml}html")>>>body=etree.SubElement(xhtml,"{http://www.w3.org/1999/xhtml}body")>>>body.text="Hello World">>>prettyprint(xhtml)<html:html xmlns:html="http://www.w3.org/1999/xhtml"> <html:body>Hello World</html:body></html:html>
The notation that ElementTree uses was originally brought up byJames Clark. It has the majoradvantage of providing a universally qualified name for a tag, regardlessof any prefixes that may or may not have been used or defined in a document.By moving the indirection of prefixes out of the way, it makes namespaceaware code much clearer and easier to get right.
As you can see from the example, prefixes only become important whenyou serialise the result. However, the above code looks somewhatverbose due to the lengthy namespace names. And retyping or copying astring over and over again is error prone. It is therefore commonpractice to store a namespace URI in a global variable. To adapt thenamespace prefixes for serialisation, you can also pass a mapping tothe Element factory function, e.g. to define the default namespace:
>>>XHTML_NAMESPACE="http://www.w3.org/1999/xhtml">>>XHTML="{%s}"%XHTML_NAMESPACE>>>NSMAP={None:XHTML_NAMESPACE}# the default namespace (no prefix)>>>xhtml=etree.Element(XHTML+"html",nsmap=NSMAP)# lxml only!>>>body=etree.SubElement(xhtml,XHTML+"body")>>>body.text="Hello World">>>prettyprint(xhtml)<html xmlns="http://www.w3.org/1999/xhtml"> <body>Hello World</body></html>
You can also use theQName helper class to build or split qualifiedtag names:
>>>tag=etree.QName('http://www.w3.org/1999/xhtml','html')>>>print(tag.localname)html>>>print(tag.namespace)http://www.w3.org/1999/xhtml>>>print(tag.text){http://www.w3.org/1999/xhtml}html>>>tag=etree.QName('{http://www.w3.org/1999/xhtml}html')>>>print(tag.localname)html>>>print(tag.namespace)http://www.w3.org/1999/xhtml>>>root=etree.Element('{http://www.w3.org/1999/xhtml}html')>>>tag=etree.QName(root)>>>print(tag.localname)html>>>tag=etree.QName(root,'script')>>>print(tag.text){http://www.w3.org/1999/xhtml}script>>>tag=etree.QName('{http://www.w3.org/1999/xhtml}html','script')>>>print(tag.text){http://www.w3.org/1999/xhtml}script
lxml.etree allows you to look up the current namespaces defined for anode through the.nsmap property:
>>>xhtml.nsmap{None: 'http://www.w3.org/1999/xhtml'}
Note, however, that this includes all prefixes known in the context ofan Element, not only those that it defines itself.
>>>root=etree.Element('root',nsmap={'a':'http://a.b/c'})>>>child=etree.SubElement(root,'child',...nsmap={'b':'http://b.c/d'})>>>len(root.nsmap)1>>>len(child.nsmap)2>>>child.nsmap['a']'http://a.b/c'>>>child.nsmap['b']'http://b.c/d'
Therefore, modifying the returned dict cannot have any meaningfulimpact on the Element. Any changes to it are ignored.
Namespaces on attributes work alike, but as of version 2.3,lxml.etreewill ensure that the attribute uses a prefixed namespacedeclaration. This is because unprefixed attribute names are notconsidered being in a namespace by the XML namespace specification(section 6.2), so they may end up losing their namespace on aserialise-parse roundtrip, even if they appear in a namespacedelement.
>>>body.set(XHTML+"bgcolor","#CCFFAA")>>>prettyprint(xhtml)<html xmlns="http://www.w3.org/1999/xhtml"> <body xmlns:html="http://www.w3.org/1999/xhtml" html:bgcolor="#CCFFAA">Hello World</body></html>>>>print(body.get("bgcolor"))None>>>body.get(XHTML+"bgcolor")'#CCFFAA'
You can also use XPath with fully qualified names:
>>>find_xhtml_body=etree.ETXPath(# lxml only !..."//{%s}body"%XHTML_NAMESPACE)>>>results=find_xhtml_body(xhtml)>>>print(results[0].tag){http://www.w3.org/1999/xhtml}body
For convenience, you can use"*" wildcards in all iterators oflxml.etree,both for tag names and namespaces:
>>>forelinxhtml.iter('*'):print(el.tag)# any element{http://www.w3.org/1999/xhtml}html{http://www.w3.org/1999/xhtml}body>>>forelinxhtml.iter('{http://www.w3.org/1999/xhtml}*'):print(el.tag){http://www.w3.org/1999/xhtml}html{http://www.w3.org/1999/xhtml}body>>>forelinxhtml.iter('{*}body'):print(el.tag){http://www.w3.org/1999/xhtml}body
To look for elements that do not have a namespace, either use theplain tag name or provide the empty namespace explicitly:
>>>[el.tagforelinxhtml.iter('{http://www.w3.org/1999/xhtml}body')]['{http://www.w3.org/1999/xhtml}body']>>>[el.tagforelinxhtml.iter('body')][]>>>[el.tagforelinxhtml.iter('{}body')][]>>>[el.tagforelinxhtml.iter('{}*')][]
The E-factory
TheE-factory provides a simple and compact syntax for generating XML andHTML:
>>>fromlxml.builderimportE>>>defCLASS(*args):# class is a reserved word in Python...return{"class":' '.join(args)}>>>html=page=(...E.html(# create an Element called "html"...E.head(...E.title("This is a sample document")...),...E.body(...E.h1("Hello!",CLASS("title")),...E.p("This is a paragraph with ",E.b("bold")," text in it!"),...E.p("This is another paragraph, with a","\n ",...E.a("link",href="http://www.python.org"),"."),...E.p("Here are some reserved characters: <spam&egg>."),...etree.XML("<p>And finally an embedded XHTML fragment.</p>"),...)...)...)>>>prettyprint(page)<html> <head> <title>This is a sample document</title> </head> <body> <h1>Hello!</h1> <p>This is a paragraph with <b>bold</b> text in it!</p> <p>This is another paragraph, with a <a href="http://www.python.org">link</a>.</p> <p>Here are some reserved characters: <spam&egg>.</p> <p>And finally an embedded XHTML fragment.</p> </body></html>
Element creation based on attribute access makes it easy to build up asimple vocabulary for an XML language:
>>>fromlxml.builderimportElementMaker# lxml only !>>>E=ElementMaker(namespace="http://my.de/fault/namespace",...nsmap={'p':"http://my.de/fault/namespace"})>>>DOC=E.doc>>>TITLE=E.title>>>SECTION=E.section>>>PAR=E.par>>>my_doc=DOC(...TITLE("The dog and the hog"),...SECTION(...TITLE("The dog"),...PAR("Once upon a time, ..."),...PAR("And then ...")...),...SECTION(...TITLE("The hog"),...PAR("Sooner or later ...")...)...)>>>prettyprint(my_doc)<p:doc xmlns:p="http://my.de/fault/namespace"> <p:title>The dog and the hog</p:title> <p:section> <p:title>The dog</p:title> <p:par>Once upon a time, ...</p:par> <p:par>And then ...</p:par> </p:section> <p:section> <p:title>The hog</p:title> <p:par>Sooner or later ...</p:par> </p:section></p:doc>
One such example is the modulelxml.html.builder, which provides avocabulary for HTML.
When dealing with multiple namespaces, it is good practice to defineone ElementMaker for each namespace URI. Again, note how the aboveexample predefines the tag builders in named constants. That makes iteasy to put all tag declarations of a namespace into one Python moduleand to import/use the tag name constants from there. This avoidspitfalls like typos or accidentally missing namespaces.
ElementPath
The ElementTree library comes with a simple XPath-like path languagecalledElementPath. The main difference is that you can use the{namespace}tag notation in ElementPath expressions. However,advanced features like value comparison and functions are notavailable.
In addition to afull XPath implementation,lxml.etree supports theElementPath language in the same way ElementTree does, even using(almost) the same implementation. The API provides four methods herethat you can find on Elements and ElementTrees:
- iterfind() iterates over all Elements that match the pathexpression
- findall() returns a list of matching Elements
- find() efficiently returns only the first match
- findtext() returns the.text content of the first match
Here are some examples:
>>>root=etree.XML("<root><a x='123'>aText<b/><c/><b/></a></root>")
Find a child of an Element:
>>>print(root.find("b"))None>>>print(root.find("a").tag)a
Find an Element anywhere in the tree:
>>>print(root.find(".//b").tag)b>>>[b.tagforbinroot.iterfind(".//b")]['b', 'b']
Find Elements with a certain attribute:
>>>print(root.findall(".//a[@x]")[0].tag)a>>>print(root.findall(".//a[@y]"))[]
In lxml 3.4, there is a new helper to generate a structural ElementPathexpression for an Element:
>>>tree=etree.ElementTree(root)>>>a=root[0]>>>print(tree.getelementpath(a[0]))a/b[1]>>>print(tree.getelementpath(a[1]))a/c>>>print(tree.getelementpath(a[2]))a/b[2]>>>tree.find(tree.getelementpath(a[2]))==a[2]True
As long as the tree is not modified, this path expression represents anidentifier for a given element that can be used tofind() it in the sametree later. Compared to XPath, ElementPath expressions have the advantageof being self-contained even for documents that use namespaces.
The.iter() method is a special case that only finds specific tagsin the tree by their name, not based on a path. That means that thefollowing commands are equivalent in the success case:
>>>print(root.find(".//b").tag)b>>>print(next(root.iterfind(".//b")).tag)b>>>print(next(root.iter("b")).tag)b
Note that the.find() method simply returns None if no match is found,whereas the other two examples would raise aStopIteration exception.