- Notifications
You must be signed in to change notification settings - Fork0
oxylabs/how-to-parse-xml-in-python
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
In this article, you'll learn how to parse XML data in Python by exploring popular Python libraries. The article will cover the basics of XML, DOM representation, built-in Python libraries for parsing XML Documents, and their differences. You'll also learn the step-by-step procedure of parsing XML files, handling invalid XML, converting to a dictionary, and saving data to a CSV file. Let's get started.
XML (Extensible Markup Language) is a popular markup language used in a wide range of applications and systems. It's a structured and hierarchical data format that allows you to store and exchange data between different platforms and applications. XML files are commonly used for data exchange, configuration files, and web services. Take a look at the below example:
<?xml version="1.0" encoding="UTF-8"?><menu> <food> <itemname="lunch"type="main">Chicken Biryani</item> <pricecurrency="USD">$12.99</price> <description> Aromatic basmati rice cooked with tender chicken pieces, spices, and herbs. </description> <calories>780</calories> <ingredients> <ingredient>Basmati rice</ingredient> <ingredient>Chicken</ingredient> <ingredient>Spices</ingredient> <ingredient>Herbs</ingredient> </ingredients> </food></menu>
This XML file starts with an XML Declaration that lets the parser know the version and encoding of the file. The root elementmenu contains information about a food item. Notice how each of the properties and attributes is structured to convey the information in a hierarchy.
The Document Object Model (DOM) provides a hierarchical representation of the document, allowing developers to interact with its elements programmatically. It provides a standardized interface to interact with web documents programmatically. It also has versatile and wide browser support enabling the creation of dynamic, interactive, and responsive web applications. It's a platform and language-neutral interface that allows you to dynamically access and update the content, structure, and style of XML and HTML documents.
<!DOCTYPE html><html><head><title>DOM Example</title></head><body><h1>Welcome to the DOM Example</h1><p>This is a sample paragraph.</p><ul><li>Item 1</li><li>Item 2</li><li>Item 3</li></ul></body></html>
The DOM serves as a powerful tool for web development. It enables efficient manipulation of the document's structure, allowing for the addition, removal, or modification of elements, attributes, and text. It provides a tree-like structure, where each element in the document is represented as a node. The root node represents the document itself, with child nodes representing elements, attributes, and text nodes. This hierarchical structure makes node traversal easy and manipulation of the document's contents a breeze.
XML parsing is a fundamental process of working with XML data. It has several key steps such as checking the syntax of the XML document, tokenizing, and building the document structure in a hierarchy. XML parsing is surprisingly difficult if you've worked with any XML document before, then you might already know this. Luckily, Python provides tons of libraries that can be utilized for parsing XML documents. All these tools have different trade-offs. Some are optimized for speed and some for memory. You can pick the necessary tool you like based on your requirements.
Almost all the Python distributions provide a standard XML library that bundles abstract interfaces for Parsing XML documents. It also enables you to supply concrete parser implementation. However, in reality, you'll hardly use a parser implementation of your own. Instead, you'll take advantage of the Python bindings of various XML parsing libraries, such asExpat. Python's standard XML library automatically binds it for you. Let's explore some of the sub-modules of Python's standard XML library.
This library enables you to parse XML documents in the DOM interface. This is a relatively old implementation of the W3c specification. All the common objects such as Document, Element, Attr, etc. are available. This module is less useful as it lacks proper documentation.
fromxml.dom.minidomimportparseStringxml_string="""<?xml version="1.0" encoding="UTF-8"?><library> <book> <title>The Great Gatsby</title> <author>F. Scott Fitzgerald</author> <year>1925</year> </book></library>"""document=parseString(xml_string)
The above code will parse thexml_string and store it in thedocument object. Now, you can use the DOM interface to access the various nodes of this XML document. Let's print the title
print(document.getElementsByTagName("title")[0].firstChild.nodeValue)
ThegetElementsByTagName method returns a list of elements from which the first element was picked. This will output the titleThe Great Gatsby
Since theminidom library follows the old w3c specification of the DOM interface, it feels old and not Pythonic. Moreover, the library source code hasn't received any updates in more than 20 years so.
The ElementTree API is a lightweight, feature-rich Interface for parsing and manipulating XML documents. The implementation is fast and elegant which attracted many third-party libraries to build on it. The documentation of this library is also better thanminidom. When it was first introduced in Python 2.5, it had a faster C implementation named cElementTree. Nowadays, you don't have to bother as the current implementation is far better in performance than the older implementation.
importxml.etree.ElementTreeasETxml_string="""<?xml version="1.0" encoding="UTF-8"?><library> <book> <title>The Great Gatsby</title> <author>F. Scott Fitzgerald</author> <year>1925</year> </book></library>"""root=ET.fromstring(xml_string)
Theroot object will contain the parsed XML document. Notice, we created an aliasET to shorten the library name in the code, this is a common convention forElementTree based Python scripts. Thefromstring method takes an XML string as an argument and returns the parsed ElementTree object. Next, you can iter over all the child nodes of the root node and print the texts, using the below code:
forchildinroot.iter():ifchild.text.strip():print(child.text)
The output will be:
| The Great GatsbyF. Scott Fitzgerald1925 |
|---|
So far, you've learned how to parse XML documents from Python string objects. Now, let's learn how to parse XML files using these libraries. Fortunately, bothminidom andElementTree provide a built-in function to parse XML files.
You can use theparse method ofminidom to read XML content from a file.
fromxml.dom.minidomimportparsedocument=parse("sample.xml")print(document.getElementsByTagName("title")[0].firstChild.nodeValue)
The ElementTree library also has aparse method to read XML files.
importxml.etree.ElementTreeasETroot=ET.parse("sample.xml")parsed_dict=dict()forchildinroot.iter():ifchild.text.strip():parsed_dict[child.tag]=child.textprint(parsed_dict)
First, we create a root XML document of thesample.xml file using theparse method. Then, we're iterating over all the child nodes and storing the data in adict object.
You can use theuntangle library to convert XML documents directly to a Python dictionary object. The code is self-explanatory.
importuntangleparsed_dict=untangle.parse("sample.xml")
The cool thing about this library is, you can pass a URL, filename, or even an XML string to theparse and it will still work.
You can use thepandas library to store the data in a CSV file.
df=pd.DataFrame(parsed_dict)df.to_csv("parsed_xml_data.csv", index=False)
If you run the above code, it'll initialize a pandas Data Framedf with theparsed_data. And, save the data in a CSV file namedparsed_xml_data.csv
Unfortunately, Python's standard XML libraries don't validate the structure of the XML file. So, it can't parse and extract data from invalid XML files containing invalid characters or, broken tags or elements. For example, take a look at the below XML file:
<?xml version="1.0" encoding="UTF-8"?><root> <person> <name> John Doe</name> <message>This is a message& an invalid XML example.</message> </person></root>
At first glance, it might seem like a valid XML but, if you try to parse this XML document you will get an error. Notice the& symbol inside the message element, this symbol is an invalid XML character thus all the standard XML libraries will fail to parse this XML file. Now let's try to parse this invalid XML using a couple of methods.
For simple XML documents like the above, you can preprocess the XML document as a string to remove the invalid elements or symbols from the XML Document before passing it to the parser.
fromxml.dom.minidomimportparseStringinvalid_xml="""<?xml version="1.0" encoding="UTF-8"?><root><person><name> John Doe</name><message>This is a message & an invalid XML example.</message></person></root>"""# preprocessingvalid_xml=invalid_xml.replace("&","&")parsed_data=parseString(valid_xml)
As you can see in the preprocessing step, the replace method will replace the& symbol of theinvalid_xml with&. Now, if you run this script it'll parse the XML document without any errors. There are many ways to preprocess the XML documents sometimes you can take advantage of Python'sre module to use RegEx and replace complex text as well. However, if the XML document is too large then this method will become cumbersome.
If the invalid XML document is too large, it'll be hard to preprocess. In such cases, instead of using a stricter XML parser that doesn't handle broken XML documents, you can use a more forgiving parsing library such as Beautiful Soup. It can take care of invalid XML literals, missing or broken tags, and elements automatically.
frombs4importBeautifulSoupinvalid_xml="""<?xml version="1.0" encoding="UTF-8"?><root><person><name> John Doe</name><message>This is a message & an invalid XML example.</message></person></root>"""soup=BeautifulSoup(invalid_xml,features="lxml-xml")print(soup.prettify())
You should keep in mind that, Beautiful Soup is slower than the other XML parsing libraries such as ElementTree or lxml. If performance is an issue then you might've to use preprocessing or other robust libraries instead.
This tutorial has equipped you with a thorough understanding of XML parsing in Python. You have explored various parsing models, delved into the standard library and third-party parsers, and learned about declarative parsing and safe XML parsing practices. With this knowledge, you are now empowered to choose the most suitable XML parser for your needs and handle XML data effectively and efficiently in your Python projects. Last but not least, whenever dealing with XML documents you must pay extra attention as some XML documents can be malicious. Python's standard XML libraries aren't secured enough to protect against such maliciously crafted XML files. To learn more about the vulnerabilities and risks read the official Python XML librarydocumentation.
Python is one of the most popular general-purpose programming languages. It has sets of powerful libraries that make Python great for file parsing. Since Python is an interpreted programming language, it doesn't provide C, and C++ like performance out of the box. However, you can combine Cython or other faster Python implementations and binaries to improve performance.
Python's built-in XML library provides multiple sub-modules and libraries with different syntax and styles giving you the flexibility to choose the one that fits best with your preference. If you're comfortable with DOM manipulation then you might find thedom module orminidom easier to learn. On the other hand, theetree module is also beginner friendly. From the third-party libraries,lxml is popular for XML parsing in the Python community. And, if you're familiar with Beautiful Soup, you can use theBeautifulSoup library as well. There is another library nameduntangle which enables users to convert XML documents into Pythondict objects with a single line of code. However, this library is not in active development. So, it's not recommended for production use.
lxml is arguably the fastest XML parsing library with support for Xpath, XSLT & XML Schema standards. it's the Python binding for the C librarieslibxml2 andlibxslt. This library is not only fast but also packs a familiar interface as ElementTree API with the broadest spectrum of functionalities. It's also compatible with Python's ElementTree. Many libraries such as Beautiful Soup can also utilize thelxml parser under the hood to get a performance boost.
About
Follow this in-depth technical tutorial to learn how to parse XML data in Python, what libraries you should use, how to handle invalid XML, and more.
Topics
Resources
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Releases
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Contributors2
Uh oh!
There was an error while loading.Please reload this page.