Repository files navigation

https://mechanicalsoup.readthedocs.io/

A Python library for automating interaction with websites.MechanicalSoup automatically stores and sends cookies, followsredirects, and can follow links and submit forms. It doesn't doJavaScript.

MechanicalSoup was created byM Hickford, who was a fond user of theMechanize library.Unfortunately, Mechanize wasincompatible with Python 3 until 2019 and its developmentstalled for several years. MechanicalSoup provides a similar API, built on PythongiantsRequests (forHTTP sessions) andBeautifulSoup (for documentnavigation). Since 2017 it is a project actively maintained by a smallteam including@hemberger and@moy.

PyPy3 is also supported (and tested against).

Download and install the latest released version fromPyPI:

pip install MechanicalSoup

Download and install the development version fromGitHub:

pip install git+https://github.com/MechanicalSoup/MechanicalSoup

Installing from source (installs the version in the current working directory):

python setup.py install

(In all cases, add--user to theinstall command toinstall in the current user's home directory.)

The full documentation is available onhttps://mechanicalsoup.readthedocs.io/. You may want to jump directly totheautomatically generated APIdocumentation.

Fromexamples/expl_qwant.py, code to get the results froma Qwant search:

"""Example usage of MechanicalSoup to get the results from the Qwantsearch engine."""importreimportmechanicalsoupimporthtmlimporturllib.parse# Connect to Qwantbrowser=mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup')browser.open("https://lite.qwant.com/")# Fill-in the search formbrowser.select_form('#search-form')browser["q"]="MechanicalSoup"browser.submit_selected()# Display the resultsforlinkinbrowser.page.select('.result a'):# Qwant shows redirection links, not the actual URL, so extract# the actual URL from the redirect link:href=link.attrs['href']m=re.match(r"^/redirect/[^/]*/(.*)$",href)ifm:href=urllib.parse.unquote(m.group(1))print(link.text,'->',href)

More examples are available inexamples/.

For an example with a more complex form (checkboxes, radio buttons andtextareas), readtests/test_browser.pyandtests/test_form.py.

Instructions for building, testing and contributing to MechanicalSoup:seeCONTRIBUTING.rst.

Read theFAQ.