Python Packaging Is Good Now

Okay folks. Time’s up. It’s too late to say that Python’s packaging ecosystemterrible any more. I’m calling it.

Python packaging is not bad any more. If you’re a developer, and you’re tryingto create or consume Python libraries, it can be a tractable, evenpleasantexperience.

I need to say this, because for a long time, Python’s packaging toolchain was …problematic. It isn’t any more, but a lot of people still seem to think thatit is, so it’s time to set the record straight.

If you’re not familiar with the history it went something like this:

The Dawn

Python first shipped in an era when adding a dependency meant a veritableOdyssey into cyberspace. First, you’d wait until nobody in your whole familywas using the phone line. Then you’d dial your ISP. Once you’d finishedfighting your SLIP or PPP client, you’d ask a netnews group if anyone knew of agood gopher site to find a library that could solve your problem. Once youwere done with that task, you’d sign off the Internet for the night, and waitabout 48 hours too see if anyone responded. If you were lucky enough to get areply, you’d set up a download at the end of your night’s web-surfing.

pip search it wasn’t.

For the time, Python’s approach to dependency-handling was incrediblyforward-looking. Theimport statement, and the pluggable module importsystem, made it easy to get dependencies from wherever made sense.

In Python 2.0¹, Distutils was introduced. This let Python developersdescribe their collections of modules abstractly, and added tool support toproducing redistributable collections of modules and packages. Again, this wastremendously forward-looking, if somewhat primitive; there was very little tocompare it to at the time.

Fast forwarding to 2004;setuptools was created to address some of theincreasingly-common tasks that open source software maintainers were facingwith distributing their modules over the internet. In 2005, it addedeasy_install, in order to provide a tool to automate resolving dependenciesand downloading them into the right locations.

The Dark Age

Unfortunately, in addition to providing basic utilities for expressingdependencies,setuptools also dragged in a tremendous amount of complexity.Its author felt thatimport should do something slightly different than whatit does, so installingsetuptools changed it. The main difference betweennormalimport andsetuptoolsimport was that it facilitated havingmultiple different versions of the same library in the same program at the sametime. It turns out that that’s adumb idea, but infairness, it wasn’t entirely clear at the time, and it is certainly useful (andnecessary!) to be able to have multiple versions of a library installed onto acomputer at the same time.

In addition to these idiosyncratic departures from standard Python semantics,setuptools suffered from being unmaintained. It became a critical part ofthe Python ecosystem at the same time as the author was moving on toother projects entirely outside of programming.No-one could agree on who the new maintainers should be for a long period oftime. The project wasforked, and manyoperating systems’ packaging toolchains calcified around a buggy, ancientversion.

From 2008 to 2012 or so, Python packaging was a total mess. It was painful touse. It was not clear which libraries or tools to use, which ones were worthinvesting in or learning. Doing things the simple way was too tedious, anddoing things the automated way involved lots of poorly-documented workaroundsand inscrutable failure modes.

This is to say nothing of the fact that there werecritical security flaws in various partsof this toolchain. There was no practical way to package and upload Pythonpackages in such a way that users didn’t need a full compiler toolchain fortheir platform.

To make matters worse for the popular perception of Python’s packagingprowess², at this same time, newer languages and environments were getting alot of buzz, ones that had packagingbuilt in at the very beginning and had amuch better binary distribution story. Theseenvironments learned lessons from the screw-ups of Python and Perl, and reallygot a lot of things right from the start.

Finally, the Python Package Index, the site which hosts all the open sourcepackages uploaded by the Python community, was basically a proof-of-conceptthat went live way too early, had almost no operational resources, and wasoffline all the dang time.

Things were looking pretty bad for Python.

Intermission

Here is where we get to the point of this post - this is where popular opinionabout Python packaging is stuck. Outdated information from this periodabounds. Blog posts complaining about problems score high in web searches.Those who used Python during this time, but have now moved on to some otherlanguage, frequently scoff and dismiss Python as impossible to package, itspackaging ecosystem as broken, PyPI as down all the time, and so on. Worst ofall, bad advice for workarounds which are no longer necessary are still easy tofind, which causes users to pre-emptively break their environments where theyreally don’t need to.

From The Ashes

In the midst of all this brokenness, there were some who were heroically,quietly, slowly fixing the mess, one gnarly bug-report at a time.pip wasstarted, and its various maintainers fixed much ofeasy_install’sovercomplexity and many of its flaws.Donald Stufftstepped in both on Pip and PyPI and improved the availability of the systems itdepended upon, as well as somepretty serious vulnerabilitiesin the tool itself.Daniel Holth wrotea PEP for thewheel format,which allows for binary redistribution of libraries. In other words, it letsauthors of packages which need a C compiler to build give their users a way tonot have one.

In 2013,setuptools anddistribute un-forked,providing a path forward for operating system vendors to start updating theirinstallations and allowing users to use something modern.

Python Core started distributingthe ensurepip modulealong with both Python 2.7 and 3.3, allowing any user with a recent Pythoninstalled to quickly bootstrap into a sensible Python development environmentwith a one-liner.

A New Renaissance

I won’t give you a full run-down of the state of the packaging art.There’s already a website for that. I will,however, give you aprécis of how much easier it is to get startednowadays. Today, if you want to get a sensible, up-to-date python developmentenvironment, without administrative privileges, all you have to do is:

123

$python-mensurepip--user$python-mpipinstall--user--upgradepip$python-mpipinstall--user--upgradevirtualenv

Then, for each project you want to do, make a new virtualenv:

123

$python-mvirtualenvlets-go$../lets-go/bin/activate(lets-go)$_

From here on out, now the world is your oyster; you canpip install to yourheart’s content, andyou probably won’t even need to compile any C formost packages. These instructions don’t depend on Python version, either: aslong as it’s up-to-date, the same steps work on Python 2, Python 3, PyPy andeven Jython. In fact, often theensurepip step isn’t even necessary sincepip comes preinstalled. Running it if it’s unnecessary is harmless, even!

Other, more advanced packaging operations are much simpler than they used tobe, too.

• Need a C compiler? OS vendors have been working with the open source community to make this easier across the board:

  12345

  $aptinstallbuild-essentialpython-dev# ubuntu$xcode-select--install# macOS$dnfinstall@development-toolspython-devel# fedoraC:\>REMwindowsC:\>starthttps://www.microsoft.com/en-us/download/details.aspx?id=44266

Okay that last one’s not as obvious as it ought to be but theydid at least make it freely available!

• Want to upload some stuff to PyPI? This should do it for almost any project:

  123	$pipinstalltwine$pythonsetup.pysdistbdist_wheel$twineuploaddist/*

• Want to build wheels for the wild and wooly world of Linux?There’s an app⁴ for that.

Importantly, PyPI willalmost certainly be online. Not only that, but anew, revamped site will be “launching” any day now³.

Again, this isn’t a comprehensive resource; I just want to give you an idea ofwhat’s possible. But, as a deeply experienced Python expert I used to swear atthese tools six times a day for years; the most serious Python packaging issueI’ve had this year to date was fixed by cleaning up my git repo to delete acache file.

Work Still To Do

While the current situation isgood, it’s still notgreat.

Here are just a few of my desiderata:

• We still need better and more universally agreed-upon tooling forend-user deployments.
• Pip should have a GUI frontend so that users can write Python stuff without learning as much command-line arcana.
• There should be tools that help you write and update asetup.py. Or asetup.python.json or something, so you don’t actually need to write code just to ship some metadata.
• The error messages that you get when you try to build something that needs a C compiler and it doesn’t work should beclearer and more actionable for users who don’t already know what they mean.
• PyPI should automatically build wheels for all platforms by default when you upload sdists; this is a huge project, of course, but it would be super awesome default behavior.

I could go on. There are lots of ways that Python packaging could be better.

The Bottom Line

The real takeaway here though, is that although it’s still not perfect,otherlanguages are no longer doing appreciably better. Go isstill working througha number of different options regarding dependency management and vendoring,and, like Python extensions that require C dependencies, CGo is sometimesnecessary andalways a problem. Node hashad its ownwell-publicized problemswith their dependency management culture and package manager. Hackage is cooland all but everything takesa literal geological epoch to compile.

As always, I’m sure none of this applies to Rust and Cargo is basicallyperfect, but that doesn’t matter, because nobody reading this isactually using Rust.

My point is not that packaging in any of these languages is particularly bad.They’re all actually doing pretty well, especially compared to the state of thegeneral programming ecosystem a few years ago; many of them are making regularprogress towards user-facing improvements.

My point is that any commentary suggesting they’re meaningfullybetter thanPython at this point is probably just out of date. Working with Pythonpackaging is more or less fine right now. It could be better, but lots ofpeople are working on improving it, and the structural problems that preventedthose improvements from being adopted by the community in a timely manner havealmost all been addressed.

Go! Make somevirtualenvs! Hack somesetup.pys! If it’s been a while andyour last experience was really miserable, I promise, it’s better now.

Am I wrong? Did I screw up a detail of your favorite language? Did I forgetto mention the one language environment that has a completely perfect, flawlesspackaging story? Do you feel the need to just yell at a stranger on theInternet about picayune details? Feel free toget in touch!