Dustin Ingram

The Problem#

One of the questions that I see come up relatively often within the PythonPackaging ecosystem is related to dependencies. It often takes many forms:

How can I produce a dependency graph for Python packages?

Why doesn’t PyPI show a project’s dependencies on it’s project page?

How can I get a project’s dependencies without downloading the package?

Can I search PyPI and filter out projects that have a certain dependency?

Essentially, all these questions are asking the same thing, and often theanswers to these questions are not quite what a user wants to hear, mostlybecause they don’t usually sound like a “yes” and more like a “it can’t bedone”.

Since it might not be immediately clearwhy it can’t always be done, andwhy, sometimes, it can, I’m going to attempt to explain some common“missed-conceptions” that are fundamental to understanding how Python packagedependencies work.

What issetup.py?#

For most folks,setup.py is a familiar name. For new users, though, this filecan often be confusing, so much so that there is even a highly-votedStackOverflow question called, appropriately,“What issetup.py?”

While some of the confusion comes from elsewhere, the main thing to note isthat whilesetup.py often looks very similar from one project to another, atthe end of the day, it isjust another Python script.

This leads us to the next question:

What is the difference betweensetup.py andrequirements.txt?#

This question is a common enough question about Python packaging that it hasalready been answered byDonald’s definitive blog post on thesubject. However,there’s something more to consider here when it comes to getting dependenciesup on PyPI:

• requirements.txt, since it’s just a text file, is guaranteed to bedeterministic (given the dependencies available on your package index);
• setup.py, since it’s a Python script, makes no guarantees about determinismat all.

This means thatsetup.py can possibly give us non-deterministic dependencies.

What do you mean by “non-deterministic dependencies”?#

For example, let’s define the smallestsetup.py possible:

fromsetuptoolsimportsetupsetup(name='paradox',version='0.0.1',description='A nondeterministic package',)

This is pretty straightforward. Now, let’s make it nondeterministic:

+ import randomfrom setuptools import setup++ dependency = random.choice(['Schrodinger', 'Cat'])setup(    name='paradox',    version='0.0.1',    description='A nondeterministic package',+     install_requires=[dependency],)

Now, attempting to install ourparadox will sometimes installSchrodinger as a dependency, andsometimes it will installhis cat.

Why would I want non-deterministic dependencies?#

Making your package install different dependencies usingrandom.choice() is agreat way to upset your users, but not a great way to write Python packages.

While this type of nondeterministic dependency might not make sense, it mightmake a lot of sense to install different dependencies based on the state of theinstalling user’s system – maybe Linux users get Package A, while Windowsusers get Package B instead.

So, why can’t PyPI know my project’s dependencies?#

At this point, you should be able to answer this question yourself!

Since the package author can doanything to come up with a list ofdependencies, given the current ecosystem, there’s no way an index like PyPIcould reliably take into account all the intricacies of your packageinstallation process and determine what they will be.

At this point, you may be asking yourself what can be done to fix this.

First, consider that the current status quo is not necessarily broken: having asetup.py which can install dependencies based onliterally anything is afeature that has enabled better compatibility across an increasingly broadspectrum of platforms.

However, this is annoying for a lot of people: it’s annoying for theninety-something percent of package maintainers that have a reliable and knownset of dependencies for their package, and just want a dependency graph, or tosee a list of projects that depend on their project, etc.

One way to be able to “predict” the dependencies of a package is to severelylimit some of the variables that are commonly used to determine whichdependencies to install, like Python version, platform, etc. This isessentially what thewheel formatis, and it’s why there issome dependency information available on PyPI.

Implementing new specifications likePEP517 will pave the way fordependency specification in a more sane way, and will allow tools likepipand PyPI to extract dependency information from source distributions as well aswheels.