Recommended Video Course
Publishing Python Packages to PyPI
Table of Contents
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding:Publishing Python Packages to PyPI
Python is famous for coming withbatteries included, and many sophisticated capabilities are available in thestandard library. However, to unlock the full potential of the language, you should also take advantage of the community contributions atPyPI: thePython Packaging Index.
PyPI, typically pronouncedpie-pee-eye, is a repository containing several hundred thousand packages. These range from trivialHello, World
implementations to advanceddeep learning libraries. In this tutorial, you’ll learn how toupload your own package to PyPI. Publishing your project is easier than it used to be. Yet, there are still a few steps involved.
In this tutorial, you’ll learn how to:
Throughout this tutorial, you’ll work with an example project: areader
package that can be used to read Real Python tutorials in your console. You’ll get a quick introduction to the project before going in depth about how to publish this package. Click the link below to access the GitHub repository containing the full source code ofreader
:
Get Source Code:Click here to get access to the source code for the Real Python Feed Reader that you’ll use to publish an open-source package to PyPI.
Packaging in Python can seemcomplicated and confusing for both newcomers and seasoned veterans. You’ll find conflicting advice across the Internet, and what was once considered good practice may now be frowned upon.
The main reason for this situation is that Python is a fairly old programming language. Indeed, thefirst version of Python was released in 1991, before theWorld Wide Web became available to the general public. Naturally, a modern, web-based system for distribution of packages wasn’t included or even planned for in the earliest versions of Python.
Note: For a detailed discussion of Python packaging, check outepisode 156 of theReal Python Podcast.
Instead, Python’s packaging ecosystem has evolved organically over the decades as user needs became clear and technology offered new possibilities. The first packaging support came in the fall of 2000, with thedistutils
library being included in Python 1.6 and 2.0. The Python Packaging Index (PyPI)came online in 2003, originally as a pure index of existing packages, without any hosting capabilities.
Note: PyPI is often referred to as thePython Cheese Shop in reference to Monty Python’s famousCheese Shop sketch. To this day,cheeseshop.python.org
redirects to PyPI.
Over the last decade, many initiatives have improved the packaging landscape, bringing it from the Wild West and into a fairly modern and capable system. This is mainly done throughPython Enhancement Proposals (PEPs) that are reviewed and implemented by thePython Packaging Authority (PyPA) working group.
The most important documents that define how Python packaging works are the following PEPs:
You don’t need to study these technical documents. In this tutorial, you’ll learn how all these specifications come together in practice as you go through the process of publishing your own package.
For a nice overview of the history of Python packaging, check outThomas Kluyver’s presentation at PyCon UK 2019:Python packaging: How did we get here, and where are we going? You can also find more presentations at thePyPA website.
In this section, you’ll get to know a small Python package that you can use as an example that can be published to PyPI. If you already have your own package that you’re looking to publish, then feel free to skim this section and join up again at thenext section.
The package that you’ll see here is calledreader
. It can be used both as a library for downloading Real Python tutorials in your own code and as an application for reading tutorials in your console.
Note: The source code as shown and explained in this section is a simplified—but fully functional—version of the Real Python feed reader. Compared to the version currently published onPyPI, this version lacks some error handling and extra options.
First, have a look at the directory structure ofreader
. The package lives completely inside a directory that can be named anything. In this case, it’s namedrealpython-reader/
. The source code is wrapped inside ansrc/
directory. This isn’t strictly necessary, but it’s usually agood idea.
Note: The use of an extrasrc/
directory when structuring packages has been a point ofdiscussion in the Python community for years. In general, a flat directory structure is slightly easier to get started with, but thesrc/
-structure provides severaladvantages as your project grows.
The innersrc/reader/
directory contains all your source code:
realpython-reader/│├── src/│ └── reader/│ ├── __init__.py│ ├── __main__.py│ ├── config.toml│ ├── feed.py│ └── viewer.py│├── tests/│ ├── test_feed.py│ └── test_viewer.py│├── LICENSE├── MANIFEST.in├── README.md└── pyproject.toml
The source code of the package is in ansrc/
subdirectory together with a configuration file. There are a few tests in a separatetests/
subdirectory. The tests themselves won’t be covered in this tutorial, but you’ll learn how to treat test directorieslater. You can learn more about testing in general inGetting Started With Testing in Python andEffective Python Testing With Pytest.
If you’re working with your own package, then you may use a different structure or have other files in your package directory.Python Application Layouts discusses several different options. The steps below for publishing to PyPI will work independently of the layout you use.
In the rest of this section, you’ll see how thereader
package works. In thenext section, you’ll learn more about the special files likeLICENSE
,MANIFEST.in
,README.md
, andpyproject.toml
that you’ll need to publish your package.
reader
is a basicweb feed reader that can download the latest Real Python tutorials from theReal Python feed.
In this section, you’ll first see a few examples of the output that you can expect fromreader
. You can’t run these examples yourself yet, but they should give you some idea of how the tool works.
Note: If you’ve downloaded the source code ofreader
, then you can follow along by first creating avirtual environment and then installing the package locally inside that virtual environment:
(venv)$python-mpipinstall-e.
Throughout the tutorial, you’ll learn more about what happens under the hood when you run this command.
The first example uses the reader to get a list of the latest articles:
$python-mreaderThe latest tutorials from Real Python (https://realpython.com/) 0 How to Publish an Open-Source Python Package to PyPI 1 The Real Python Podcast – Episode #110 2 Build a URL Shortener With FastAPI and Python 3 Using Python Class Constructors 4 Linear Regression in Python 5 The Real Python Podcast – Episode #109 6 pandas GroupBy: Your Guide to Grouping Data in Python 7 Deploying a Flask Application Using Heroku 8 Python News: What's New From April 2022 9 The Real Python Podcast – Episode #108 10 Top Python Game Engines 11 Testing Your Code With pytest 12 Python's min() and max(): Find Smallest and Largest Values 13 Real Python at PyCon US 2022 14 Why Is It Important to Close Files in Python? 15 Combining Data in Pandas With merge(), .join(), and concat() 16 The Real Python Podcast – Episode #107 17 Python 3.11 Preview: Task and Exception Groups 18 Building a Django User Management System 19 How to Get the Most Out of PyCon US
This list shows the most recent tutorials, so your list may be different from what you see above. Still, notice that each article is numbered. To read one particular tutorial, you use the same command but include the number of the tutorial as well.
Note: The Real Python feed contains limited previews of articles. Therefore, you won’t be able to read the full tutorials withreader
.
In this case, to readHow to Publish an Open-Source Python Package to PyPI, you add0
to the command:
$python-mreader0How to Publish an Open-Source Python Package to PyPIPython is famous for coming with batteries included, and many sophisticatedcapabilities are available in the standard library. However, to unlock thefull potential of the language, you should also take advantage of thecommunity contributions at PyPI: the Python Packaging Index.PyPI, typically pronounced pie-pee-eye, is a repository containing severalhundred thousand packages. These range from trivial Hello, Worldimplementations to advanced deep learning libraries. In this tutorial,you'll learn how to upload your own package to PyPI. Publishing yourproject is easier than it used to be. Yet, there are still a fewsteps involved.[...]
This prints the article to the console using theMarkdown format.
Note:python -m
isused to execute amodule or apackage. It works similarly topython
for modules and regular scripts. For examplepython module.py
andpython -m module
are mostly equivalent.
When you run a package with-m
, the file__main__.py
within the package is executed. SeeCall the Reader for more information.
For now, you’ll need to run thepython -m reader
command from inside thesrc/
directory.Later, you’ll learn how you can run the command from any working directory.
By changing the number on the command line, you can read any of the available tutorials.
The details of howreader
works aren’t important for the purpose of this tutorial. However, if you’re interested in learning more about the implementation, then you can expand the sections below. The package consists of five files:
config.toml
is a configuration file used to specify the URL of thefeed of Real Python tutorials. It’s a text file that can be read by thetomli
third-party library:
# config.toml[feed]url="https://realpython.com/atom.xml"
In general, TOML files contain key-value pairs separated into sections, or tables. This particular file contains only one section,feed
, with one key,url
.
Note: A configuration file might be overkill for this simple package. You could instead define the URL as a module level constant directly in your source code. The configuration file is included here to demonstrate how to work with non-code files.
TOML is a configuration file format that has gained popularity lately. Python uses it for thepyproject.toml
file that you’ll learn aboutlater. To dive deeper into TOML, check outPython and TOML: New Best Friends.
Support for reading TOML files willbe added to the standard library in Python 3.11 with a newtomllib
library. Until then, you can use the third-partytomli
package.
The first source code file that you’ll have a look at is__main__.py
. Thedouble underscores indicate that this file has aspecial meaning in Python. Indeed, when executing a package withpython -m
as you did earlier, Python runs the contents of__main__.py
.
In other words,__main__.py
acts as the entry point of your program and takes care of the main flow, calling other parts as needed:
1# __main__.py 2 3importsys 4 5fromreaderimportfeed,viewer 6 7defmain(): 8"""Read the Real Python article feed""" 910# If an article ID is given, then show the article11iflen(sys.argv)>1:12article=feed.get_article(sys.argv[1])13viewer.show(article)1415# If no ID is given, then show a list of all articles16else:17site=feed.get_site()18titles=feed.get_titles()19viewer.show_list(site,titles)2021if__name__=="__main__":22main()
Notice thatmain()
is called on the last line. If you don’t callmain()
, then the program won’t do anything. As you saw earlier, the program can either list all tutorials or print one specific tutorial. This is handled by theif
…else
block on lines 11 to 19.
The next file is__init__.py
. Again, the double underscores in the filename tell you that this is a special file.__init__.py
represents the root of your package. It should usually be kept quite simple, but it’s a good place to put package constants, documentation, and so on:
# __init__.pyfromimportlibimportresourcestry:importtomllibexceptModuleNotFoundError:importtomliastomllib# Version of the realpython-reader package__version__="1.0.0"# Read URL of the Real Python feed from config file_cfg=tomllib.loads(resources.read_text("reader","config.toml"))URL=_cfg["feed"]["url"]
The special variable__version__
is a convention in Python for adding version numbers to your package. It was introduced inPEP 396. You’ll learn more about versioninglater.
Note: You can useimportlib.metadata
to inspect the version of any package that you’ve installed:
>>>fromimportlibimportmetadata>>>metadata.version("realpython-reader")'1.0.0'
importlib.metadata
identifies packages by their PyPI names, which is why you look uprealpython-reader
and notreader
. You’ll learn more about different names for your projectlater.
Specifying__version__
is—strictly speaking—not necessary. Theimportlib.metadata
machinery uses project metadata to find version numbers. However, it’s still a useful convention to follow, and tools like Setuptools and Flit can use it to automatically update your package metadata.
To read the URL to the feed from the configuration file, you usetomllib
ortomli
for TOML support. Usingtry
…except
as above ensures that you usetomllib
if it’s available and that youfall back totomli
if it’s not.
importlib.resources
is used to import non-code or resource files from a package without having to figure out their full file paths. This is especially helpful when you publish your package to PyPI and don’t have full control over where your package is installed and how it’s used. Resource files might even end upinside zip archives.
importlib.resources
became a part of Python’s standard library inPython 3.7. SeeBarry Warsaw’s presentaion at PyCon 2018 for more information.
Variables defined in__init__.py
become available as variables in the package namespace:
>>>importreader>>>reader.__version__'1.0.0'>>>reader.URL'https://realpython.com/atom.xml'
You can access the package constants as attributes directly onreader
.
In__main__.py
, you import two modules,feed
andviewer
and use those to read from the feed and show the results. Those modules do most of the actual work.
First considerfeed.py
. This file contains functions for reading from a web feed and parsing the result. Luckily, there are already great libraries available to do this.feed.py
depends on two modules that are already available on PyPI:feedparser
andhtml2text
.
feed.py
consists of several functions. You’ll look at them one at a time.
Reading a web feed can potentially take a bit of time. To avoid reading from the web feed more than necessary, you use@cache
tocache the feed the first time it’s read:
# feed.pyfromfunctoolsimportcacheimportfeedparserimporthtml2textimportreader@cachedef_get_feed(url=reader.URL):"""Read the web feed, use caching to only read it once"""returnfeedparser.parse(url)
feedparser.parse()
reads a feed from the web and returns it in a structure that looks like adictionary. To avoid downloading the feed more than once, the function isdecorated with@cache
, which remembers the value returned by_get_feed()
and reuses it on later invocations.
Note: The@cache
decorator was introduced tofunctools
inPython 3.9. On older versions of Python, you can use@lru_cache
instead.
You prefix_get_feed()
with an underscore to indicate that it’s a support function and isn’t meant to be used directly by users of your package.
You can get some basic information about the feed by looking in the.feed
metadata. The following function picks out the title and link to the website containing the feed:
# feed.py# ...defget_site(url=reader.URL):"""Get name and link to website of the feed"""feed=_get_feed(url).feedreturnf"{feed.title} ({feed.link})"
In addition to.title
and.link
,attributes like.subtitle
,.updated
, and.id
are also available.
The articles available in the feed can be found inside the.entries
list. Article titles can be found with alist comprehension:
# feed.py# ...defget_titles(url=reader.URL):"""List titles in the feed"""articles=_get_feed(url).entriesreturn[a.titleforainarticles]
.entries
lists the articles in the feed sorted chronologically, so that the newest article is.entries[0]
.
In order to get the contents of one specific article, you use its index in.entries
as an article ID:
# feed.py# ...defget_article(article_id,url=reader.URL):"""Get article from feed with the given ID"""articles=_get_feed(url).entriesarticle=articles[int(article_id)]html=article.content[0].valuetext=html2text.html2text(html)returnf"#{article.title}\n\n{text}"
After picking the correct article from.entries
, you find the text of the article in HTML format and store it asarticle
. Next,html2text
does a decent job of translating the HTML into more readable Markdown text. The HTML doesn’t contain the article title, so the title is added before the article text is returned.
The final module isviewer.py
. It consists of two small functions. In practice, you could’ve usedprint()
directly in__main__.py
instead of callingviewer
functions. However, having the functionality split off makes it more straightforward to replace it with something more advanced later.
Here, you use plainprint()
statements to show contents to the user. As an improvement, maybe you’d want to addricher formatting or aGUI interface to your reader. To do so, you’d only need to replace the following two functions:
# viewer.pydefshow(article):"""Show one article"""print(article)defshow_list(site,titles):"""Show list of article titles"""print(f"The latest tutorials from{site}")forarticle_id,titleinenumerate(titles):print(f"{article_id:>3}{title}")
show()
prints one article to the console, whileshow_list()
prints a list of titles. The latter also creates article IDs that are used when choosing to read one particular article.
In addition to these source code files, you need to add some special files before you can publish your package. You’ll cover those files in later sections.
One challenge when your project grows in complexity is letting your users know how they can use your project. Sincereader
consists of four different source code files, how does the user know which file to execute in order to use the application?
Note: A single Python file is typically referred to as ascript or amodule. You can think of apackage as a collection of modules.
Most commonly, yourun a Python script by providing its filename. For instance, if you have a script calledhello.py
, then you can run it as follows:
$pythonhello.pyHi there!
This hypothetical script printsHi there!
to your console when you run it. Equivalently, you can use the-m
option of thepython
interpreter program to run a script by specifying its module name instead of the filename:
$python-mhelloHi there!
For modules in your current directory, the module name is the same as the filename except that the.py
-suffix is left out.
One advantage of using-m
is that it allows you to call all modules that are in yourPython path, including those that are built into Python. One example is callingantigravity
:
$python-mantigravityCreated new window in existing browser session.
If you want to run a built-in module without-m
, then you’ll need to first look up where it’s stored on your system and then call it with its full path.
Another advantage of using-m
is that it works for packages as well as modules. As you learned earlier, you can call thereader
package with-m
as long as thereader/
directory is available in your working directory:
$cdsrc/$python-mreader
Becausereader
is a package, the name only refers to a directory. How does Python decide which code inside that directory to run? It looks for a file named__main__.py
. If such a file exists, then it’s executed. If it doesn’t exist, then an error message is printed:
$python-murllibpython: No module named urllib.__main__; 'urllib' is a package and cannot be directly executed
The error message says that the standard library’surllib
package hasn’t defined a__main__.py
file.
If you’re creating a package that’s supposed to be executed, then you should include a__main__.py
file. You can also followRich’s great example of usingpython -m rich
todemonstrate the capabilities of your package.
Later, you’ll see how you can also createentry points to your package that behave like regular command-line programs. These will be even easier for your end users to use.
You’ve got a package that you want to publish. Maybe you’ve copiedreader
, or maybe you have your own package. In this section, you’ll see which steps you need to take before you upload your package to PyPI.
The first—and possibly thehardest—step is to come up with a good name for your package. All packages on PyPI need to have unique names. There are now several hundred thousand packages on PyPI, so chances are that your favorite name is already taken.
As a case in point, there’salready a package on PyPI namedreader
. One way to make a package name unique is to add a recognizable prefix to the name. In this example, you’ll userealpython-reader
as the PyPI name for thereader
package.
Whichever PyPI name you choose for your package, that’s the one you’ll use when you install it withpip
:
$python-mpipinstallrealpython-reader
Note that the PyPI name does not need to match the package name. Here, the package is still namedreader
, and that’s the name you need to use when importing the package:
>>>importreader>>>reader.__version__'1.0.0'>>>fromreaderimportfeed>>>feed.get_titles()['How to Publish an Open-Source Python Package to PyPI', ...]
Sometimes you need to use different names for your package. However, you’re keeping things simpler for your users if the package name and the PyPI name are the same.
Be aware that even though the package name doesn’t need to be globally unique like the PyPI name, it does need to be unique across the environment that you’re running it in.
If you install two packages with the same package name, for examplereader
andrealpython-reader
, then a statement likeimport reader
is ambigous. Python resolves this by importing the package that it finds first in the import path. Often, this will be the first package when sorting the names alphabetically. However, you shouldn’t depend on this behavior.
Usually, you’d want your package names to be as unique as possible as well, while balancing this with the convenience of a short and succinct name. Therealpython-reader
is a specialized feed reader, whilereader
on PyPI is more general. For the purposes of this tutorial, there’s no reason to need both, so the compromise with the non-unique name might be worth it.
In order to prepare your package for publication on PyPI, you need to provide some information about it. In general, you need to specify two kinds of information:
Abuild system is responsible for creating the actual files that you’ll upload to PyPI, typically in thewheel or thesource distribution (sdist) format. For a long time, this was done bydistutils
orsetuptools
. However,PEP 517 andPEP 518 introduced a way to specify custom build systems.
Note: You can choose which build system you use in your projects. The main difference between different build systems is how you configure your package and which commands you run to build and upload your package.
This tutorial will focus on usingsetuptools
as a build system. Still,later you’ll learn how to use alternatives like Flit and Poetry.
Each Python project should use a file namedpyproject.toml
to specify its build system. You can usesetuptools
by adding thefollowing topyproject.toml
:
# pyproject.toml[build-system]requires=["setuptools>=61.0.0","wheel"]build-backend="setuptools.build_meta"
This specifies that you’re usingsetuptools
as a build system as well as which dependencies Python must install in order to build your package. Typically, thedocumentation of your chosen build system will tell you how to write thebuild-system
table inpyproject.toml
.
The more interesting information that you need to provide concerns your package itself.PEP 621 defines how metadata about your package can also be included inpyproject.toml
in a way that’s as uniform as possible across different build systems.
Note: Historically, Setuptools usedsetup.py
to configure your package. Because this is an actual Python script that’s run at installation, it’s very powerful, and it may still be required when building complex packages.
However, it’s usually better to use a declarative configuration file to express how to build your package, as it’s more straightforward to reason about and comes with fewer pitfalls to worry about. Usingsetup.cfg
is the most common way to configure Setuptools.
However, Setuptools ismoving toward usingpyproject.toml
as specified in PEP 621. In this tutorial, you’ll be usingpyproject.toml
for all your package configuration.
To learn more about usingpyproject.toml
for configuring Python projects, managing dependencies, and streamlining builds, check out theHow to Manage Python Projects With pyproject.toml tutorial.
A fairly minimal configuration of thereader
package can look like this:
# pyproject.toml[build-system]requires=["setuptools>=61.0.0","wheel"]build-backend="setuptools.build_meta"[project]name="realpython-reader"version="1.0.0"description="Read the latest Real Python tutorials"readme="README.md"authors=[{name="Real Python",email="info@realpython.com"}]license={file="LICENSE"}classifiers=["License :: OSI Approved :: MIT License","Programming Language :: Python","Programming Language :: Python :: 3",]keywords=["feed","reader","tutorial"]dependencies=["feedparser >= 5.2.0","html2text",'tomli; python_version < "3.11"',]requires-python=">=3.9"[project.optional-dependencies]dev=["black","bumpver","isort","pip-tools","pytest"][project.urls]Homepage="https://github.com/realpython/reader"[project.scripts]realpython="reader.__main__:main"
Most of this information is optional, and there are other settings you can use that aren’t included in this example. Check out thedocumentation for all the details.
The minimal information that you must include in yourpyproject.toml
is the following:
name
specifies the name of your package as it will appear on PyPI.version
sets the current version of your package.As the example above shows, you can include much more information. A few of the other keys inpyproject.toml
are interpreted as follows:
classifiers
describes your project using a list ofclassifiers. You should use these as they make your project more searchable.dependencies
lists anydependencies your package has to third-party libraries.reader
depends onfeedparser
,html2text
, andtomli
, so they’re listed here.project.urls
adds links that you can use to present additional information about your package to your users. You can include several links here.project.scripts
creates command-line scripts that call functions within your package. Here, the newrealpython
command callsmain()
within thereader.__main__
module.Theproject.scripts
table is one of three tables that can handleentry points. You can alsoincludeproject.gui-scripts
andproject.entry-points
, which specify GUI applications andplugins, respectively.
The purpose of all this information is to make your package attractive and findable on PyPI. Have a look at therealpython-reader
project page on PyPI and compare the information withpyproject.toml
above:
All the information on PyPI comes frompyproject.toml
andREADME.md
. For example, the version number is based on the lineversion = "1.0.0"
inproject.toml
, whileRead the latest Real Python tutorials is copied fromdescription
.
Furthermore, the project description is lifted from yourREADME.md
file. In the sidebar, you can find information fromproject.urls
in theProject links section and fromlicense
andauthors
in theMeta section. The values you specified inclassifiers
are visible at the bottom of the sidebar.
SeePEP 621 for details about all the keys. You’ll learn more aboutdependencies
, as well asproject.optional-dependencies
, in the next subsection.
Your package will likely depend on third-party libraries that aren’t part of the standard library. You should specify these in thedependencies
list inpyproject.toml
. In the example above, you did the following:
dependencies=["feedparser >= 5.2.0","html2text",'tomli; python_version < "3.11"',]
This specifies thatreader
depends onfeedparser
,html2text
, andtomli
. Furthermore, it says the following:
feedparser
must be version 5.2.0 or later.html2text
can be any version.tomli
can be any version, but is only required on Python 3.10 or earlier.This shows a few possibilities that you can use when specifying dependencies, includingversion specifiers andenvironment markers. You can use the latter to account for different operating systems, Python versions, and so on.
Note, however, that you should strive to only specify the minimum requirements needed for your library or application to work. This list will be used bypip
to resolve dependencies any time your package is installed. By keeping this list minimal, you ensure that your package is as compatible as possible.
You may have heard that you shouldpin your dependencies. That’sgreat advice! However, it doesn’t hold in this case. You pin your dependencies to make sure your environment is reproducible. Your package, on the other hand, should hopefully work across many different Python environments.
When adding packages todependencies
you should follow theserules of thumb:
reader
importsfeedparser
,html2text
, andtomli
, so those are listed. On the other hand,feedparser
depends onsgmllib3k
, butreader
doesn’t use this library directly, so it’s not specified.==
.>=
to add a lower bound if you depend on functionality that was added in a particular version of your dependency.<
to add an upper bound if you worry that a dependency may break compatibility in a major version upgrade. In this case, you should diligently test such upgrades and remove or increase the upper bound if possible.Note that these rules hold when you’re configuring a package that you’re making available for others. If you’re deploying your package, then you should pin your dependencies inside avirtual environment.
Thepip-tools project is a great way to manage pinned dependencies. It comes with apip-compile
command that can create or update a complete list of dependencies.
As an example, say that you’re deployingreader
into a virtual environment. You can then create a reproducible environment with pip-tools. In fact,pip-compile
can work directly with yourpyproject.toml
file:
(venv)$python-mpipinstallpip-tools(venv)$pip-compilepyproject.tomlfeedparser==6.0.8 via realpython-reader (pyproject.toml)html2text==2020.1.16 via realpython-reader (pyproject.toml)sgmllib3k==1.0.0 via feedparsertomli==2.0.1 ; python_version < "3.11" via realpython-reader (pyproject.toml)
pip-compile
creates a detailedrequirements.txt
file with contents similar to the output above. You can usepip install
orpip-sync
to install these dependencies into your environment:
(venv)$pip-syncCollecting feedparser==6.0.8 ...Installing collected packages: sgmllib3k, tomli, html2text, feedparserSuccessfully installed feedparser-6.0.8 html2text-2020.1.16 sgmllib3k-1.0.0 tomli-2.0.1
See thepip-tools documentation for more information.
You’re also allowed to specify optional dependencies of your package in a separate table namedproject.optional-dependencies
. Often you use this to specify dependencies that you use during development or testing. However, you can also specify extra dependencies that are used to support certain features in your package.
In the example above, you included the following section:
[project.optional-dependencies]dev=["black","bumpver","isort","pip-tools","pytest"]
This adds one group,dev
, of optional dependencies. You can have several such groups, and you can name the groups however makes sense.
By default, optional dependencies aren’t included when a package is installed. However, by adding the group name in square brackets when runningpip
, you can manually specify that they should be installed. For example, you can install the extradev
dependencies ofreader
by doing the following:
(venv)$python-mpipinstallrealpython-reader[dev]
You can also include optional dependencies when pinning your dependencies withpip-compile
by using the--extra
command line option:
(venv)$pip-compile--extradevpyproject.tomlattrs==21.4.0 via pytestblack==22.3.0 via realpython-reader (pyproject.toml)...tomli==2.0.1 ; python_version < "3.11" via black pytest realpython-reader (pyproject.toml)
This creates a pinnedrequirements.txt
file that includes both your regular and development dependencies.
You should add somedocumentation before you release your package to the world. Depending on your project, your documentation can be as small as a singleREADME
file or as comprehensive as a full web page with tutorials, example galleries, and an API reference.
At a minimum, you should include aREADME
file with your project. A goodREADME
should quickly describe your project, as well as explain how to install and use your package. Often, you want to reference yourREADME
in thereadme
key inpyproject.toml
. This will display the information on the PyPI project page as well.
You can useMarkdown orreStructuredText as formats for project descriptions. PyPIfigures out which format you’re using based on the file extension. If you don’t need any of the advanced features of reStructuredText, then you’re usually better off using Markdown for yourREADME
. It’s simpler and has wider support outside of PyPI.
For bigger projects, you might want to offer more documentation than can reasonably fit in a single file. In that case, you can host your documentation on sites likeGitHub orRead the Docs and link to it from the PyPI project page.
You can link to other URLs by specifying them in theproject.urls
table inpyproject.toml
. In the example, the URLs section is used to link to thereader
GitHub repository.
Tests are useful when you’re developing your package, and you should include them. As noted, you won’t cover testing in this tutorial, but you can have a look at the tests ofreader
in thetests/
source code directory.
You can learn more about testing inEffective Python Testing With Pytest, and get some hands-on experience with test-driven development (TDD) inBuild a Hash Table in Python With TDD andPython Practice Problems: Parsing CSV Files.
When preparing your package for publication, you should be conscious of the role that tests play. They’re typically only interesting for developers, so they shouldnot be included in the package that you distribute through PyPI.
Later versions of Setuptools are quite good atcode discovery and will normally include your source code in the package distribution but leave out your tests, documentation, and similar development artifacts.
You can control exactly what’s included in your package by usingfind
directives inpyproject.toml
. See theSetuptools documentation for more information.
Your package needs to have a version. Furthermore, PyPI will only let you upload a particular version of your package once. In other words, if you want to update your package on PyPI, then you need to increase the version number first. This is a good thing because it helps guarantee reproducibility: two environments with the same version of a given package should behave the same.
There are many differentversioning schemes that you can use. For Python projects,PEP 440 gives some recommendations. However, in order to be flexible, the description in that PEP is complicated. For a simple project, you should stick with a simple versioning scheme.
Semantic versioning is a good default scheme to use, although it’snot perfect. You specify the version as three numerical components, for instance1.2.3
. The components are called MAJOR, MINOR, and PATCH, respectively. The following are recommendations about when to increment each component:
- Increment the MAJOR version when you make incompatible API changes.
- Increment the MINOR version when you add functionality in a backwards compatible manner.
- Increment the PATCH version when you make backwards compatible bug fixes. (Source)
You should reset PATCH to0
when you increment MINOR, and reset both PATCH and MINOR to0
when you increment MAJOR.
Calendar versioning is an alternative to semantic versioning that’s gaining popularity and is used by projects likeUbuntu,Twisted,Black, andpip
. Calendar versions also consist of several numerical components, but one or several of these are tied to the current year, month, or week.
Often, you want to specify the version number in different files within your project. For example, the version number is mentioned in bothpyproject.toml
andreader/__init__.py
in thereader
package. To help you make sure that version numbers stay consistent, you can use a tool likeBumpVer.
BumpVer allows you to write version numbers directly into your files and then update these as necessary. As an example, you can install and integrate BumpVer into your project as follows:
(venv)$python-mpipinstallbumpver(venv)$bumpverinitWARNING - Couldn't parse pyproject.toml: Missing version_patternUpdated pyproject.toml
Thebumpver init
command creates a section in yourpyproject.toml
that allows you to configure the tool for your project. Depending on your needs, you may need to change many of the default settings. Forreader
, you may end up with something like the following:
[tool.bumpver]current_version="1.0.0"version_pattern="MAJOR.MINOR.PATCH"commit_message="Bump version {old_version} -> {new_version}"commit=truetag=truepush=false[tool.bumpver.file_patterns]"pyproject.toml"=['current_version = "{version}"','version = "{version}"']"src/reader/__init__.py"=["{version}"]
For BumpVer to work properly, you must specify all files that contain your version number in thefile_patterns
subsection. Note that BumpVer plays well with Git and can automatically commit, tag, and push when you update the version numbers.
Note: BumpVer integrates with yourversion control system. It’ll refuse to update your files if you have uncommitted changes in your repository.
After you’ve set up the configuration, you can bump the version in all your files with a single command. For example, to increase the MINOR version ofreader
, you would do the following:
(venv)$bumpverupdate--minorINFO - Old Version: 1.0.0INFO - New Version: 1.1.0
This changes the version number from 1.0.0 to 1.1.0 in bothpyproject.toml
and__init__.py
. You can use the--dry
flag to see which changes BumpVer would make, without actually executing them.
Sometimes, you’ll have files inside your package that aren’t source code files. Examples include data files, binaries, documentation, and—as you have in this example—configuration files.
To make sure such files are included when your project is built, you use a manifest file. For many projects, you don’t need to worry about the manifest: by default, Setuptools includes all source code files andREADME
files in the build.
If you have other resource files and need to update the manifest, then you need to create a file namedMANIFEST.in
next topyproject.toml
in your project’s base directory. This file specifies rules for which files to include and which files to exclude:
# MANIFEST.ininclude src/reader/*.toml
This example will include all.toml
files in thesrc/reader/
directory. In effect, this is the configuration file.
See thedocumentation for more information about setting up your manifest. Thecheck-manifest tool can also be useful for working withMANIFEST.in
.
If you’re sharing your package with others, then you need to add a license to your package that explains how others are allowed to use your package. For example,reader
is distributed according to theMIT license.
Licenses are legal documents, and you typically don’t want to write your own. Instead, you shouldchoose one of themany licenses already available.
You should add a file namedLICENSE
to your project that contains the text of the license you choose. You can then reference this file inpyproject.toml
to make the license visible on PyPI.
You’ve done all the necessary setup and configuration for your package. In thenext section, you’ll learn how to finally get your package on PyPI. First, though, you’ll learn abouteditable installs. This is a way of usingpip
to install your package locally in a way that lets you edit your code after it’s installed.
Note: Normally,pip
does aregular install, which places a package into yoursite-packages/
folder. If you install your local project, then the source code will be copied tosite-packages/
.The effect of this is that later changes that you make won’t take effect. You’ll need to reinstall your package first.
During development, this can be both ineffective and frustrating. Editable installs work around this by linking directly to your source code.
Editable installs have been formalized inPEP 660. These are useful when you’re developing your package, as you can test all the functionality of your package and update your source code without needing to do a reinstallation.
You usepip
to install your package in editable mode by adding the-e
or--editable
flag:
(venv)$python-mpipinstall-e.
Note the period (.
) at the end of the command. It’s a necessary part of the command and tellspip
that you want to install the package located in the current working directory. In general, this should be the path to the directory containing yourpyproject.toml
file.
Note: You may get an error message saying“Project file has a ‘pyproject.toml’ and its build backend is missing the ‘build_editable’ hook.” This is due to alimitation in Setuptools support for PEP 660. You can work around this by adding a file namedsetup.py
with the following contents:
# setup.pyfromsetuptoolsimportsetupsetup()
This shim delegates the job of doing an editable install to Setuptools’ legacy mechanism until native support for PEP 660 is available.
Once you’ve successfully installed your project, it’s available inside your environment, independent of your current directory. Additionally, your scripts are set up so you can run them. Recall thatreader
defined a script namedrealpython
:
(venv)$realpythonThe latest tutorials from Real Python (https://realpython.com/) 0 How to Publish an Open-Source Python Package to PyPI [...]
You can also usepython -m reader
from any directory, or import your package from a REPL or another script:
>>>fromreaderimportfeed>>>feed.get_titles()['How to Publish an Open-Source Python Package to PyPI', ...]
Installing your package in editable mode during development makes your development experience much more pleasant. It’s also a good way of locating certain bugs where you may have unconsciously depended on files being available in your current working directory.
It’s taken some time, but this wraps up the preparations you need to do to your package. In the next section, you’ll learn how to actually publish it!
Your package is finally ready to meet the world outside your computer! In this section, you’ll learn how to build your package and upload it to PyPI.
If you don’t already have an account on PyPI, then now is the time toregister your account on PyPI. While you’re at it, you should alsoregister an account on TestPyPI. TestPyPI is very useful! You can try out all the steps of publishing a package without any consequences if you mess up.
To build and upload your package to PyPI, you’ll use two tools calledBuild andTwine. You can install them usingpip
as usual:
(venv)$python-mpipinstallbuildtwine
You’ll learn how to use these tools in the upcoming subsections.
Packages on PyPI aren’t distributed as plain source code. Instead, they’re wrapped into distribution packages. The most common formats for distribution packages are source archives andPython wheels.
Note: Wheels arenamed in reference tocheese wheels, which are the most important items in acheese shop.
A source archive consists of your source code and any supporting files wrapped into onetar
file. Similarly, a wheel is essentially a zip archive containing your code. You should provide both source archives and wheels for your package. Wheels are usually faster and more convenient for your end users, while source archives provide a flexible backup alternative.
To create a source archive and a wheel for your package, you use Build:
(venv)$python-mbuild[...]Successfully built realpython-reader-1.0.0.tar.gz and realpython_reader-1.0.0-py3-none-any.whl
As the final line in the output says, this creates a source archive and a wheel. You can find them in a newly createddist
directory:
realpython-reader/│└── dist/ ├── realpython_reader-1.0.0-py3-none-any.whl └── realpython-reader-1.0.0.tar.gz
The.tar.gz
file is your source archive, while the.whl
file is your wheel. These are the files that you’ll upload to PyPI and thatpip
will download when it installs your package later.
Before uploading your newly built distribution packages, you should check that they contain the files you expect. The wheel file is really aZIP file with a different extension. You can unzip it and inspect its contents as follows:
(venv)PS>cd.\dist(venv)PS>Copy-Item.\realpython_reader-1.0.0-py3-none-any.whlreader-whl.zip(venv)PS>Expand-Archivereader-whl.zip(venv)PS>tree.\reader-whl\/FC:\REALPYTHON-READER\DIST\READER-WHL├───reader│ config.toml│ feed.py│ viewer.py│ __init__.py│ __main__.py│└───realpython_reader-1.0.0.dist-info entry_points.txt LICENSE METADATA RECORD top_level.txt WHEEL
You first rename the wheel file to have a.zip
extension so that you can expand it.
(venv)$cddist/(venv)$unziprealpython_reader-1.0.0-py3-none-any.whl-dreader-whl(venv)$treereader-whl/reader-whl/├── reader│ ├── config.toml│ ├── feed.py│ ├── __init__.py│ ├── __main__.py│ └── viewer.py└── realpython_reader-1.0.0.dist-info ├── entry_points.txt ├── LICENSE ├── METADATA ├── RECORD ├── top_level.txt └── WHEEL2 directories, 11 files
You should see all your source code listed, as well as a few new files that have been created and contain information you provided inpyproject.toml
. In particular, make sure that all subpackages and supporting files likeconfig.toml
are included.
You can also have a look inside the source archive as it’s packaged as atar ball. However, if your wheel contains the files you expect, then the source archive should be fine as well.
Twine can alsocheck that your package description will render properly on PyPI. You can runtwine check
on the files created indist
:
(venv)$twinecheckdist/*Checking distribution dist/realpython_reader-1.0.0-py3-none-any.whl: PassedChecking distribution dist/realpython-reader-1.0.0.tar.gz: Passed
This won’t catch all the problems that you might run into, but it’s a good first line of defense.
Now you’re ready to actually upload your package to PyPI. For this, you’ll again use the Twine tool, telling it to upload the distribution packages that you have built.
First, you should upload toTestPyPI to make sure everything works as expected:
(venv)$twineupload-rtestpypidist/*
Twine will ask you for your username and password.
Note: If you’ve followed the tutorial using thereader
package as an example, then the previous command will probably fail with a message saying you aren’t allowed to upload to therealpython-reader
project.
You can changename
inpyproject.toml
to something unique, for exampletest-<your-username>
. Then build the project again and upload the newly built files to TestPyPI.
If the upload succeeds, then you can quickly head over toTestPyPI, scroll down, and look at your project being proudly displayed among the new releases! Click on your package and make sure everything looks okay.
If you’ve been following along using thereader
package, then the tutorial ends here! While you can play with TestPyPI as much as you want, you shouldn’t upload sample packages to PyPI just for testing.
Note: TestPyPI is great for checking that your package uploads correctly and that your project page looks as you intended. You can also try to install your package from TestPyPI:
(venv)$python-mpipinstall-ihttps://test.pypi.org/simplerealpython-reader
However, note that this may fail because not all your dependencies are available on TestPyPI. This isn’t a problem. Your package should still work when you upload it to PyPI.
If you have your own package to publish, then the moment has finally arrived! With all the preparations taken care of, this final step is short:
(venv)$twineuploaddist/*
Provide your username and password when requested. That’s it!
Head over toPyPI and look up your package. You can find it either bysearching, by looking at theYour projects page, or by going directly to the URL of your project:pypi.org/project/your-package-name/.
Congratulations! Your package is published on PyPI!
Take a moment to bask in the blue glow of the PyPI web page and brag to your friends.
Then open up a terminal again. There’s one more great payoff!
With your package uploaded to PyPI, you can install it withpip
as well. First, create a new virtual environment and activate it. Then run the following command:
(venv)$python-mpipinstallyour-package-name
Replaceyour-package-name
with the name that you chose for your package. For instance, to install thereader
package, you would do the following:
(venv)$python-mpipinstallrealpython-reader
Seeing your own code installed bypip
—just like any other third-party library—is a wonderful feeling!
In this tutorial, you’ve used Setuptools to build your package. Setuptools is, for better and worse, the long-term standard for creating packages. While it’s widespread and trusted, it also comes with a lot of features that may not be relevant to you.
There are several alternative build systems that you can use instead of Setuptools. Over the last few years, the Python community has done the important job of standardizing the Python packaging ecosystem. This makes it simpler to move between the different build systems and use the one that’s best for your workflow and packages.
In this section, you’ll briefly learn about two alternative build systems that you can use to create and publish your Python packages. In addition to Flit and Poetry, which you’ll learn about next, you can also check outpbr,enscons, andHatchling. Additionally, thepep517
package provides support for creating your own build system.
Flit is a great little project that aims to “make the easy things easy” when it comes to packaging (source). Flit doesn’t support advanced packages like those creatingC extensions, and in general, it doesn’t give you many choices when setting up your package. Instead, Flit subscribes to the philosophy that there should be one obvious workflow to publish a package.
Note: You can’t configure your package with both Setuptools and Flit at the same time. To test out the workflow in this section, you should safely store your Setuptools configuration in your version control system and then delete thebuild-system
andproject
sections inpyproject.toml
.
First install Flit withpip
:
(venv)$python-mpipinstallflit
As much as possible, Flit automates thepreparations that you need to do with your package. To start configuring a new package, runflit init
:
(venv)$flitinitModule name [reader]:Author: Real PythonAuthor email: info@realpython.comHome page: https://github.com/realpython/readerChoose a license (see http://choosealicense.com/ for more info)1. MIT - simple and permissive2. Apache - explicitly grants patent rights3. GPL - ensures that code based on this is shared with the same terms4. Skip - choose a license laterEnter 1-4: 1Written pyproject.toml; edit that file to add optional extra info.
Theflit init
command will create apyproject.toml
file based on the answers that you give to a few questions. You might need to edit this file slightly before using it. For thereader
project, thepyproject.toml
file for Flit ends up looking as follows:
# pyproject.toml[build-system]requires=["flit_core >=3.2,<4"]build-backend="flit_core.buildapi"[project]name="realpython-reader"authors=[{name="Real Python",email="info@realpython.com"}]readme="README.md"license={file="LICENSE"}classifiers=["License :: OSI Approved :: MIT License","Programming Language :: Python :: 3",]dynamic=["version","description"][project.urls]Home="https://github.com/realpython/reader"[project.scripts]realpython="reader.__main__:main"
Note that most of theproject
items are identical to your originalpyproject.toml
. One difference, though, is thatversion
anddescription
are specified in adynamic
field. Flit actually figures these out itself by using__version__
and thedocstring defined in the__init__.py
file.Flit’s documentation explains everything about thepyproject.toml
file.
Flit can build your package and publish it to PyPI. You don’t need to use Build and Twine. To build your package, simply do the following:
(venv)$flitbuild
This creates a source archive and a wheel, similar to what you did withpython -m build
earlier. You can also still use Build if you prefer.
To upload your package to PyPI, you can use Twine as you did earlier. However, you can also use Flit directly:
(venv)$flitpublish
Thepublish
command will build your package if necessary, and then upload the files to PyPI, prompting you for your username and password.
To see an early but recognizable version of Flit in action, have a look at Thomas Kluyver’slightning talk from EuroSciPy 2017. Thedemo shows how you configure your package, build it, and publish it to PyPI in the space of two minutes.
Poetry is another tool that you can use to build and upload your package. Compared to Flit, Poetry has more features that can help you during the development of your packages, including powerfuldependency management.
Before you use Poetry, you need to install it. It’s possible to install Poetry withpip
. However, themaintainers recommend that you use a custom installation script to avoid potential dependency conflicts. Seethe documentation for instructions.
Note: You can’t configure your package with both Setuptools and Poetry at the same time. To test out the workflow in this section, you should safely store your Setuptools configuration in your version control system and then delete thebuild-system
andproject
sections inpyproject.toml
.
With Poetry installed, you start using it with aninit
command, similar to Flit:
(venv)$poetryinitThis command will guide you through creating your pyproject.toml config.Package name [code]: realpython-readerVersion [0.1.0]: 1.0.0Description []: Read the latest Real Python tutorials...
This will create apyproject.toml
file based on your answers to questions about your package.
Note: Poetry doesnot currently support PEP 621, so the actual specifications inside thepyproject.toml
currently differ between Poetry and the other tools.
For Poetry, thepyproject.toml
file ends up looking like the following:
# pyproject.toml[build-system]requires=["poetry-core>=1.0.0"]build-backend="poetry.core.masonry.api"[tool.poetry]name="realpython-reader"version="1.0.0"description="Read the latest Real Python tutorials"authors=["Real Python <info@realpython.com>"]readme="README.md"homepage="https://github.com/realpython/reader"license="MIT"[tool.poetry.dependencies]python=">=3.9"feedparser="^6.0.8"html2text="^2020.1.16"tomli="^2.0.1"[tool.poetry.scripts]realpython="reader.__main__:main"
You should recognize all these items from the earlier discussion ofpyproject.toml
, even though the sections are named differently.
One thing to note is that Poetry will automatically add classifiers based on the license and the version of Python you specify. Poetry also requires you to be explicit about versions of your dependencies. In fact, dependency management is one of the strong points of Poetry.
Just like Flit, Poetry can build and upload packages to PyPI. Thebuild
command creates a source archive and a wheel:
(venv)$poetrybuild
This will create the two usual files in thedist
subdirectory, which you can upload using Twine as earlier. You can also use Poetry to publish to PyPI:
(venv)$poetrypublish
This will upload your package to PyPI. In addition to aiding in building and publishing, Poetry can help you earlier in the process. Poetry can help you start a new project with thenew
command. It also supports working with virtual environments. SeePoetry’s documentation for all the details.
Apart from the slightly different configuration files, Flit and Poetry work very similarly. Poetry is broader in scope, as it also aims to help with dependency management, while Flit has been around a little longer.
You now know how to prepare your project and upload it to PyPI, so that it can be installed and used by other people. While there are a few steps that you need to go through, seeing your own package on PyPI is a great payoff. Having others find your project useful is even better!
In this tutorial, you’ve learned how to publish your own package by:
pyproject.toml
In addition, you’ve learned about the initiatives in the Python packaging community to standardize the tools and processes.
If you still have questions, then feel free to reach out in the comments section below. Also, thePython Packaging User Guide has a lot of information with more detail than what you’ve seen in this tutorial.
Get Source Code:Click here to get access to the source code for the Real Python Feed Reader that you’ll use to publish an open-source package to PyPI.
Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding:Publishing Python Packages to PyPI
🐍 Python Tricks 💌
Get a short & sweetPython Trick delivered to your inbox every couple of days. No spam ever. Unsubscribe any time. Curated by the Real Python team.
AboutGeir Arne Hjelle
Geir Arne is an avid Pythonista and a member of the Real Python tutorial team.
» More about Geir ArneMasterReal-World Python Skills With Unlimited Access to Real Python
Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:
MasterReal-World Python Skills
With Unlimited Access to Real Python
Join us and get access to thousands of tutorials, hands-on video courses, and a community of expert Pythonistas:
What Do You Think?
What’s your #1 takeaway or favorite thing you learned? How are you going to put your newfound skills to use? Leave a comment below and let us know.
Commenting Tips: The most useful comments are those written with the goal of learning from or helping out other students.Get tips for asking good questions andget answers to common questions in our support portal.
Keep Learning
Related Topics:intermediatebest-practicestools
Recommended Video Course:Publishing Python Packages to PyPI
Related Tutorials:
Already have an account?Sign-In
Almost there! Complete this form and click the button below to gain instant access:
How to Publish an Open-Source Python Package to PyPI (Source Code)