This PEP describes a mirroring infrastructure for PyPI.
The main PyPI web service was moved behind the Fastly caching CDN in May 2013:https://mail.python.org/pipermail/distutils-sig/2013-May/020848.html
Subsequently, this arrangement was formalised as an in-kind sponsorship withthe PSF, and the PSF has also taken on the task of risk management in the eventthat that sponsorship arrangement were to ever cease.
The download statistics that were previously provided directly on PyPI, are nowpublished indirectly via Google Big Query:https://packaging.python.org/guides/analyzing-pypi-package-downloads/
Accordingly, the mirroring proposal described in this PEP is no longer required,and has been marked as Withdrawn.
PyPI is hosting over 6000 projects and is used on a daily basisby people to build applications. Especially systems likeeasy_installandzc.buildout make intensive usage of PyPI.
For people making intensive use of PyPI, it can act as a single pointof failure. People have started to set up some mirrors, both privateand public. Those mirrors are active mirrors, which means that theyare browsing PyPI to get synced.
In order to make the system more reliable, this PEP describes:
People that wants to mirror PyPI make a proposal on catalog-SIG.When a mirror is proposed on the mailing list, it is manuallyadded in a mirror list in the PyPI application after ithas been checked to be compliant with the mirroring rules.
The mirror list is provided as a list of host names of theform
X.pypi.python.org
The values of X are the sequence a,b,c,…,aa,ab,…a.pypi.python.org is the master server; the mirrors startwith b. A CNAME record last.pypi.python.org points to thelast host name. Mirror operators should use a static address,and report planned changes to that address in advance todistutils-sig.
The new mirror also appears athttp://pypi.python.org/mirrorswhich is a human-readable page that gives the list of mirrors.This page also explains how to register a new mirror.
PyPI provides statistics on downloads at/stats. This page iscalculated daily by PyPI, by reading all mirrors’ local stats andsumming them.
The stats are presented in daily or monthly files, under/stats/daysand/stats/months. Each file is abzip2 file with these formats:
Examples:
With a distributed mirroring system, clients may want to verify thatthe mirrored copies are authentic. There are multiple threats toconsider:
This specification only deals with the second threat. Some provisionsare made to detect man-in-the-middle attacks. To detect the firstattack, package authors need to sign their packages using PGP keys, sothat users verify that the package comes from the author they trust.
The central index provides a DSA key at the URL /serverkey, in the PEMformat as generated by “openssl dsa -pubout” (i.e.RFC 3280SubjectPublicKeyInfo, with the algorithm 1.3.14.3.2.12). This URL mustnot be mirrored, and clients must fetch the official serverkey fromPyPI directly, or use the copy that came with the PyPI client software.Mirrors should still download the key, to detect a key rollover.
For each package, a mirrored signature is provided at/serversig/<package>. This is the DSA signature of the parallel URL/simple/<package>, in DER form, using SHA-1 with DSA (i.e. as aRFC 3279 Dsa-Sig-Value, created by algorithm 1.2.840.10040.4.3)
Clients using a mirror need to perform the following steps to verifya package:
An implementation of the verification algorithm is available fromhttps://svn.python.org/packages/trunk/pypi/tools/verify.py
Verification is not needed when downloading from central index, andshould be avoided to reduce the computation overhead.
About once a year, the key will be replaced with a new one. Mirrorswill have to re-fetch all /serversig pages. Clients using mirrors needto find a trusted copy of the new server key. One way to obtain oneis to download it fromhttps://pypi.python.org/serverkey. To detectman-in-the-middle attacks, clients need to verify the SSL servercertificate, which will be signed by the CACert authority.
A mirror is a subset copy of PyPI, so it provides the same structureby copying it.
It also needs to provide two specific elements:
CPAN uses a freshness date system where the mirror’s lastsynchronisation date is made available.
For PyPI, each mirror needs to maintain a URL with simple text contentthat represents the last synchronisation date the mirror maintains.
The date is provided in GMT time, using the ISO 8601 format[2].Each mirror will be responsible to maintain its last modified date.
This page must be located at :/last-modified and must be atext/plain page.
Each mirror is responsible to count all the downloads that where donevia it. This is used by PyPI to sum up all downloads, to be able todisplay the grand total.
These statistics are in CSV-like form, with a header in the firstline. It needs to obeyPEP 305. Basically, it should bereadable by Python’scsv module.
The fields in this file are:
The content will look like this:
# package,filename,useragent,countzc.buildout,zc.buildout-1.6.0.tgz,MyAgent,142...
The counting starts the day the mirror is launched, and there is onefile per day, compressed using thebzip2 format. Each file is namedlike the day. For example,2008-11-06.bz2 is the file for the 6th ofNovember 2008.
They are then provided in a folder calleddays. For example:
This page must be located at/local-stats.
A mirroring protocol calledSimpleIndex was described andimplemented by Martin v. Loewis and Jim Fulton, based on howeasy_install works. This section synthesizes it and gives a fewrelevant links, plus a small part aboutUser-Agent.
Mirrors must reduce the amount of data transferred between the centralserver and the mirror. To achieve that, they MUST use the changelog()PyPI XML-RPC call, and only refetch the packages that have beenchanged since the last time. For each package P, they MUST copydocuments /simple/P/ and /serversig/P. If a package is deleted on thecentral server, they MUST delete the package and all associated files.To detect modification of package files, they MAY cache the file’sETag, and MAY request skipping it using the If-none-match header.
Each mirroring tool MUST identify itself using a descripte User-agentheader.
The pep381client package[1] provides an application thatrespects this protocol to browse PyPI.
In order to be able to differentiate actions taken by clients overPyPI, a specific user agent name should be provided by all mirroringsoftware.
This is also true for all clients like:
XXX user agent registering mechanism at PyPI ?
Clients that are browsing PyPI should be able to use alternativemirrors, by getting the list of the mirrors usinglast.pypi.python.org.
Code example:
>>>importsocket>>>socket.gethostbyname_ex('last.pypi.python.org')[0]'h.pypi.python.org'
The clients so far that could use this mechanism:
Clients that are browsing PyPI should be able to use a fail-overmechanism when PyPI or the used mirror is not responding.
It is up to the client to decide which mirror should be used, maybe bylooking at its geographical location and its responsiveness.
This PEP does not describe how this fail-over mechanism should work,but it is strongly encouraged that the clients try to use the nearestmirror.
The clients so far that could use this mechanism:
It is obvious that some packages will not be uploaded to PyPI, whetherbecause they are private or whether because the project maintainerruns their own server where people might get the project package.However, it is strongly encouraged that a public package index followsPyPI and Distutils protocols.
In other words, theregister andupload command should becompatible with any package index server out there.
Software that are compatible with PyPI and Distutils so far:
An extra package index is not a mirror of PyPI, but can have somemirrors itself.
When a client needs to get some packages from several distinctindexes, it should be able to use each one of them as a potentialsource of packages. Different indexes should be defined as a sortedlist for the client to look for a package.
Each independent index can of course provide a list of its mirrors.
XXX define how to get the hostname for the mirrors of an arbitraryindex.
That permits all combinations at client level, for a reliablepackaging system with all levels of privacy.
It is up the client to deal with the merging.
Georg Brandl.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0381.rst
Last modified:2025-02-01 08:55:40 GMT