Abstract

This PEP specifies how to normalizedistribution extranames when performing comparisons.This prevents tools from either failing to find an extra name, oraccidentally matching against an unexpected name.

Motivation

TheProvides-Extra core metadata specification states that an extra’sname “must be a valid Python identifier”.PEP 508 specifies that the value of anextra marker may contain aletter, digit, or any one of.,-, or_ after the initial character.There is no otherPyPA specificationwhich outlines how extra names should be written or normalized for comparison.Due to the amount of packaging-related code in existence,it is important to evaluate current practices by the community andstandardize on one that doesn’t break most existing code, while beingsomething tool authors can agree to following.

The issue of there being no consistent standard was brought forward by aninitial discussionnoting that the extraadhoc-ssl was not considered equal to the nameadhoc_ssl by pip 22.

Rationale

PEP 503 specifies how to normalize distribution names:

re.sub(r"[-_.]+","-",name).lower()

This collapses any run of the characters-,_ and.down to a single-.For example,---. and__ all get converted to just-.This doesnot produce a valid Python identifier, perthe core metadata 2.2 specification for extra names.

Setuptools 60 performs normalizationvia:

re.sub(r'[^A-Za-z0-9-.]+','_',name).lower()

The use of an underscore/_ differs from PEP 503’s use of a hyphen/-,and it also normalizes characters outside of those allowed byPEP 508.Runs of. and-, unlike PEP 503, donot get normalized to one_,e.g... stays the same. To note, this is inconsistent with this function’sdocstring, whichdoes specify that all non-alphanumeric characters(which would include- and.) are normalized and collapsed.

For pip 22, its“extra normalisation behaviour is quite convoluted and erratic”[pip-erratic]and so its use is not considered.

Specification

When comparing extra names, tools MUST normalize the names being comparedusing the semantics outlined inPEP 503for names:

re.sub(r"[-_.]+","-",name).lower()

Thecore metadata specification will be updated such that the allowednames forProvides-Extra matches whatPEP 508 specifies for names.This will bring extra naming in line with that of theName field.Because this changes what is considered valid, it will lead to a coremetadata version increase to2.3.

For tools writingcore metadata,they MUST write out extra names in their normalized form.This applies to theProvides-Extra field and theextra marker when used in theRequires-Dist field.

Tools generating metadata MUST raise an error if a user specifiedtwo or more extra names which would normalize to the same name.Tools generating metadata MUST raise an error if an invalid extraname is provided as appropriate for the specified core metadata version.If a project’s metadata specifies an older core metadata version andthe name would be invalid with newer core metadata versions,tools reading that metadata SHOULD warn the user.Tools SHOULD warn users when an invalid extra name is read and SHOULDignore the name to avoid ambiguity.Tools MAY raise an error instead of a warning when reading aninvalid name, if they so desire.

Backwards Compatibility

Moving toPEP 503 normalization andPEP 508 name acceptanceallows for all preexisting, valid names to continue to be valid.

Based on research looking at a collection of wheels on PyPI[pypi-results],the risk of extra name clashes is limited to 73 instances when consideringall extras names on PyPI, valid or not (not just those within a single package)whileonly looking at valid names leads to only 3 clashes:

• dev-test:dev_test,dev-test,dev.test
• dev-lint:dev-lint,dev.lint,dev_lint
• apache-beam:apache-beam,apache.beam

By requiring tools writing core metadata to only record the normalized name,the issue of preexisting, invalid extra names should diminish over time.

Security Implications

It is possible that for a distribution that has conflicting extra names, atool ends up installing dependencies that somehow weaken the securityof the system.This is only hypothetical and if it were to occur,it would probably be more of a security concern for the distributionsspecifying such extras names rather than the distribution that pulledthem in together.

How to Teach This

This should be transparent to users on a day-to-day basis.It will be up to tools to educate/stop users when they select extranames which conflict.

Reference Implementation

No reference implementation is provided aside from the code above,but the expectation is thepackaging project will provide afunction in itspackaging.utils module that will implement extra namenormalization.It will also implement extra name comparisons appropriately.Finally, if the project ever gains the ability to write out metadata,it will also implement this PEP.

Transition Plan

There is a risk that a build tool will produce core metadataconforming to version 2.3 and thus this PEP but which is consumed by atool that is unaware of this PEP (if that tool chooses to attempt toread a core metadata version it does not directly support).In such a case there is a chance that a user may specify an extrausing an non-normalized name which worked previously but which failsnow.

As such, consumers of this PEP should be prioritized more thanproducers so that users can be notified that they are specifying extranames which are not normalized (and thus may break in the future).

Rejected Ideas

Open Issues

N/A

Copyright

This document is placed in the public domain or under theCC0-1.0-Universal license, whichever is more permissive.