Repository files navigation

Read the Trail of Bits blog post about how we find bugs withabi3audit!

abi3audit scans Python extensions forabi3 violations and inconsistencies.

It can scan individual (unpackaged) shared objects, packaged wheels, or entirepackage version histories.

This project is maintained in part byTrail of Bits.This is not an official Trail of Bits product.

• Motivation
• Installation
• Usage
  • Examples
• Limitations
• Licensing

CPython (the reference implementation of Python) defines a stable API and correspondingABI ("abi3"). In principle, any CPython extension can be built against thisAPI/ABI and will remain forward compatible with future minor versions of CPython.In other words: if you build against the stable ABI for Python 3.5, yourextension should work without modification on Python 3.9.

The stable ABI simplifies packaging of CPython extensions, since the packageronly needs to build oneabi3 wheel that targets the minimum supported Pythonversion.

To signal that a Python wheel containsabi3-compatible extensions,the Python packaging ecosystem uses theabi3 wheel tag, e.g.:

pyrage-1.0.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl

Unfortunately, there isno actual enforcement ofabi3 compliancein Python extensions at install or runtime: a wheel (or independentshared object) that is tagged asabi3 is assumed to beabi3, butis not validated in any way.

To make matters worse, there isno formal connection between the flag(--py-limited-api)that controls wheel tagging and the build macros(Py_LIMITED_API)that actually lock a Python extension into a specificabi3 version.

As a result: it is very easy to compile a Python extension for the wrongabi3version, or to tag a Python wheel asabi3 without actually compiling itasabi3-compatible.

This has serious security and reliability implications: non-stable partsof the CPython ABI can change between minor versions, resulting in crashes,unpredictable behavior, or potentially exploitable memory corruption whena Python extension incorrectly assumes the parameters of a functionor layout of a structure.

abi3audit is available viapip:

pip install abi3audit

You can runabi3audit as a standalone program, or viapython -m abi3audit:

abi3audit --helppython -m abi3audit --help

Top-level:

usage: abi3audit [-h] [-V] [--debug] [-v] [-R] [-o OUTPUT] [-s] [-S]                 [--assume-minimum-abi3 ASSUME_MINIMUM_ABI3]                 SPEC [SPEC ...]Scans Python extensions for abi3 violations and inconsistenciespositional arguments:  SPEC                  the files or other dependency specs to scanoptions:  -h, --help            show this help message and exit  -V, --version         show program's version number and exit  --debug               emit debug statements; this setting also overrides                        `ABI3AUDIT_LOGLEVEL` and is equivalent to setting it                        to `debug`  -v, --verbose         give more output, including pretty-printed results for                        each audit step  -R, --report          generate a JSON report; uses --output  -o OUTPUT, --output OUTPUT                        the path to write the JSON report to (default: stdout)  -s, --summary         always output a summary even if there are no                        violations/ABI version mismatches  -S, --strict          fail the entire audit if an individual audit step                        fails  --assume-minimum-abi3 ASSUME_MINIMUM_ABI3                        assumed abi3 version (3.x, with x>=2) if it cannot be                        detected

Audit a single shared object, wheel, or PyPI package:

# audit a local copy of an abi3 extensionabi3audit procmaps.abi3.so# audit a local copy of an abi3 wheelabi3audit procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl# audit every abi3 wheel for the package 'procmaps' on PyPIabi3audit procmaps

Show additional detail (pretty tables and individual violations) while auditing:

abi3audit procmaps --verbose

yields:

[17:59:46] 👎 procmaps:           procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl: procmaps.abi3.so           uses the Python 3.10 ABI, but is tagged for the Python 3.6 ABI           ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓           ┃ Symbol                  ┃ Version ┃           ┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩           │ PyUnicode_AsUTF8AndSize │ 3.10    │           └─────────────────────────┴─────────┘[17:59:47] 💁 procmaps: 2 extensions scanned; 1 ABI version mismatches and 0           ABI violations found

Generate a JSON report for each input:

abi3audit procmaps --report| python -m json.tool

yields:

{"specs": {"procmaps": {"kind":"package","package": {"procmaps-0.5.0-cp36-abi3-manylinux2010_x86_64.whl": [          {"name":"procmaps.abi3.so","result": {"is_abi3":true,"is_abi3_baseline_compatible":false,"baseline":"3.6","computed":"3.10","non_abi3_symbols": [],"future_abi3_objects": {"PyUnicode_AsUTF8AndSize":"3.10"              }            }          }        ],"procmaps-0.6.1-cp37-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl": [          {"name":"procmaps.abi3.so","result": {"is_abi3":true,"is_abi3_baseline_compatible":true,"baseline":"3.7","computed":"3.7","non_abi3_symbols": [],"future_abi3_objects": {}            }          }        ]      }    }  }}

abi3audit is abest-effort tool, with some of the same limitations asauditwheel. In particular:

• abi3audit cannot check fordynamic abi3 violations, such as an extensionthat callsdlsym(3)to invoke a non-abi3 function at runtime.

• abi3audit can confirm the presence of abi3-compatible symbols, but doesnot have an exhaustive list of abi3-incompatible symbols. Instead, it looksfor violations by looking for symbols that start withPy_ or_Py_ thatare not in the abi3 compatibility list. This isunlikely to result in falsepositives, butcould if an extension incorrectly uses those reservedprefixes.

• When auditing a "bare" shared object (e.g.foo.abi3.so),abi3audit cannotassume anything about the minimumintended abi3 version. Instead, itdefaults to the lowest known abi3 version (abi3-cp32) and warns on anyversion mismatches (e.g., a symbol that was only stabilized in 3.6).This can result in false positives, so users are encouraged to audit entirewheels or packages instead (since they contain the sufficient metadata).

• abi3audit considers the abi3 version when a symbol wasstabilized,notintroduced. In other words:abi3audit will produce a warningwhen anabi3-cp36 extension contains a function stabilized in 3.7, evenif that function was introduced in 3.6. This isnot a false positive(it is an ABI version mismatch), but it'sgenerally not a source of bugs.

• abi3audit checks both the "local" and "external" symbols for each extension,for formats that support both. It does this to catch symbols that have beeninlined, such as_Py_DECREF. However, if the extension's symbol tablehas been stripped, these may be missed.

abi3audit is licensed under the MIT license.

abi3audit includes ASN.1 and Mach-O parsers generated fromdefinitions provided by theKaitai Struct project.These vendored parsers are licensed by the Kaitai Struct authors under the MITlicense.