What stubtest does and does not do¶

Stubtest will import your code and introspect your code objects at runtime, forexample, by using the capabilities of theinspect module. Stubtestwill then analyse the stub files, and compare the two, pointing out things thatdiffer between stubs and the implementation at runtime.

It’s important to be aware of the limitations of this comparison. Stubtest willnot make any attempt to statically analyse your actual code and relies only ondynamic runtime introspection (in particular, this approach means stubtest workswell with extension modules). However, this means that stubtest has limitedvisibility; for instance, it cannot tell if a return type of a function isaccurately typed in the stubs.

For clarity, here are some additional things stubtest can’t do:

• Type check your code – usemypy instead

• Generate stubs – usestubgen orpyright--createstub instead

• Generate stubs based on running your application or test suite – usemonkeytype instead

• Apply stubs to code to produce inline types – useretype orlibcst instead

In summary, stubtest works very well for ensuring basic consistency betweenstubs and implementation or to check for stub completeness. It’s used totest Python’s official collection of library stubs,typeshed.

Example¶

Here’s a quick example of what stubtest can do:

$python3-mpipinstallmypy$catlibrary.pyx="hello, stubtest"deffoo(x=None):print(x)$catlibrary.pyix:intdeffoo(x:int)->None:...$python3-mmypy.stubtestlibraryerror:library.fooisinconsistent,runtimeargument"x"hasadefaultvaluebutstubargumentdoesnotStub:atline3def(x:builtins.int)Runtime:infile~/library.py:3def(x=None)error:library.xvariablediffersfromruntimetypeLiteral['hello, stubtest']Stub:atline1builtins.intRuntime:'hello, stubtest'

Usage¶

Running stubtest can be as simple asstubtestmodule_to_check.Runstubtest--help for a quick summary of options.

Stubtest must be able to import the code to be checked, so make sure that mypyis installed in the same environment as the library to be tested. In somecases, settingPYTHONPATH can help stubtest find the code to import.

Similarly, stubtest must be able to find the stubs to be checked. Stubtestrespects theMYPYPATH environment variable – consider using this if youreceive a complaint along the lines of “failed to find stubs”.

Note that stubtest requires mypy to be able to analyse stubs. If mypy is unableto analyse stubs, you may get an error on the lines of “not checking stubs dueto mypy build errors”. In this case, you will need to mitigate those errorsbefore stubtest will run. Despite potential overlap in errors here, stubtest isnot intended as a substitute for running mypy directly.

Allowlist¶

If you wish to ignore some of stubtest’s complaints, stubtest supports apretty handy--allowlist system.

Let’s say that you have this python module calledex:

try:importoptional_expensive_depexceptImportError:optional_expensive_dep=Nonefirst=1ifoptional_expensive_dep:second=2

Let’s say that you can’t installoptional_expensive_dep in CI for some reason,but you still want to includesecond:int in the stub file:

first:intsecond:int

In this case stubtest will correctly complain:

error:ex.secondisnotpresentatruntimeStub:infile/.../ex.pyi:2builtins.intRuntime:MISSINGFound1error(checked1module)

To fix this, you can add anallowlist entry:

# Allowlist entries in `allowlist.txt` file:# Does not exist if `optional_expensive_dep` is not installed:ex.second

And now when running stubtest with--allowlist=allowlist.txt,no errors will be generated anymore.

Allowlists also support regular expressions,which can be useful to ignore many similar errors at once.They can also be useful for suppressing stubtest errors that occur sometimes,but not on every CI run. For example, if some CI workers haveoptional_expensive_dep installed, stubtest might complain with this messageon those workers if you had theex.second allowlist entry:

note:unused allowlist entry ex.secondFound 1 error (checked 1 module)

Changingex.second to be(ex\.second)? will make this error optional,meaning that stubtest will pass whether or not a CI runnerhas``optional_expensive_dep`` installed.

CLI¶

The rest of this section documents the command line interface of stubtest.

--concise¶

Makes stubtest’s output more concise, one line per error

--ignore-missing-stub¶

Ignore errors for stub missing things that are present at runtime

--ignore-positional-only¶

Ignore errors for whether an argument should or shouldn’t be positional-only

--allowlistFILE¶

Use file as an allowlist. Can be passed multiple times to combine multipleallowlists. Allowlists can be created with--generate-allowlist.Allowlists support regular expressions.

The presence of an entry in the allowlist means stubtest will not generateany errors for the corresponding definition.

--generate-allowlist¶

Print an allowlist (to stdout) to be used with--allowlist.

When introducing stubtest to an existing project, this is an easy way tosilence all existing errors.

--ignore-unused-allowlist¶

Ignore unused allowlist entries

Without this option enabled, the default is for stubtest to complain if anallowlist entry is not necessary for stubtest to pass successfully.

Note if an allowlist entry is a regex that matches the empty string,stubtest will never consider it unused. For example, to get--ignore-unused-allowlist behaviour for a single allowlist entry likefoo.bar you could add an allowlist entry(foo\.bar)?.This can be useful when an error only occurs on a specific platform.

--mypy-config-fileFILE¶

Use specified mypy configfile to determine mypy plugins and mypy path

--custom-typeshed-dirDIR¶

Use the custom typeshed inDIR

--check-typeshed¶

Check all stdlib modules in typeshed

--help¶

Show a help message :-)