Repository files navigation

Atox plugin that generates a JSON version of your tox.ini envlist,which can be used in a GitHub workflowbuild matrix(or potentially with other CI systems).

This plugin is useful when:

• Your tox.ini envlist covers a complex set of factors(e.g., all supported combinations of Django and Python versions).
• You use GitHub actions and want to run each tox testenvin a separate workflow job, so that tests run in parallel and soGitHub's actions log breaks out the result for each testenv.
• You're tired of manually syncing your workflow build matrix andyour tox.ini envlist.

tox-gh-matrix adds a newtox --gh-matrix command line option thatoutputs a JSON representation of your tox envlist:

[{"name":"py35-django22","factors":["py35","django22"],"python":{"version":"3.5","spec":"3.5.0-alpha - 3.5"}},{"name":"py36-django22","factors":["py36","django22"],"python":{"version":"3.6","spec":"3.6.0-alpha - 3.6"}},// ...{"name":"py310-django40","factors":["py310","django40"],"python":{"version":"3.10","spec":"3.10.0-alpha - 3.10","installed":"3.10.2"}},{"name":"py311-django40","factors":["py311","django40"],"python":{"version":"3.11","spec":"3.11.0-alpha - 3.11"},"ignore_outcome":true},// ...{"name":"docs","factors":["docs"]},{"name":"lint","factors":["lint"]}]

Your workflow can use this to define a build matrix from the tox envlist:

jobs:get-envlist:outputs:envlist:${{ steps.generate-envlist.outputs.envlist }}steps:# ... (details omitted; see complete example below)      -id:generate-envlistrun:python -m tox --gh-matrixtest:needs:get-envliststrategy:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.envlist) }}steps:# ... (details omitted; see complete example below)      -uses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}      -run:python -m tox -e ${{ matrix.tox.name }}

See theusage section below for a complete annotated exampleand other variations.

• Usage
  • Complete example
  • Installation
  • Handling ignore outcome
  • Using setup-python
  • Testing PyPy and older Python versions
  • Filtering the tox envlist
  • Examining tox factors
  • Matrix output names and multiple envlists
  • Additional build matrix dimensions
  • Debugging the matrix
• Contributing, issues, help
• Similar projects
• License

The basic approach to running GitHub workflow jobs based onyour tox envlist is:

1. Runtox --gh-matrix in a preliminary job, to generate aJSON version of your tox envlist.

2. In your main test job, define a workflowbuild matrixproperty that iterates that list, usingfromJSON().

Here's a complete, annotated example workflow:

name:teston:pushjobs:# First, use tox-gh-matrix to construct a build matrix# from your tox.ini:get-envlist:runs-on:ubuntu-latest# Make the JSON envlist available to the test job:outputs:envlist:${{ steps.generate-envlist.outputs.envlist }}steps:# Checkout project code to get tox.ini:      -uses:actions/checkout@v3# Install tox and tox-gh-matrix:      -run:python -m pip install tox tox-gh-matrix# Run `tox --gh-matrix` to generate the JSON list:      -id:generate-envlistrun:python -m tox --gh-matrix# Now run your tests using that matrix:test:# Pull in the JSON generated in the previous job:needs:get-envliststrategy:# Define a build matrix property `tox` that iterates# the envlist:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.envlist) }}# Run all matrix jobs, even if some fail:fail-fast:false# The workflow treats everything below as a template# to run a separate job for each build matrix item.name:Test ${{ matrix.tox.name }}runs-on:ubuntu-lateststeps:      -uses:actions/checkout@v3# Install the required Python version if necessary:      -name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}# Install tox (you don't need tox-gh-matrix at this point):      -run:python -m pip install tox# Run `tox -e {name}` to test the single tox environment# for this matrix entry:      -run:python -m tox -e ${{ matrix.tox.name }}

Some other variations are covered below. Also, see this project'sownworkflow definition.

Install tox-gh-matrix from PyPI using pip:

python -m pip install tox-gh-matrix

(Typically you'd do this in a GitHub workflow as shown above,but that's not required. You can run tox-gh-matrix in your localdevelopment environment to examine its output.)

tox-gh-matrix requires Python 3.6 or later and tox 3.5.12 or later.(It is not currently compatible with tox 4 alpha.)

The minimum Python version only applies to running thetox --gh-matrixcommand itself. (tox can generate virtual environments withany versionof Python to run your tests.)

If you use tox'signore_outcome setting toallow failures in certain testenvs, those jobs will always appearsuccessful in GitHub's actions log.

You may prefer to hoist the failure handling up to the workflow level,so you can see which environments have failed in the actions log.

tox-gh-matrix adds"ignore_outcome": true to each matrixitem where your tox.ini specifies that option. You can check thisin the workflow'scontinue-on-error job step setting.

You'll also need prevent tox fromactually ignoring failures duringthose workflow runs. Tox doesn't have a built-in way to "ignoreignore_outcome", but we can simulate it with an environment variable.This example calls itTOX_OVERRIDE_IGNORE_OUTCOME (but the exact namedoesn't matter).

First, update your workflow to set the workflow'scontinue-on-errorfrom the matrix and set the environment variable tofalse (meaningwe want tox to pretendignore_outcome = false everywhere, regardlessof what tox.ini says):

jobs:test:steps:# ...      -run:python -m tox -e ${{ matrix.tox.name }}continue-on-error:${{ matrix.tox.ignore_outcome == true }}env:TOX_OVERRIDE_IGNORE_OUTCOME:false

(Only add this variable in thetest job, not theget-envlist job.)

Then, in your tox.ini change everyignore_outcome = true to use theenvironment variable (using tox'senvironment variable substitutionsyntax with a default value oftrue):

[testenv:experimental]ignore_outcome = {env:TOX_OVERRIDE_IGNORE_OUTCOME:true}# This also works with factor-conditional settings:[testenv]ignore_outcome =djangoDev = {env:TOX_OVERRIDE_IGNORE_OUTCOME:true}py322 = {env:TOX_OVERRIDE_IGNORE_OUTCOME:true}

Now when tox is run from the GitHub workflow, itwon't ignore failures,so the workflow will catch and report them (and then continue). But whenyou run tox locally (or anywhere the environment variable isn't set),toxwill ignore failures in those testenvs.

The tox-gh-matrix JSON includespython objects providing versiondata that works with GitHub'sactions/setup-python.

For example, if your tox.ini hasenvlist = py36,py38,docs,thetox --gh-matrix JSON might look something like this:

[{"name":"py36","python":{"version":"3.6","spec":"3.6.0-alpha - 3.6",},},{"name":"py38","python":{"version":"3.8","spec":"3.8.0-alpha - 3.8","installed":"3.8.10"},},{"name":"docs"}]

Thepython field is only present if the tox testenv calls fora specific Python version–either implied with apy* factor orexplicitly via the testenv'sbasepythonsetting. (So in this example, there's nopython fieldfor the "docs" testenv. If you wanted a specific Python there,you could add something likebasepython = py310 in yourtox.ini's[testenv:docs] section .)

When present,python is an object with two or three fields:

• python.version is a Python version compatible with setup-python'spython-version parameter. E.g.,"3.10" or"2.7" or"pypy-3.8".
• python.spec is a versionrange specifier, also compatiblewith setup-python'spython-version parameter, which allowspre-release versions. E.g.,"3.22.0-alpha - 3.22".
• python.installed is only present if tox found a compatiblePython version already available on the runner instance. If so, it isthe actual version reported by that interpreter. (This is useful forskipping setup-python if tox can already find a compatible interpreteron the runner.)

Putting this all together, the recommended way to call setup-pythonfor most workflows is:

steps:      -name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}

If you don't want to use pre-release Python interpreters, changepython.spec topython.version in both places.

If yourget-envlist job (that runstox --gh-matrix) has a differentruns-on runner type than thetest job (tox -e ${{matrix.tox.name}}),you will have different Python versions available on your test runners.In that case, you should ignorepython.installed change the checkto justif: matrix.tox.python.spec.

If your tox envlist includes PyPy or outdated Python versions, you may needan extra build step to ensure tox runs under a Python version it supports.

For example, say your tox envlist includespy34. Tox dropped Python 3.4support in 2019. It can still generate a Python 3.4 virtualenv and runtests in it, but you must run tox itself on a newer version of Python.

To make that work, runactions/setup-python twice: once to installthe (possibly old) version of Python needed for the test environment,and a second time to change back to a newer version of Python to run tox:

steps:      -uses:actions/checkout@v3# Install the required Python version if necessary:      -name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}# Now restore the default Python to something newer for tox:      -name:Restore modern Pythonuses:actions/setup-python@v4with:python-version:"3.8"# Install and run tox with that newer Python:      -run:python -m pip install tox      -run:python -m tox -e ${{ matrix.tox.name }}

You can choose anypython-version that makes sense for the second setup-python.(If you use the version that comes pre-installed on your runner platform,it will be nearly instantaneous because there's nothing to install;it's just changing some paths to alter the default.)

Your tox envlist may include environments you don't want to testin your workflow. You can either restrict the envlist whenyou calltox --gh-matrix to generate it, or you can use workflowconditionals to skip jobs based on tox factors or other information.

By default, tox-gh-matrix includes your entire tox.inienvlistin its JSON output. You can limit this with tox command line optionsor environment variables thatfilter the envlist,such as-e envlist,TOXENV orTOX_SKIP_ENV.

For example, if you wanted the matrix to omit all tox testenvscontainingwin ormac, you could use:

get-envlist:steps:# ...      -id:generate-envlistenv:# (TOX_SKIP_ENV is a Python regular expression)TOX_SKIP_ENV:".*(win|mac).*"run:python -m tox --gh-matrix

tox-gh-matrix should also work with other tox plugins thatmanipulate the envlist, such astox-factor andtox-envlist.

The tox-gh-matrix JSON includes a list of toxfactors for each toxenvironment. You can use this with GitHub workflowconditional executionto skip or include steps for certain factors.

For example, you might useif: contains(matrix.tox.factors, "pre")to only execute a particular job step for tox environments containing a "pre"factor. Contrast that withcontains(matrix.tox.name, "pre")which would do something similar but also match environments containingfactors like "prep" or "present", which may or may not be what you want.

(factors is a list of strings;name is a single string. In workflowexpression syntax,matrix.tox.name is equivalent tojoin(matrix.tox.factors, '-').)

Runningtox --gh-matrix sets a GitHub workflowoutput parameterto the JSON build matrix. (It appends to theGITHUB_OUTPUT file;you'll get an error if that environment variable isn't set properly.)

The default output name isenvlist, but you change this withtox --gh-matrix=VAR.You can use this to (in combination with filtering) to create multiple matrices.

Here's an example that uses custom output names, along with thetox-factorfiltering plugin, to construct separate matrices for Mac- and Windows specifictests (environments withmac orwin factors, respectively):

jobs:get-envlist:runs-on:ubuntu-latestoutputs:mac-envlist:${{ steps.generate-envlist.outputs.mac-envlist }}win-envlist:${{ steps.generate-envlist.outputs.win-envlist }}steps:      -uses:actions/checkout@v3# Also install the tox-factor plugin:      -run:python -m pip install tox tox-factor tox-gh-matrix# Run --gh-matrix twice with different filters and output names:      -id:generate-envlistrun:|          python -m tox -f mac --gh-matrix=mac-envlist          python -m tox -f win --gh-matrix=win-envlisttest-mac:runs-on:macos-latestneeds:get-envliststrategy:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.mac-envlist) }}# ...test-win:runs-on:windows-latestneeds:get-envliststrategy:matrix:tox:${{ fromJSON(needs.get-envlist.outputs.win-envlist) }}# ...

Your workflow can define additionalbuild matrixproperties alongside the tox envlist. GitHub will run all combinationsof properties.

For example, your workflow could repeat the entire tox envliston macOS, Windows, and Ubuntu by adding in anos property:

jobs:# ...test:strategy:matrix:os:[macos-latest, windows-latest, ubuntu-latest]tox:${{ fromJSON(needs.get-envlist.outputs.envlist) }}name:Test ${{ matrix.tox.name }} on ${{ matrix.os }}runs-on:${{ matrix.os }}steps:      -uses:actions/checkout@v3      -name:Setup Python ${{ matrix.tox.python.version }}if:matrix.tox.python.spec && ! matrix.tox.python.installeduses:actions/setup-python@v4with:python-version:${{ matrix.tox.python.spec }}      -run:python -m pip install tox      -run:python -m tox -e ${{ matrix.tox.name }}

Runtox --gh-matrix-dump to display a nicely formatted (multiline,indented) JSON build matrix, without any GitHub-specific output parametersyntax.

This can be helpful for debugging the generated matrix (either run in yourlocal development environment, or as a step in yourget-envlist job).

It could also be useful for integrating tox with other (non-GitHub) CI systems.

(Or perhaps you were interested indebuggingThe Matrix ?)

Contributions of all types are very welcome, including bug reports, fixes,documentation corrections and improvements, and new features.

If you have any questions or need help with tox-gh-matrix, pleaseask in the discussions.

If you encounter any problems, pleasefile an issuealong with as much detail as possible to help reproduce the problem.

For bug fixes and other code changes, tests can be run withtox (naturally).We try to keep test coverage high before merging new code, but please don'tlet incomplete tests keep you from opening a PR. (We'll be happy to work withyou to add tests, etc.)

To propose a new feature, it's often helpful to open adiscussion beforeinvesting significant time or effort in code.

Other tox + GitHub integrations tend to take the opposite approach:you fully declare the build matrix in your GitHub workflow definition,and the plugin then simplifies running the correct tox environment(s)for each matrix job.

• tox-gh-actions detects which tox environments to run based on thecurrent active Python version, platform, environment variables, and othercontext, with flexible mapping to tox factors. It also improves integrationwith GitHub's actions logging.
• tox-gh is a newer project with goals similar to tox-gh-actions,but some different design philosophies.

Distributed under the terms of the MIT License, tox-gh-matrix isfree and open source software.