- Notifications
You must be signed in to change notification settings - Fork40
Publish coverage report as PR comment, and create a coverage badge & dashboard to display on the Readme for Python projects, all inside GitHub without third party servers
License
py-cov-action/python-coverage-comment-action
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
This action analyses the coverage data produced by the Pythoncoverage library and produces:
- A badge to display on your README
- A comment in the pull requests detailing how the PR impacts the coverage:
- Old and new coverage rates per file and total
- Coverage of new lines per file and total
- (optional) Annotations on lines missing coverage displayed in the PR directly
- AJob Summary
- A browsable folder containing:
- The coverage summary
- The full HTML coverage in a browsable format (not available for private repos)
All of this runs on top of GitHub Action without extra-charges and runs on the GitHubinfrastructure: your code isn't sent anywhere out of GitHub.
This action operates on an already generated.coverage file fromcoverage.
It has two main modes of operation:
On PRs, it will analyze the.coverage file, and produce a comment thatwill be posted to the PR. If a comment had already previously be written,it will be updated. The comment contains information on the evolutionof coverage rate attributed to this PR, as well as the rate of coveragefor lines that this PR introduces. There's also a small analysis for eachfile in a collapsed block.
This comment will also be output as ajob summary.
Seean example.
On repository's default branch, it will extract the coverage rate and createfiles that will be stored on a dedicated independent branch in your repository.
These files include:
- a
svgbadge to include in your README - a
jsonfile that can be used byshields.io if yourrepository is public to customize the look of your badge - Another
jsonfile used internally by the action to report on coverageevolution (does a PR make the coverage go up or down?) - A short file-by-file coverage report embedded directly into the branch's README. An excerpt from this is also output directly as ajob summary.
- The full HTML coverage report and links to make this report browsable
Seean example
Please ensure that your.coverage file(s) is created with the optionrelative_files = true.
Please ensure that the branchpython-coverage-comment-action-data is notprotected (there's no reason that it would be the case, except if you have veryspecific wildcard rules). If it is, either adjust your rules, or set theCOVERAGE_DATA_BRANCH parameter as described below. GitHub Actions will createthis branch with initial data at the first run if it doesn't exist, and willindependently commit to that branch after each commit to your default branch.
Once the action has run on your default branch, all the details for how to integrate thebadge to your Readme will be displayed in:
- The Readme of the
python-coverage-comment-action-databranch - The text output of the workflow run
The following snippet is targeted for cases where you expect PRs fromusers that don't have write access to the repository. Posting the commentis done in 2 steps:
- Checkout the repository and generate the comment to be posted. For securityreasons, we don't want to give permissions to a workflow that checks outuntrusted code
- From a trusted workflow, publish the comment on the PR
# .github/workflows/ci.ymlname:CIon:pull_request:push:branches: -"main"jobs:test:name:Run tests & display coverageruns-on:ubuntu-latestpermissions:# Gives the action the necessary permissions for publishing new# comments in pull requests.pull-requests:write# Gives the action the necessary permissions for pushing data to the# python-coverage-comment-action branch, and for editing existing# comments (to avoid publishing multiple comments in the same PR)contents:writesteps: -uses:actions/checkout@v4 -name:Install everything, run the tests, produce the .coverage filerun:make test# This is the part where you put your own test command -name:Coverage commentid:coverage_commentuses:py-cov-action/python-coverage-comment-action@v3with:GITHUB_TOKEN:${{ github.token }} -name:Store Pull Request comment to be posteduses:actions/upload-artifact@v4if:steps.coverage_comment.outputs.COMMENT_FILE_WRITTEN == 'true'with:# If you use a different name, update COMMENT_ARTIFACT_NAME accordinglyname:python-coverage-comment-action# If you use a different name, update COMMENT_FILENAME accordinglypath:python-coverage-comment-action.txt
# .github/workflows/coverage.ymlname:Post coverage commenton:workflow_run:workflows:["CI"]types: -completedjobs:test:name:Run tests & display coverageruns-on:ubuntu-latestif:github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'permissions:# Gives the action the necessary permissions for publishing new# comments in pull requests.pull-requests:write# Gives the action the necessary permissions for editing existing# comments (to avoid publishing multiple comments in the same PR)contents:write# Gives the action the necessary permissions for looking up the# workflow that launched this workflow, and download the related# artifact that contains the comment to be publishedactions:readsteps:# DO NOT run actions/checkout here, for security reasons# For details, refer to https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ -name:Post commentuses:py-cov-action/python-coverage-comment-action@v3with:GITHUB_TOKEN:${{ secrets.GITHUB_TOKEN }}GITHUB_PR_RUN_ID:${{ github.event.workflow_run.id }}# Update those if you changed the default values:# COMMENT_ARTIFACT_NAME: python-coverage-comment-action# COMMENT_FILENAME: python-coverage-comment-action.txt
If you don't expect external contributors, you don't need all the shenaniganswith the artifacts and the 2nd workflow. This is likely to be the most straightforwardway to configure it for private repositories. It might look like this:
# .github/workflows/ci.ymlname:CIon:pull_request:push:branches: -"main"jobs:test:name:Run tests & display coverageruns-on:ubuntu-latestpermissions:# Gives the action the necessary permissions for publishing new# comments in pull requests.pull-requests:write# Gives the action the necessary permissions for pushing data to the# python-coverage-comment-action branch, and for editing existing# comments (to avoid publishing multiple comments in the same PR)contents:writesteps: -uses:actions/checkout@v4 -name:Install everything, run the tests, produce the .coverage filerun:make test# This is the part where you put your own test command -name:Coverage commentuses:py-cov-action/python-coverage-comment-action@v3with:GITHUB_TOKEN:${{ github.token }}
In case you have a job matrix and you want the report to be on the globalcoverage, you can configure yourci.yml like this (coverage.yml remains thesame)
name:CIon:pull_request:push:branches: -"master"tags: -"*"jobs:build:strategy:matrix:include: -python_version:"3.7" -python_version:"3.8" -python_version:"3.9" -python_version:"3.10"name:"Python ${{ matrix.python_version }}"runs-on:ubuntu-lateststeps: -uses:actions/checkout@v4 -name:Set up Pythonid:setup-pythonuses:actions/setup-python@v4with:python-version:${{ matrix.python_version }} -name:Install everything, run the tests, produce a .coverage.xxx filerun:make test# This is the part where you put your own test commandenv:COVERAGE_FILE:".coverage.${{ matrix.python_version }}"# The file name prefix must be ".coverage." for "coverage combine"# enabled by "MERGE_COVERAGE_FILES: true" to work. A "subprocess"# error with the message "No data to combine" will be triggered if# this prefix is not used. -name:Store coverage fileuses:actions/upload-artifact@v4with:name:coverage-${{ matrix.python_version }}path:.coverage.${{ matrix.python_version }}# By default hidden files/folders (i.e. starting with .) are ignored.# You may prefer (for security reasons) not setting this and instead# set COVERAGE_FILE above to not start with a `.`, but you cannot# use "MERGE_COVERAGE_FILES: true" later on and need to manually# combine the coverage file using "pipx run coverage combine"include-hidden-files:truecoverage:name:Coverageruns-on:ubuntu-latestneeds:buildpermissions:pull-requests:writecontents:writesteps: -uses:actions/checkout@v4 -uses:actions/download-artifact@v4id:downloadwith:pattern:coverage-*merge-multiple:true -name:Coverage commentid:coverage_commentuses:py-cov-action/python-coverage-comment-action@v3with:GITHUB_TOKEN:${{ secrets.GITHUB_TOKEN }}MERGE_COVERAGE_FILES:true -name:Store Pull Request comment to be posteduses:actions/upload-artifact@v4if:steps.coverage_comment.outputs.COMMENT_FILE_WRITTEN == 'true'with:name:python-coverage-comment-actionpath:python-coverage-comment-action.txt
-name:Display coverageid:coverage_commentuses:py-cov-action/python-coverage-comment-action@v3with:GITHUB_TOKEN:${{ github.token }}# Change this in case you use GitHub Entreprise with a different API endpointGITHUB_BASE_URL:https://api.github.com# Only necessary in the "workflow_run" workflow.GITHUB_PR_RUN_ID:${{ inputs.GITHUB_PR_RUN_ID }}# Use this in case the folder to run coverage commands from is not the# top level of your repositoryCOVERAGE_PATH:my_project/# If the coverage percentage is above or equal to this value, the badge will be green.MINIMUM_GREEN:100# Same with orange. Below is red.MINIMUM_ORANGE:70# Maximum number of files to display in the comment. If there are more# files than this number, they will only appear in the workflow summary.# The selected files are the ones with the most new uncovered lines. The# closer this number gets to 35, the higher the risk that it reaches# GitHub's maximum comment size limit of 65536 characters. If you want# more files, you may need to use a custom comment template (see below).# (Feel free to open an issue.)MAX_FILES_IN_COMMENT:25# If true, will run `coverage combine` before reading the `.coverage` file.MERGE_COVERAGE_FILES:false# If true, will create an annotation on every line with missing coverage on a pull request.ANNOTATE_MISSING_LINES:false# Only needed if ANNOTATE_MISSING_LINES is set to true. This parameter allows you to choose between# notice, warning and error as annotation type. For more information look here:# https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-a-notice-messageANNOTATION_TYPE:warning# Name of the artifact in which the body of the comment to post on the PR is stored.# You typically don't have to change this unless you're already using this name for something else.COMMENT_ARTIFACT_NAME:python-coverage-comment-action# Name of the file in which the body of the comment to post on the PR is stored.# In monorepo setting, see SUBPROJECT_ID.COMMENT_FILENAME:python-coverage-comment-action.txt# This setting is only necessary if you plan to run the action multiple times# in the same repository. It will be appended to the value of all the# settings that need to be unique, so as for the action to avoid mixing# up results of multiple runs.# Affects `COMMENT_FILENAME`, `COVERAGE_DATA_BRANCH`.# Ideally, use dashes (`-`) rather than underscrores (`_`) to split words,# for consistencySUBPROJECT_ID:null / "lib-name"# An alternative template for the comment for pull requests. See details below.COMMENT_TEMPLATE:The coverage rate is `{{ coverage.info.percent_covered | pct }}`{{ marker }}# Name of the branch in which coverage data will be stored on the repository.# Default is 'python-coverage-comment-action-data'. Please make sure that this# branch is not protected.# In monorepo setting, see SUBPROJECT_ID.COVERAGE_DATA_BRANCH:python-coverage-comment-action-data# Deprecated, see https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/enabling-debug-loggingVERBOSE:false
This action's PR comments with coverage reports is designed to work whenrunning on thepull_request events. That being said, if your CI is running onfeature branches on thepush events and not on thepull_request events, wepartly support a mode where the action can comment on the PR when running onthepush events instead. This is most likely only useful for setups notaccepting external PRs and you will not have the best user experience.If that's something you need to do, please have a look atthis issue.
By default, comments are generated from aJinja template that you can readhere.
If you want to change this template, you can setCOMMENT_TEMPLATE. This isan advanced usage, so you're likely to run into more road bumps.
You will need to follow some rules for your template to be valid:
- Your template needs to be syntactically correct with Jinja2 rules
- You may define a new template from scratch, but in this case you are requiredto include
{{ marker }}, which includes an HTML comment (invisible onGitHub) that the action uses to identify its own comments. - If you'd rather want to change parts of the default template, you can do soby starting your comment with
{% extends "base" %}, and then override theblocks ({% block foo %}) that you wish to change. If you're unsure how itworks, seethe Jinjadocumentation - In either case, you will most likely want to get yourself familiar with theavailable context variables, the best is to read the code fromhere.Should those variables change, we'll do our best to bump the action's major version.
In the first example, we change the emoji that illustrates coverage going down from:down_arrow: to:sob::
{%extends"base"%}{%blockemoji_coverage_down%}:sob:{%endblockemoji_coverage_down%}
In this second example, we replace the whole comment by something much shorter with thecoverage (percentage) of the whole project from the PR build:
"Coverage: {{ coverage.info.percent_covered | pct }}{{ marker }}"In case you want to use the action multiple times with different parts of yoursource (so you have multiple codebases into a single repo), you'llneed to use SUBPROJECT_ID with a different value for each launch. You maystill use the same step for storing all files as artifacts. You'll end up witha different comment for each launch. Feel free to use theCOMMENT_TEMPLATE ifyou want each comment to clearly state what it relates to.
# .github/workflows/ci.ymlname:CIon:pull_request:push:branches: -"main"jobs:test:name:Run tests & display coverageruns-on:ubuntu-latestpermissions:pull-requests:writecontents:writesteps: -uses:actions/checkout@v3 -name:Test project 1run:make -C project_1 test -name:Test project 2run:make -C project_2 test -name:Coverage comment (project 1)id:coverage_comment_1uses:py-cov-action/python-coverage-comment-action@v3with:COVERAGE_PATH:project_1SUBPROJECT_ID:project-1GITHUB_TOKEN:${{ github.token }} -name:Coverage comment (project 2)id:coverage_comment_2uses:py-cov-action/python-coverage-comment-action@v3with:COVERAGE_PATH:project_2/srcSUBPROJECT_ID:project-2GITHUB_TOKEN:${{ github.token }} -name:Store Pull Request comment to be posteduses:actions/upload-artifact@v4if:steps.coverage_comment_1.outputs.COMMENT_FILE_WRITTEN == 'true' || steps.coverage_comment_2.outputs.COMMENT_FILE_WRITTEN == 'true'with:name:python-coverage-comment-action# Note the starpath:python-coverage-comment-action*.txt
# .github/workflows/coverage.ymlname:Post coverage commenton:workflow_run:workflows:["CI"]types: -completedjobs:test:name:Run tests & display coverageruns-on:ubuntu-latestif:github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'permissions:pull-requests:writecontents:writeactions:readsteps: -name:Post commentuses:py-cov-action/python-coverage-comment-action@v3with:GITHUB_TOKEN:${{ secrets.GITHUB_TOKEN }}GITHUB_PR_RUN_ID:${{ github.event.workflow_run.id }}SUBPROJECT_ID:project-1COVERAGE_PATH:project_1 -name:Post commentuses:py-cov-action/python-coverage-comment-action@v3with:GITHUB_TOKEN:${{ secrets.GITHUB_TOKEN }}GITHUB_PR_RUN_ID:${{ github.event.workflow_run.id }}SUBPROJECT_ID:project-2COVERAGE_PATH:project_2/src
On the examples above, the version was set to the tagv3. Pinning to a major versionwill give you the latest release on this version. (Note that we release every time aftera PR is merged). Pinning to a specific version (v3.1 for example) would make theaction more reproducible, though you'd have to update it regularly (e.g. usingDependabot). You can also pin a commit hash if you want to be 100% sure of what you run,given that tags are mutable. Finally, You can also decide to pin to main, if you're OKwith the action maybe breaking when (if) we release a v4.
This action is tested with 100% coverage. That said, coverage isn't all, andthere may be a lot of remaining issues :)
We accept Pull Requests (for bug fixes and previously-discussed features), and bugreports. For feature requests, this might depend on how much time we have on our handsat the moment, and how well you manage to sell it but don't get your hopes too high.
Initially, the first iteration of this action was using the more genericcoverage.xml (Cobertura) in order to be language independent. It was laterdiscovered that this format is very badly specified, as are mostly all coverageformats. For this reason, we switched to the much more specialized.coveragefile that is only produced for Python projects (also, the action was rewrittenfrom the ground up). Because this would likely completely break compatibility,a brand new action (this action) was created.
You can find the (unmaintained) language-generic versionhere.
Yes, I agree, this is annoying! The reason is that by default, coverage writesthe full path to the file in the.coverage file, but the path is most likelydifferent between the moment where your coverage is generated (in your workflow)and the moment where the report is computed (in the action, which runs inside adocker).
A previous version of this action did things with the wiki. This is not the caseanymore.
If your project's coverage was built on Windows, you may get an error like:
CoverageWarning: Couldn't parse 'yourproject\__init__.py': No source for code: 'yourproject\__init__.py'. (couldnt-parse)This is likely due to coverage being confused with the coverage being computed with\ but read with/. You can most probably fix it with the following in yourcoverage configuration:
[paths]source = */project/module *\project\moduleThis action is supposedly compatible with private repository. Just make sureto use the svg badge directly, and not theshields.io URL.
This action should be compatible with GitHub Enterprise. Just make sure to set theGITHUB_BASE_URL input to your GHE URL.
When upgrading, we change the location and format where the coveragedata is kept. Pull request that have not been re-based may be displayingslightly wrong information.
Starting with 3.19, the format for the Pull Request changed to a tablewith badges. We've been iterating a lot on the new format.It's perfectly ok if you preferred the old format. In that case, see#335 for instructions on how to emulate the old format usingCOMMENT_TEMPLATE.
About
Publish coverage report as PR comment, and create a coverage badge & dashboard to display on the Readme for Python projects, all inside GitHub without third party servers
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Uh oh!
There was an error while loading.Please reload this page.