Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Take Julia code coverage and memory allocation results, do useful things with them

License

NotificationsYou must be signed in to change notification settings

JuliaCI/Coverage.jl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build Statuscoverallscodecov

"Take Julia code coverage and memory allocation results, do useful things with them"

Code coverage: Julia can track how many times, if any, each line of your code is run. This is useful for measuring how much of your code base your tests actually test, and can reveal the parts of your code that are not tested and might be hiding a bug. You can use Coverage.jl to summarize the results of this tracking, or to send them to a service likeCoveralls.io orCodecov.io.

Memory allocation: Julia can track how much memory is allocated by each line of your code. This can reveal problems like type instability, or operations that you might have thought were cheap (in terms of memory allocated) but aren't (i.e. accidental copying).

Comparison of coverage packages

  • Coverage.jl (this package): allows you to take coverage results and submit them to online web services such as Codecov.io and Coveralls.io
  • CoverageTools.jl: core functionality for processing code coverage and memory allocation results

Most users will want to useCoverage.jl.

Working locally

Code coverage

Step 1: collect coverage data. If you are using your default test suite, you can collect coverage data withPkg.test("MyPkg"; coverage=true). Alternatively, you can collect coverage data manually: in the terminal, navigate to whatever directory you want to start from as the working directory, and run julia with the--code-coverage option:

julia --code-coverage=user

or more comprehensively (if you're interested in getting coverage for Julia's standard libraries)

julia --code-coverage=tracefile-%p.info --code-coverage=user# available in Julia v1.1+

You can add other options (e.g.,--project) as needed. After the REPL starts, execute whatever commands you wish, and then quit Julia. Coverage data are written to files when Julia exits.

Step 2: collect summary statistics (optional). Navigate to the top-level directory of your package, restart Julia (with no special flags) and analyze your code coverage:

using Coverage# process '*.cov' filescoverage=process_folder()# defaults to src/; alternatively, supply the folder name as argumentcoverage=append!(coverage,process_folder("deps"))# useful if you want to analyze more than just src/# process '*.info' files, if you collected themcoverage=merge_coverage_counts(coverage,filter!(let prefixes= (joinpath(pwd(),"src",""),joinpath(pwd(),"deps",""))        c->any(p->startswith(c.filename, p), prefixes)end,    LCOV.readfolder("test")))# Get total coverage for all Julia filescovered_lines, total_lines=get_summary(coverage)# Or process a single file@showget_summary(process_file(joinpath("src","MyPkg.jl")))

The fraction of total coverage is equal tocovered_lines/total_lines.

Step 3: identify uncovered lines (optional). To discover which functions lack testing, browse through the*.cov files in yoursrc/directory and look for lines starting with- or0, which mark lines that were never executed.Numbers larger than 0 are counts of the number of times the respective line was executed.Note that blank lines, comments, lines withend statements, etc. are marked with- but do not count against your coverage.

Be aware of a few limitations:

  • a line that can take one of two branches gets marked as covered even if only one branch is tested
  • currently, code run by Julia's internal interpreteris not marked as covered.

Exclude specific lines or sections from coverage

To exclude specific code blocks, surround the section withCOV_EXCL_START andCOV_EXCL_STOP comments:

# COV_EXCL_STARTfoo()=nothing# COV_EXCL_STOP

To exclude a single line, add a comment withCOV_EXCL_LINE:

const a=1# COV_EXCL_LINE

Memory allocation

Start julia with

julia --track-allocation=user

Then:

  • Run whatever commands you wish to test. This first run is to ensure that everything is compiled (because compilation allocates memory).
  • CallProfile.clear_malloc_data()
  • Run your commands again
  • Quit julia

Finally, navigate to the directory holding your source code. Start julia (without command-line flags), and analyze the results using

using Coverageanalyze_malloc(dirnames)# could be "." for the current directory, or "src", etc.

This will return a vector ofMallocInfo objects, specifying the number of bytes allocated, the file name, and the line number.These are sorted in increasing order of allocation size.

LCOV export

There are many tools to work with LCOV info-format files as generated by thegeninfo tool. Coverage.jl can generate these files:

coverage=process_folder()LCOV.writefile("coverage-lcov.info", coverage)

Cleaning up .cov files

When using Coverage.jl locally, over time a lot of.cov files can accumulate. Coverage.jl provides theclean_folder andclean_file methods to either clean up all.cov files in a directory (and subdirectories) or only clean the.cov files associated with a specific source file.

Tracking Coverage withCodecov.io

Codecov.io is a test coverage tracking tool that integrates with your continuous integration servers (e.g.TravisCI) or with HTTP POSTs from your very own computer at home.

  1. EnableCodecov.io for your repository.

    • If it is public on GitHub and you are using using Travis, CircleCI orAppveyor, this is all you need to do. You can sign into Codecov using yourGithub identity.
    • Otherwise you will need to define aCODECOV_TOKEN environment variablewith the Repository Upload Token available under the Codecov settings.
  2. Use the command line option when you run your tests:

    • Either with something likejulia --code-coverage test/runtests.jl, or
    • with something likejulia -e 'Pkg.test("MyPkg", coverage=true)'
  3. Configure your CI service to upload coverage data:

    • If you are using Travis withlanguage: julia, simply addcodecov: trueto your.travis.yml.

    • You can also add the following to the end of your.travis.yml. Thisline downloads this package, collects the per-file coverage data, thenbundles it up and submits to Codecov. Coverage.jl assumes that theworking directory is the package directory, so it changes to that first(so don't forget to replaceMyPkg with your package's name!

    • On Travis CI:

      after_success:-julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'
    • On AppVeyor:

      after_test:-C:\projects\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Codecov.submit(process_folder())"
    • If you're running coverage on your own machine and want to upload resultsto Codecov, make a bash script like the following:

      #!/bin/bashCODECOV_TOKEN=$YOUR_TOKEN_HERE julia -e'using Pkg; using Coverage; Codecov.submit_local(process_folder())'

Tracking Coverage withCoveralls.io

Coveralls.io is a test coverage tracking tool that integrates with your continuous integration solution (e.g.TravisCI).

  1. EnableCoveralls.io for your repository. If it ispublic on GitHub and you are using TravisCI, this is all you need to do. Ifyou are using AppVeyor, you need to add a secure environment variablecalledCOVERALLS_TOKEN to your.appveyor.yml (seehere).Your repo token can be found in your Coveralls repo settings. If neither ofthese are true, please submit an issue, and we can work on addingadditional functionality for your use case.

  2. Activate the--code-coverage command line option when you run your tests

    • Either with something likejulia --code-coverage test/runtests.jl, or
    • with something likejulia -e 'Pkg.test("MyPkg", coverage=true)'
  3. Configure your CI service to upload coverage data:

    • If you are using Travis withlanguage: julia, simply addcoveralls: trueto your.travis.yml.

    • You can also add the following to the end of your.travis.yml. Thisline downloads this package, collects the per-file coverage data, thenbundles it up and submits to Coveralls. Coverage.jl assumes that theworking directory is the package directory, so it changes to that first(so don't forget to replaceMyPkg with your package's name!

    • On Travis CI:

      after_success:-julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Coveralls.submit(process_folder())'
    • On AppVeyor:

      after_test:-C:\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Coveralls.submit(process_folder())"

A note for advanced users

Coverage tracking in Julia is not yet quite perfect. One problem is that (atleast in certain scenarios), the coverage data emitted by Julia does not markfunctions which are never called (and thus are not covered) as code. Thus,they end up being treated like comments, and arenot counted as uncoveredcode, even though they clearly are. This can arbitrarily inflate coveragescores, and in the extreme case could even result in a project showing 100%coverage even though it contains not a single test.

To overcome this, Coverage.jl applies a workaround which ensures that alllines of code in all functions of your project are properly marked as "this iscode". This resolves the problem of over reporting coverage.

Unfortunately, this workaround itself can have negative consequences, and leadto under reporting coverage, for the following reason: when Julia compilescode with inlining and optimizations, it can happen that some lines of Juliacode do not correspond to any generated machine code; in that case, Julia'scode coverage tracking will never mark these lines as executed, and also won'tmark them as code. One may now argue whether this is a bug in itself or not,but that's how it is, and normally would be fine -- except that our workaroundnow does mark these lines as code, but code which now never has a chance asbeing marked as executed.

We may be able to improve our workaround to deal with this better in thefuture (see also#188), but thishas not yet been done and it is unclear whether it will take care of allinstances. Even better would be if Julia improved the coverage information itproduces to be on par with what e.g. C compilers like GCC and clang produce.Since it is unclear when or if any of these will happen, we have added anexpert option which allows Julia module owners to disable our workaround code,by setting the environment variableDISABLE_AMEND_COVERAGE_FROM_SRC toyes.

For Travis, this can be achieved by adding the following to.travis.yml:

env:  global:    - DISABLE_AMEND_COVERAGE_FROM_SRC=yes

For AppVeyor, add this to.appveyor.yml:

environment:  DISABLE_AMEND_COVERAGE_FROM_SRC: yes

Some Julia packages using Coverage.jl

Pull requests to add your package welcome (or open an issue)

About

Take Julia code coverage and memory allocation results, do useful things with them

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages


[8]ページ先頭

©2009-2025 Movatter.jp