- Notifications
You must be signed in to change notification settings - Fork65
Take Julia code coverage and memory allocation results, do useful things with them
License
JuliaCI/Coverage.jl
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
"Take Julia code coverage and memory allocation results, do useful things with them"
Code coverage: Julia can track how many times, if any, each line of your code is run. This is useful for measuring how much of your code base your tests actually test, and can reveal the parts of your code that are not tested and might be hiding a bug. You can use Coverage.jl to summarize the results of this tracking, or to send them to a service likeCoveralls.io orCodecov.io.
Memory allocation: Julia can track how much memory is allocated by each line of your code. This can reveal problems like type instability, or operations that you might have thought were cheap (in terms of memory allocated) but aren't (i.e. accidental copying).
- Coverage.jl (this package): allows you to take coverage results and submit them to online web services such as Codecov.io and Coveralls.io
- CoverageTools.jl: core functionality for processing code coverage and memory allocation results
Most users will want to useCoverage.jl.
Step 1: collect coverage data. If you are using your default test suite, you can collect coverage data withPkg.test("MyPkg"; coverage=true)
. Alternatively, you can collect coverage data manually: in the terminal, navigate to whatever directory you want to start from as the working directory, and run julia with the--code-coverage
option:
julia --code-coverage=user
or more comprehensively (if you're interested in getting coverage for Julia's standard libraries)
julia --code-coverage=tracefile-%p.info --code-coverage=user# available in Julia v1.1+
You can add other options (e.g.,--project
) as needed. After the REPL starts, execute whatever commands you wish, and then quit Julia. Coverage data are written to files when Julia exits.
Step 2: collect summary statistics (optional). Navigate to the top-level directory of your package, restart Julia (with no special flags) and analyze your code coverage:
using Coverage# process '*.cov' filescoverage=process_folder()# defaults to src/; alternatively, supply the folder name as argumentcoverage=append!(coverage,process_folder("deps"))# useful if you want to analyze more than just src/# process '*.info' files, if you collected themcoverage=merge_coverage_counts(coverage,filter!(let prefixes= (joinpath(pwd(),"src",""),joinpath(pwd(),"deps","")) c->any(p->startswith(c.filename, p), prefixes)end, LCOV.readfolder("test")))# Get total coverage for all Julia filescovered_lines, total_lines=get_summary(coverage)# Or process a single file@showget_summary(process_file(joinpath("src","MyPkg.jl")))
The fraction of total coverage is equal tocovered_lines/total_lines
.
Step 3: identify uncovered lines (optional). To discover which functions lack testing, browse through the*.cov
files in yoursrc/
directory and look for lines starting with-
or0
, which mark lines that were never executed.Numbers larger than 0 are counts of the number of times the respective line was executed.Note that blank lines, comments, lines withend
statements, etc. are marked with-
but do not count against your coverage.
Be aware of a few limitations:
- a line that can take one of two branches gets marked as covered even if only one branch is tested
- currently, code run by Julia's internal interpreteris not marked as covered.
To exclude specific code blocks, surround the section withCOV_EXCL_START
andCOV_EXCL_STOP
comments:
# COV_EXCL_STARTfoo()=nothing# COV_EXCL_STOP
To exclude a single line, add a comment withCOV_EXCL_LINE
:
const a=1# COV_EXCL_LINE
Start julia with
julia --track-allocation=user
Then:
- Run whatever commands you wish to test. This first run is to ensure that everything is compiled (because compilation allocates memory).
- Call
Profile.clear_malloc_data()
- Run your commands again
- Quit julia
Finally, navigate to the directory holding your source code. Start julia (without command-line flags), and analyze the results using
using Coverageanalyze_malloc(dirnames)# could be "." for the current directory, or "src", etc.
This will return a vector ofMallocInfo
objects, specifying the number of bytes allocated, the file name, and the line number.These are sorted in increasing order of allocation size.
There are many tools to work with LCOV info-format files as generated by thegeninfo
tool. Coverage.jl can generate these files:
coverage=process_folder()LCOV.writefile("coverage-lcov.info", coverage)
When using Coverage.jl locally, over time a lot of.cov
files can accumulate. Coverage.jl provides theclean_folder
andclean_file
methods to either clean up all.cov
files in a directory (and subdirectories) or only clean the.cov
files associated with a specific source file.
Tracking Coverage withCodecov.io
Codecov.io is a test coverage tracking tool that integrates with your continuous integration servers (e.g.TravisCI) or with HTTP POSTs from your very own computer at home.
EnableCodecov.io for your repository.
- If it is public on GitHub and you are using using Travis, CircleCI orAppveyor, this is all you need to do. You can sign into Codecov using yourGithub identity.
- Otherwise you will need to define a
CODECOV_TOKEN
environment variablewith the Repository Upload Token available under the Codecov settings.
Use the command line option when you run your tests:
- Either with something like
julia --code-coverage test/runtests.jl
, or - with something like
julia -e 'Pkg.test("MyPkg", coverage=true)'
- Either with something like
Configure your CI service to upload coverage data:
If you are using Travis with
language: julia
, simply addcodecov: true
to your.travis.yml
.You can also add the following to the end of your
.travis.yml
. Thisline downloads this package, collects the per-file coverage data, thenbundles it up and submits to Codecov. Coverage.jl assumes that theworking directory is the package directory, so it changes to that first(so don't forget to replaceMyPkg
with your package's name!On Travis CI:
after_success:-julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'
On AppVeyor:
after_test:-C:\projects\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Codecov.submit(process_folder())"
If you're running coverage on your own machine and want to upload resultsto Codecov, make a bash script like the following:
#!/bin/bashCODECOV_TOKEN=$YOUR_TOKEN_HERE julia -e'using Pkg; using Coverage; Codecov.submit_local(process_folder())'
Tracking Coverage withCoveralls.io
Coveralls.io is a test coverage tracking tool that integrates with your continuous integration solution (e.g.TravisCI).
EnableCoveralls.io for your repository. If it ispublic on GitHub and you are using TravisCI, this is all you need to do. Ifyou are using AppVeyor, you need to add a secure environment variablecalled
COVERALLS_TOKEN
to your.appveyor.yml
(seehere).Your repo token can be found in your Coveralls repo settings. If neither ofthese are true, please submit an issue, and we can work on addingadditional functionality for your use case.Activate the
--code-coverage
command line option when you run your tests- Either with something like
julia --code-coverage test/runtests.jl
, or - with something like
julia -e 'Pkg.test("MyPkg", coverage=true)'
- Either with something like
Configure your CI service to upload coverage data:
If you are using Travis with
language: julia
, simply addcoveralls: true
to your.travis.yml
.You can also add the following to the end of your
.travis.yml
. Thisline downloads this package, collects the per-file coverage data, thenbundles it up and submits to Coveralls. Coverage.jl assumes that theworking directory is the package directory, so it changes to that first(so don't forget to replaceMyPkg
with your package's name!On Travis CI:
after_success:-julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Coveralls.submit(process_folder())'
On AppVeyor:
after_test:-C:\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Coveralls.submit(process_folder())"
Coverage tracking in Julia is not yet quite perfect. One problem is that (atleast in certain scenarios), the coverage data emitted by Julia does not markfunctions which are never called (and thus are not covered) as code. Thus,they end up being treated like comments, and arenot counted as uncoveredcode, even though they clearly are. This can arbitrarily inflate coveragescores, and in the extreme case could even result in a project showing 100%coverage even though it contains not a single test.
To overcome this, Coverage.jl applies a workaround which ensures that alllines of code in all functions of your project are properly marked as "this iscode". This resolves the problem of over reporting coverage.
Unfortunately, this workaround itself can have negative consequences, and leadto under reporting coverage, for the following reason: when Julia compilescode with inlining and optimizations, it can happen that some lines of Juliacode do not correspond to any generated machine code; in that case, Julia'scode coverage tracking will never mark these lines as executed, and also won'tmark them as code. One may now argue whether this is a bug in itself or not,but that's how it is, and normally would be fine -- except that our workaroundnow does mark these lines as code, but code which now never has a chance asbeing marked as executed.
We may be able to improve our workaround to deal with this better in thefuture (see also#188), but thishas not yet been done and it is unclear whether it will take care of allinstances. Even better would be if Julia improved the coverage information itproduces to be on par with what e.g. C compilers like GCC and clang produce.Since it is unclear when or if any of these will happen, we have added anexpert option which allows Julia module owners to disable our workaround code,by setting the environment variableDISABLE_AMEND_COVERAGE_FROM_SRC
toyes
.
For Travis, this can be achieved by adding the following to.travis.yml
:
env: global: - DISABLE_AMEND_COVERAGE_FROM_SRC=yes
For AppVeyor, add this to.appveyor.yml
:
environment: DISABLE_AMEND_COVERAGE_FROM_SRC: yes
Pull requests to add your package welcome (or open an issue)
- ArgParse.jl
- AstroLib.jl
- AudioIO.jl
- Augur.jl
- Bootstrap.jl
- CAIRS.jl
- ClimateTools.jl
- DASSL.jl
- DataFrames.jl
- Decimals.jl
- Distributions.jl
- DSP.jl
- ExtractMacro.jl
- FastaIO.jl
- FiniteStateMachine.jl
- FourierFlows.jl
- Gadfly.jl
- GeometricalPredicates.jl
- Glob.jl
- GradientBoost.jl
- GraphCentrality.jl
- GraphLayout.jl
- Homebrew.jl
- HttpParser.jl
- IntervalTrees.jl
- IPNets.jl
- JointMoments.jl
- JuMP.jl
- LibGit2.jl
- LightGraphs.jl
- LinearExpressions.jl
- Orchestra.jl
- ODE.jl
- OnlineStats.jl
- OpenCL.jl
- OpenStreetMap.jl
- PValueAdjust.jl
- QuantEcon.jl
- QuantileRegression.jl
- RationalSimplex.jl
- RDF.jl
- Requests.jl
- Restful.jl
- Robotics.jl
- RouletteWheels.jl
- SASLib.jl
- SimJulia.jl
- SIUnits.jl
- StatsBase.jl
- TaylorIntegration.jl
- TaylorSeries.jl
- TextWrap.jl
- TimeData.jl
- TypeCheck.jl
- Unitful.jl
- URIParser.jl
- URITemplate.jl
- Voting.jl
- WAV.jl
- Weave.jl
- WeightedStats.jl
- YAML.jl
About
Take Julia code coverage and memory allocation results, do useful things with them
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.