Repository files navigation

Code coverage for Ruby

• Source Code
• API documentation
• Changelog
• Rubygem
• Continuous Integration

SimpleCov is a code coverage analysis tool for Ruby. It usesRuby's built-in Coverage library to gather codecoverage data, but makes processing its results much easier by providing a clean API to filter, group, merge, format,and display those results, giving you a complete code coverage suite that can be set up with just a couple lines ofcode.SimpleCov/Coverage track covered ruby code, gathering coverage for common templating solutions like erb, slim and haml is not supported.

In most cases, you'll want overall coverage results for your projects, including all types of tests, Cucumber features,etc. SimpleCov automatically takes care of this by caching and merging results when generating reports, so yourreport actually includes coverage across your test suites and thereby gives you a better picture of blank spots.

The official formatter of SimpleCov is packaged as a separate gem calledsimplecov-html, but will be installed andconfigured automatically when you launch SimpleCov. If you're curious, you can find iton GitHub, too.

Code and Bug Reports

• Issue Tracker
• SeeCONTRIBUTING for how to contribute alongwith some common problems to check out before creating an issue.

Questions, Problems, Suggestions, etc.

• Mailing List "Open mailing list for discussion and announcementson Google Groups"

1. Add SimpleCov to yourGemfile andbundle install:

   gem'simplecov',require:false,group::test

2. Load and launch SimpleCovat the very top of yourtest/test_helper.rb(orspec_helper.rb,rails_helper, cucumberenv.rb, or whatever your preferred testframework uses):

   require'simplecov'SimpleCov.start# Previous content of test helper now starts here

   Note: If SimpleCov starts after your application code is already loaded(viarequire), it won't be able to track your files and their coverage!TheSimpleCov.startmust be issuedbefore any of your applicationcode is required!

   This is especially true if you use anything that keeps your tests application loaded like spring, check out thespring section.

   SimpleCov must be running in the process that you want the code coverageanalysis to happen on. When testing a server process (e.g. a JSON APIendpoint) via a separate test process (e.g. when using Selenium) where youwant to see all code executed by therails server, and not just codeexecuted in your actual test files, you need to require SimpleCov in theserver process. For rails for instance, you'll want to add something like thisto the top ofbin/rails, but below the "shebang" line (#! /usr/bin/env ruby) and after config/boot is required:

   ifENV['RAILS_ENV'] =='test'require'simplecov'SimpleCov.start'rails'puts"required simplecov"end

3. Run your full test suite to see the percent coverage that your application has.

4. After running your tests, opencoverage/index.html in the browser of your choice. For example, in a Mac Terminal,run the following command from your application's root directory:

   open coverage/index.html

   in a debian/ubuntu Terminal,

   xdg-open coverage/index.html

   Note:This guide can help if you're unsure which command your particularoperating system requires.

5. Add the following to your.gitignore file to ensure that coverage resultsare not tracked by Git (optional):

   echo coverage >> .gitignore

   If you're making a Rails application, SimpleCov comes with built-in configurations (see below for information onprofiles) that will get you started with groups for your Controllers, Models and Helpers. To use it, thefirst two lines of your test_helper should be like this:

   require'simplecov'SimpleCov.start'rails'

Coverage results report, fully browsable locally with sorting and much more:

Source file coverage details view:

Similarly to the usage with Test::Unit described above, the only thing you have to do is to add the SimpleCovconfig to the very top of your Cucumber/RSpec/whatever setup file.

Add the setup code to thetop offeatures/support/env.rb (for Cucumber) orspec/spec_helper.rb (for RSpec).Other test frameworks should work accordingly, whatever their setup file may be:

require'simplecov'SimpleCov.start'rails'

You could even track what kind of code your UI testers are touching if you want to go overboard with things. SimpleCovdoes not care what kind of framework it is running in; it just looks at what code is being executed and generates areport about it.

For some frameworks and testing tools there are quirks and problems you might want to know about if you wantto use SimpleCov with them. Here's an overview of the known ones:

Configuration settings can be applied in three formats, which are completely equivalent:

• The most common way is to configure it directly in your start block:

  SimpleCov.startdosome_config_option'foo'end

• You can also set all configuration options directly:

  SimpleCov.some_config_option'foo'

• If you do not want to start coverage immediately after launch or want to add additional configuration later on in aconcise way, use:

  SimpleCov.configuredosome_config_option'foo'end

Please check out theConfiguration API documentation to find out what you can customize.

If you use SimpleCov to merge multiple test suite results (e.g. Test/Unit and Cucumber) into a single report, you'dnormally have to set up all your config options twice, once intest_helper.rb and once inenv.rb.

To avoid this, you can place a file called.simplecov in your project root. You can then just leave therequire 'simplecov' in each test setup helper (at the top) and move theSimpleCov.start code with all yourcustom config options into.simplecov:

# test/test_helper.rbrequire'simplecov'# features/support/env.rbrequire'simplecov'# .simplecovSimpleCov.start'rails'do# any custom configs like groups and filters can be here at a central placeend

Using.simplecov rather than separately requiring SimpleCov multiple times is recommended if you are merging multipletest frameworks like Cucumber and RSpec that rely on each other, as invoking SimpleCov multiple times can cause coverageinformation to be lost.

Add branch coverage measurement statistics to your results. Supported in CRuby versions 2.5+.

SimpleCov.startdoenable_coverage:branchend

Branch coverage is a feature introduced in Ruby 2.5 concerning itself with whether aparticular branch of a condition had been executed. Line coverage on the other handis only interested in whether a line of code has been executed.

This comes in handy for instance for one line conditionals:

number.odd? ?"odd" :"even"

In line coverage this line would always be marked as executed but you'd never know if bothconditions were met. Guard clauses have a similar story:

returnifnumber.odd?# more code

If all the code in that method was covered you'd never know if the guard clause was evertriggered! With line coverage as just evaluating the condition marks it as covered.

In the HTML report the lines of code will be annotated likebranch_type: hit_count:

• then: 2 - the then branch (of anif) was executed twice
• else: 0 - the else branch (of anif orcase) was never executed

Not that even if you don't declare anelse branch it will still show up in the coveragereports meaning that the condition of theif was not hit or that nowhen ofcasewas hit during the test runs.

Is branch coverage strictly better? No. Branch coverage really only concerns itself withconditionals - meaning coverage of sequential code is of no interest to it. A file withoutconditional logic will have no branch coverage data and SimpleCov will report 0 of 0branches covered as 100% (as everything that can be covered was covered).

Hence, we recommend looking at both metrics together. Branch coverage might also be a goodoverall metric to look at - while you might be missing only 10% of your lines that mightaccount for 50% of your branches for instance.

By default, the primary coverage type isline. To set the primary coverage to something else, use the following:

# or in configure SimpleCov.primary_coverage :branchSimpleCov.startdoenable_coverage:branchprimary_coverage:branchend

Primary coverage determines what will come first in all output, and the type of coverage to check if you don't specify the type of coverage when customizing exit behavior (SimpleCov.minimum_coverage 90).

Note that coverage must first be enabled for non-default coverage types.

You can measure coverage for code that is evaluated byKernel#eval. Supported in CRuby versions 3.2+.

SimpleCov.startdoenable_coverage_for_evalend

This is typically useful for ERB. SetERB#filename= to make it possible for SimpleCov to trace the original .erb source file.

Filters can be used to remove selected files from your coverage data. By default, a filter is applied that removes allfiles OUTSIDE of your project's root directory - otherwise you'd end up with billions of coverage reports for sourcefiles in the gems you are using.

You can define your own to remove things like configuration files, tests or whatever you don't need in your coveragereport.

You can currently define a filter using either a String or Regexp (that will then be Regexp-matched against each sourcefile's path), a block or by passing in your own Filter class.

SimpleCov.startdoadd_filter"/test/"end

This simple string filter will remove all files that match "/test/" in their path.

SimpleCov.startdoadd_filter%r{^/test/}end

This simple regex filter will remove all files that start with /test/ in their path.

SimpleCov.startdoadd_filterdo |source_file|source_file.lines.count <5endend

Block filters receive a SimpleCov::SourceFile instance and expect your block to return either true (if the file is to beremoved from the result) or false (if the result should be kept). Please check out the RDoc for SimpleCov::SourceFile tolearn about the methods available to you. In the above example, the filter will remove all files that have less than 5lines of code.

classLineFilter <SimpleCov::Filterdefmatches?(source_file)source_file.lines.count <filter_argumentendendSimpleCov.add_filterLineFilter.new(5)

Defining your own filters is pretty easy: Just inherit from SimpleCov::Filter and define a method'matches?(source_file)'. When running the filter, a true return value from this method will result in the removal of thegiven source_file. The filter_argument method is being set in the SimpleCov::Filter initialize method and thus is set to5 in this example.

SimpleCov.startdoproc=Proc.new{ |source_file|false}add_filter["string",/regex/,proc,LineFilter.new(5)]end

You can pass in an array containing any of the other filter types.

You can exclude code from the coverage report by wrapping it in# :nocov:.

# :nocov:defskip_this_methodnever_reachedend# :nocov:

The name of the token can be changed to your liking.Learn more about the nocov feature.

Note: You shouldn't have to use the nocov token to skip private methods that are being included in your coverage. Ifyou appropriately test the public interface of your classes and objects you should automatically get full coverage ofyour private methods.

By default, SimpleCov filters everything outside of theSimpleCov.root directory. However, sometimes you may wantto include coverage reports for things you include as a gem, for example a Rails Engine.

Here's an example by@lsaffie from#221that shows how you can achieve just that:

SimpleCov.start:railsdofilters.clear# This will remove the :root_filter and :bundler_filter that come via simplecov's defaultsadd_filterdo |src|    !(src.filename =~/^#{SimpleCov.root}/)unlesssrc.filename =~/my_engine/endend

You can separate your source files into groups. For example, in a Rails app, you'll want to have separate listings forModels, Controllers, Helpers, and Libs. Group definition works similarly to Filters (and also accepts customfilter classes), but source files end up in a group when the filter passes (returns true), as opposed to filteringresults, which exclude files from results when the filter results in a true value.

Add your groups with:

SimpleCov.startdoadd_group"Models","app/models"add_group"Controllers","app/controllers"add_group"Long files"do |src_file|src_file.lines.count >100endadd_group"Multiple Files",["app/models","app/controllers"]# You can also pass in an arrayadd_group"Short files",LineFilter.new(5)# Using the LineFilter class defined in Filters section aboveend

You normally want to have your coverage analyzed across ALL of your test suites, right?

Simplecov automatically caches coverage results in your(coverage_path)/.resultset.json, and will merge or override those withsubsequent runs, depending on whether simplecov considers those subsequent runsas different test suites or as the same test suite as the cached results. Tomake this distinction, simplecov has the concept of "test suite names".

SimpleCov tries to guess the name of the currently running test suite based upon the shell command the testsare running on. This should work fine for Unit Tests, RSpec, and Cucumber. If it fails, it will use the shellcommand that invoked the test suite as a command name.

If you have some non-standard setup and still want nicely labeled test suites, you have to give Simplecov acue as to what the name of the currently running test suite is. You can do so by specifyingSimpleCov.command_name in one test file that is part of your specific suite.

To customize the suite names on a Rails app (yeah, sorry for being Rails-biased, but everyone knows whatthe structure of those projects is. You can apply this accordingly to the RSpecs in yourOutlook-WebDAV-Calendar-Sync gem), you could do something like this:

# test/unit/some_test.rbSimpleCov.command_name'test:units'# test/functionals/some_controller_test.rbSimpleCov.command_name"test:functionals"# test/integration/some_integration_test.rbSimpleCov.command_name"test:integration"# features/support/env.rbSimpleCov.command_name"features"

Note that this only has to be invoked ONCE PER TEST SUITE, so even if you have 200 unit test files,specifying it insome_test.rb is enough.

Last but not leastif multiple suites resolve to the samecommand_name be aware that the coverage resultswillclobber each other instead of being merged. SimpleCov is smart enough to detect unique names for the most commonsetups, but if you have more than one test suite that doesn't follow a common pattern then you will want to manuallyensure that each suite gets a uniquecommand_name.

If you are running tests in parallel each process has the potential to clobber results from the other test processes.If you are relying on the defaultcommand_name then SimpleCov will attempt to detect and avoid parallel test suitecommand_name collisions based on the presence ofENV['PARALLEL_TEST_GROUPS'] andENV['TEST_ENV_NUMBER']. If yourparallel test runner does not set one or both of these thenyou must set acommand_name and ensure that it is uniqueper process (eg.command_name "Unit Tests PID #{$$}").

If you are using parallel_tests, you must incorporateTEST_ENV_NUMBER into the command name yourself, inorder for SimpleCov to merge the results correctly. For example:

# spec/spec_helper.rbSimpleCov.command_name"features" +(ENV['TEST_ENV_NUMBER'] ||'')

simplecov-html prints the used test suites in the footer of the generated coverage report.

Test results are automatically merged with previous runs in the same executionenvironment when generating the result, so when coverage is set up properly forCucumber and your unit / functional / integration tests, all of those testsuites will be taken into account when building the coverage report.

Of course, your cached coverage data is likely to become invalid at some point. Thus, when automatically mergingsubsequent test runs, result sets that are older thanSimpleCov.merge_timeout will not be used any more. By default,the timeout is 600 seconds (10 minutes), and you can raise (or lower) it by specifyingSimpleCov.merge_timeout 3600(1 hour), or, inside a configure/start block, with justmerge_timeout 3600.

You can deactivate this automatic merging altogether withSimpleCov.use_merging false.

If your tests are done in parallel across multiple build machines, you can fetch them all and merge them into a singleresult set using theSimpleCov.collate method. This can be added to a Rakefile or script file, having downloaded a set of.resultset.json files from each parallel test run.

# lib/tasks/coverage_report.rakenamespace:coveragedodesc"Collates all result sets generated by the different test runners"task:reportdorequire'simplecov'SimpleCov.collateDir["simplecov-resultset-*/.resultset.json"]endend

SimpleCov.collate also takes an optional simplecov profile and an optionalblock for configuration, just the same asSimpleCov.start orSimpleCov.configure. This means you can configure a separate formatter forthe collated output. For instance, you can make the formatter inSimpleCov.start theSimpleCov::Formatter::SimpleFormatter, and only use morecomplex formatters in the finalSimpleCov.collate run.

# spec/spec_helper.rbrequire'simplecov'SimpleCov.start'rails'do# Disambiguates individual test runscommand_name"Job#{ENV["TEST_ENV_NUMBER"]}"ifENV["TEST_ENV_NUMBER"]ifENV['CI']formatterSimpleCov::Formatter::SimpleFormatterelseformatterSimpleCov::Formatter::MultiFormatter.new([SimpleCov::Formatter::SimpleFormatter,SimpleCov::Formatter::HTMLFormatter])endtrack_files"**/*.rb"end

# lib/tasks/coverage_report.rakenamespace:coveragedotask:reportdorequire'simplecov'SimpleCov.collateDir["simplecov-resultset-*/.resultset.json"],'rails'doformatterSimpleCov::Formatter::MultiFormatter.new([SimpleCov::Formatter::SimpleFormatter,SimpleCov::Formatter::HTMLFormatter])endendend

SimpleCov.enable_for_subprocesses will allow SimpleCov to observe subprocesses starting usingProcess.fork.This modifies ruby's core Process.fork method so that SimpleCov can see into it, appending" (subprocess #{pid})"to theSimpleCov.command_name, with results that can be merged together using SimpleCov's merging feature.

To configure this, use.at_fork.

SimpleCov.enable_for_subprocessestrueSimpleCov.at_forkdo |pid|# This needs a unique name so it won't be overwrittenSimpleCov.command_name"#{SimpleCov.command_name} (subprocess:#{pid})"# be quiet, the parent process will be in charge of output and checking coverage totalsSimpleCov.print_error_status=falseSimpleCov.formatterSimpleCov::Formatter::SimpleFormatterSimpleCov.minimum_coverage0# startSimpleCov.startend

NOTE: SimpleCov must have already been started beforeProcess.fork was called.

Perhaps you're testing a ruby script withPTY.spawn orOpen3.popen, orProcess.spawn or etc.SimpleCov can cover this too.

Add a .simplecov_spawn.rb file to your project root

# .simplecov_spawn.rbrequire'simplecov'# this will also pick up whatever config is in .simplecov# so ensure it just contains configuration, and doesn't call SimpleCov.start.SimpleCov.command_name'spawn'# As this is not for a test runner directly, script doesn't have a pre-defined base command_nameSimpleCov.at_fork.call(Process.pid)# Use the per-process setup described previouslySimpleCov.start# only now can we start.

Then, instead of calling your script directly, like:

PTY.spawn('my_script.rb')do# ...

Use bin/ruby to require the new .simplecov_spawn file, then your script

PTY.spawn('ruby -r./.simplecov_spawn my_script.rb')do# ...

The Ruby STDLIB Coverage library that SimpleCov builds upon isvery fast (on a ~10 min Rails test suite, the speeddrop was only a couple seconds for me), and therefore it's SimpleCov's policy to just generate coverage every time yourun your tests because it doesn't do your test speed any harm and you're always equipped with the latest and greatestcoverage results.

Because of this, SimpleCov has no explicit built-in mechanism to run coverage only on demand.

However, you can still accomplish this very easily by introducing an ENV variable conditional into your SimpleCov setupblock, like this:

SimpleCov.startifENV["COVERAGE"]

Then, SimpleCov will only run if you execute your tests like this:

COVERAGE=true raketest

To aid in debugging issues, if an error is raised, SimpleCov will print a message toSTDERRwith the exit status of the error, like:

SimpleCov failed with exit 1

ThisSTDERR message can be disabled with:

SimpleCov.print_error_status = false

By default, SimpleCov's only config assumption is that you only want coverage reports for files inside your projectroot. To save yourself from repetitive configuration, you can use predefined blocks of configuration, called 'profiles',or define your own.

You can then pass the name of the profile to be used as the first argument to SimpleCov.start. For example, simplecovcomes bundled with a 'rails' profile. It looks somewhat like this:

SimpleCov.profiles.define'rails'doadd_filter'/test/'add_filter'/config/'add_group'Controllers','app/controllers'add_group'Models','app/models'add_group'Helpers','app/helpers'add_group'Libraries','lib'end

As you can see, it's just a SimpleCov.configure block. In your test_helper.rb, launch SimpleCov with:

SimpleCov.start'rails'

or

SimpleCov.start'rails'do# additional config hereend

You can load additional profiles with the SimpleCov.load_profile('xyz') method. This allows you to build upon anexisting profile and customize it so you can reuse it in unit tests and Cucumber features. For example:

# lib/simplecov_custom_profile.rbrequire'simplecov'SimpleCov.profiles.define'myprofile'doload_profile'rails'add_filter'vendor'# Don't include vendored stuffend# features/support/env.rbrequire'simplecov_custom_profile'SimpleCov.start'myprofile'# test/test_helper.rbrequire'simplecov_custom_profile'SimpleCov.start'myprofile'

You can define what SimpleCov should do when your test suite finishes by customizing the at_exit hook:

SimpleCov.at_exitdoSimpleCov.result.format!end

Above is the default behaviour. Do whatever you like instead!

You can define the minimum coverage percentage expected. SimpleCov will return non-zero if unmet.

SimpleCov.minimum_coverage90# same as above (the default is to check line coverage)SimpleCov.minimum_coverageline:90# check for a minimum line coverage of 90% and minimum 80% branch coverageSimpleCov.minimum_coverageline:90,branch:80

You can define the minimum coverage by file percentage expected. SimpleCov will return non-zero if unmet. This is usefulto help ensure coverage is relatively consistent, rather than being skewed by particularly good or bad areas of the code.

SimpleCov.minimum_coverage_by_file80# same as above (the default is to check line coverage by file)SimpleCov.minimum_coverage_by_fileline:80# check for a minimum line coverage by file of 90% and minimum 80% branch coverageSimpleCov.minimum_coverage_by_fileline:90,branch:80

You can define the maximum coverage drop percentage at once. SimpleCov will return non-zero if exceeded.

SimpleCov.maximum_coverage_drop5# same as above (the default is to check line drop)SimpleCov.maximum_coverage_dropline:5# check for a maximum line drop of 5% and maximum 10% branch dropSimpleCov.maximum_coverage_dropline:5,branch:10

You can also entirely refuse dropping coverage between test runs:

SimpleCov.refuse_coverage_drop# same as above (the default is to only refuse line drop)SimpleCov.refuse_coverage_drop:line# refuse drop for line and branchSimpleCov.refuse_coverage_drop:line,:branch

You can use your own formatter with:

SimpleCov.formatter=SimpleCov::Formatter::HTMLFormatter

CallingSimpleCov.result.format! will be invoked withSimpleCov::Formatter::YourFormatter.new.format(result),andresult is an instance ofSimpleCov::Result. Do whatever your wish with that!

As of SimpleCov 0.9, you can specify multiple result formats. Formatters besides the default HTML formatter require separate gems, however.

require"simplecov-html"SimpleCov.formatters=[SimpleCov::Formatter::HTMLFormatter,SimpleCov::Formatter::CSVFormatter,]

SimpleCov is packaged with a separate gem calledsimplecov_json_formatter that provides you with a JSON formatter, this formatter could be useful for different use cases, such as for CI consumption or for reporting to external services.

In order to use it you will need to manually load the installed gem like so:

require"simplecov_json_formatter"SimpleCov.formatter=SimpleCov::Formatter::JSONFormatter

Note: In case you plan to report your coverage results to CodeClimate services, know that SimpleCov will automatically use theJSON formatter along with the HTML formatter when theCC_TEST_REPORTER_ID variable is present in the environment.

• Open Source formatter and integration plugins for SimpleCov
• Editor Integration
• Hosted (commercial) services

SimpleCov is built inContinuous Integration on Ruby 2.5+ as well as JRuby 9.2+.

Note for JRuby => You need to pass JRUBY_OPTS="--debug" or create .jrubyrc and add debug.fullTrace=true

TryCoverband.

If you're usingSpring to speed up test suite runs and want to run SimpleCov alongwith them, you'll find that it often misreports coverage with the default config due to some sort of eager loadingissue. Don't despair!

One solution is toexplicitly call eagerloadin yourtest_helper.rb /spec_helper.rb after callingSimpleCov.start.

require'simplecov'SimpleCov.start'rails'Rails.application.eager_load!

Alternatively, you could disable Spring while running SimpleCov:

DISABLE_SPRING=1 rake test

Or you could removegem 'spring' from yourGemfile.

Themost common problem is that simplecov isn't required and started before everything else. In order to trackcoverage for your whole applicationsimplecov needs to be the first one so that it (and the underlying coveragelibrary) can subsequently track loaded files and their usage.

If you are missing coverage for some code a simple trick is to put a puts statement in there and right afterSimpleCov.start so you can see if the file really was loaded after simplecov was started.

# my_code.rbclassMyCodeputs"MyCode is being loaded!"defmy_method# ...endend# spec_helper.rb/rails_helper.rb/test_helper.rb/.simplecov whateverSimpleCov.startputs"SimpleCov started successfully!"

Now when you run your test suite and you see:

SimpleCov started successfully!MyCode is being loaded!

then it's good otherwise you likely have a problem :)

Everyone participating in this project's development, issue trackers and other channels is expected to follow ourCode of Conduct

See thecontributing guide.

Thanks to Aaron Patterson for the original idea for this!

Copyright (c) 2010-2017 Christoph Olszowka. See MIT-LICENSE for details.