Code Coverage for ROS

This is day 3 of my 2020 National Robotics Week blog marathon!

About two years ago I created a little package called code_coverage. This package is a bit of CMake which makes it easier to run coverage testing on your ROS packages. Initially it only supported C++, but recently it has been expanded to cover Python code as well.

What is Code Coverage?

Before I get into how to use the code_coverage package, let’s discuss what coverage testing is all about. We all know it is important to have tests for your code so that it does not break as you implement new features and inevitably refactor code. Coverage testing tells you what parts of your code your tests actually test. This can help you find branch paths or even entire modules of the code that are not properly tested. It can also help you know if new code is actually getting tested.

The output of a coverage test is generally some really nice webpages that show you line-by-line what code is getting executed during the test:

Using code_coverage for C++

We will start by discussing the usage of code_coverage with C++ code first, because it is actually quite a bit simpler. C++ coverage can be done almost entirely in CMake.

First, update your package.xml to have a test_depend on code_coverage package.

Next, we need to update two places in the CMakeLists.txt file. The first change should be right after you call to catkin_package. The second change is where you define your test targets. You need to define a new target, which we will typically call {package_name}_coverage_report

# After catkin_package()

if(CATKIN_ENABLE_TESTING AND ENABLE_COVERAGE_TESTING)
  find_package(code_coverage REQUIRED)
  # Add compiler flags for coverage instrumentation before defining any targets
  APPEND_COVERAGE_COMPILER_FLAGS()
endif()

# Add your targets here

if (CATKIN_ENABLE_TESTING)
  # Add your tests here

  # Create a target ${PROJECT_NAME}_coverage_report
  if(ENABLE_COVERAGE_TESTING)
    set(COVERAGE_EXCLUDES "*/${PROJECT_NAME}/test*" "*/${PROJECT_NAME}/other_dir_i_dont_care_about*")
    add_code_coverage(
      NAME ${PROJECT_NAME}_coverage_report
      DEPENDENCIES tests
    )
  endif()
endif()

That’s the configuration needed. Now we can compile the code (with coverage turned on) and run the coverage report (which in turn will run the tests):

catkin_make -DENABLE_COVERAGE_TESTING=ON -DCMAKE_BUILD_TYPE=Debug PACKAGE_NAME_coverage_report

You can find these same instructions (and how to use catkin tools) in the code_coverage README.

Using code_coverage for Python

Python unit tests will automatically get coverage turned on just with the CMake configuration shown above, but Python-based rostests (those that are launched in a launch file) need some extra configuration.

First, we need to turn on coverage testing in each node using the launch-prefix. You can decide on a node-by-node basis which nodes should actually generate coverage information:

<launch>

    <!-- Add an argument to the launch file to turn on coverage -->
    <arg name="coverage" default="false"/>

    <!-- This fancy line forces nodes to generate coverage -->
    <arg name="pythontest_launch_prefix" value="$(eval 'python-coverage run -p' if arg('coverage') else '')"/>

    <!-- This node will NOT generate coverage information -->
    <node pkg="example_pkg" name="publisher_node" type="publisher_node.py" />

    <!-- But this node WILL generate coverage -->
    <node pkg="example_pkg" name="subscriber_node" type="subscriber_node.py"
          launch-prefix="$(arg pythontest_launch_prefix)" />

    <!-- The test can also generate coverage information if you include the launch-prefix -->
    <test time-limit="10" test-name="sample_rostest" pkg="example_pkg" type="sample_rostest.py"
          launch-prefix="$(arg pythontest_launch_prefix)" />

</launch>

Then we turn on coverage by adding the argument in our CMakeLists.txt:

add_rostest(example_rostest.test ARGS coverage:=ENABLE_COVERAGE_TESTING)

You can find this full Python example from my co-worker Survy Vaish on GitHub.

Using codecov.io For Visualization

codecov.io is a cloud-based solution for visualizing the output of your coverage testing. It can combine all of the reports from individual packages, as well as the C++ and Python reports into some nice graphs and track results over multiple commits:

codecov.io dashboard for robot_calibration
A Full Working Example

The robot_calibration package use code_coverage, codecov.io, and Travis-CI to run code coverage testing on every pull request and commit to master branch. It uses the popular industrial-ci package as the base line and then the following changes are made: