To catch regressions in our code base, we run thousands of tests on all the packages in the ROS distributions. The tests range from small unit tests, to large simulation based integration tests. Our Jenkins server manages and schedules many of the tests. Below you find more details on each class of test that we run.


Development Tests

What gets tested?

The development tests will build and test all packages from the development branch of each repository. The development branch is specified in the source sections of the distribution.yaml rosdistro file (One from ROS Indigo | ROS Jade | ROS Kinetic). The devel branch is chosen by the maintainer of a repository, and is often trunk/default/master or <DISTRO>-devel. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Currently the development tests are only run for Ubuntu LTS releases for amd64 due to limited resources on the build farm.

When do tests run?

Every time you commit to a development branch the corresponding development tests will get triggered within an hour. It might take a while until the job is processed if the build farm is busy though. You can find the development jobs for Indigo on Jenkins and for Jade on Jenkins.

Who gets a notification?

The person that did the commit will receive an email if the build succeeds or fails. If the development job is marked as unstable (due to warnings or failing tests) all committers since the last stable build get notified.

How long does it take to run them?

One test typically takes less than 10 minutes. If the build farm is busy rebuilding Debian packages it might take longer until the job actually gets started.

Pull Request Tests

Pull request tests are identical to development tests but they are being triggered for the branches of pull requests. Currently only GitHub repositories are supported. Commonly pull requests need to be specifically enabled (see REP 143). You can find the pull request jobs for Indigo on Jenkins and for Jade on Jenkins.

Prerelease Tests

Before pushing out a new release of your package(s), you should run the prerelease tests on your repository. This will help you to avoid breaking the ROS Debian builds for the entire ROS community, and having to push out a bugfix release five minutes later. Starting with ROS Jade, prereleases are run locally on your machine using docker. There is a webpage to help you configure the command line tool that runs the prerelease tests on your machine:

What gets tested?

For the selected repositories (you can select one or multiple) all packages will be built and the tests will be run. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Additionally, it can download, build, and test packages which depend on your packages. The prerelease website can help you to figure out which settings will test the right amount of packages for you. You can control the number of packages that will be tested by adjusting level of recursive dependent packages. You can additionally specify arbitrary additional packages to test. If you want to avoid building a package, for example a package which is known to be broken for another reason, you can also explicitly exclude individual packages, though this might prevent the testing of other packages. The prerelease website can help you see how different settings affect the packages that will be tested.

Which repositories can be tested?

  • For repositories of catkin packages, you can choose between multiple versions:

    • version number: a specific released version of the repo/stack which is fetched from the release repository as listed in the distribution.yaml rosdistro file

    • latest: the latest released version of the repo/stack which is fetched from the release repository as listed in the distribution.yaml rosdistro file. While the distribution.yaml file might not contain a version the latest released version from the GBP repository will be used.

    • devel: the development branch which is fetched from the source repository as listed in the same file. If the source rosdistro file does not contain a version for the repository the default branch is used. Note that this file is manually generated and not part of the automatic bloom release process - you will need to seed it for your repositories with a pull request.

    • All dependent repos/stacks will always be fetched from their released version as listed in the rosdistro file
    • Here is an example "repository" entry in the distribution.yaml rosdistro file:

      •    1   rviz:  # <-- this is the repository's name
           2     doc:  # <-- this describes what to checkout for doc generation
           3       type: git
           4       url:
           5       version: hydro-devel
           6     release:  # <-- this is the "release" entry
           7       tags:
           8         release: release/hydro/{package}/{version}
           9       url:
          10       # ^-- url to the release repository
          11       version: 1.10.18-0  # <-- maps to "latest"
          12     source:  # <-- this is the "source" entry
          13     # ^-- maps to the "devel" prerelease option
          14       type: git
          15       url:
          16       version: hydro-devel
          17     status: maintained

When do tests run?

The prerelease tests only run when you run them on your local machine.

How long does it take to run them?

Low level repositories can take more than an hour to test, while high level repositories can be finished in 5 minutes. The test duration mainly depends on how many repositories depend on the repositories being tested. You can use the prerelease website to help you adjust the number of packages that are tested and built to avoid extremely long builds due to a large number of dependent packages being tested.

How do I setup my system to run a prerelease?

First you need to install docker, we recommend following the guide at because you'll need a newer version of docker than what comes with Ubuntu 14.04:

NB: Make sure to follow the directions to add your user to the docker group so you can invoke docker without sudo.

So that you can install our tools, you'll need to setup apt-get sources to include, you won't need to do this step if you've installed ROS before:

  • $ sudo sh -c 'echo "deb $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
    $ sudo apt-key adv --keyserver hkp:// --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
    $ sudo apt-get update

Then you can install the ros_buildfarm package to get the scripts and tools you need to run the prerelease:

  • $ sudo apt-get install python3-ros-buildfarm

You can also install the Python 2 equivalent which is called python-ros-buildfarm. For the Python 2 package the template evaluation of the prerelease scripts still use Python 3 internally. Therefore you'll need to have EmPy for Python 3 installed. The Debian package is named python3-empy is available as of Trusty. Otherwise or on other platforms it can be installed with the following commands:

  • $ sudo apt-get install python3 python3-pip
    $ sudo python3 -m pip install -U EmPy

At the end of the prerelease it will instruct you to run catkin_test_results to get the result of the prerelease. In order to run that command you need catkin installed and sourced. So you'll want to install catkin now if you haven't already, for example:

  • $ sudo apt-get install ros-jade-catkin

Then you need to source the setup.bash file at some point before you try to use catkin_test_results:

  • $ source /opt/ros/jade/setup.bash

You can do this at any point, rest assured that your system and environment will not affect the prerelease since they are executed inside the docker container.

Running prerelease test

Once you've set up your computer, use the prerelease website to help you generate a command to run the prerelease:

The website will prompt you to select a ROS distribution, select which repositories to run the prerelease on, some additional configurations, and finally what packages to test on top of the repositories you've selected for the prerelease.

Then you'll have a couple of command lines to run in your terminal. These will run the prerelease locally on your machine. Once it is done you'll be prompted to run catkin_test_results in order to summarize the results. If you have trouble running this, make sure you've installed catkin and sourced the appropriate setup.bash file.

Below a few common examples are described. The ros_buildfarm documentation contains additional information and examples on prereleases:

A set of commands the website provides

The command set provided by the prerelease test website above would look like the following. Variables that you can configure on the website are surrounded with %%.

mkdir -p /tmp/prerelease_job
cd /tmp/prerelease_job \ \
  %ROS_DISTRO% default ubuntu %UBUNTU_DISTRO% amd64 \
  --level %DOWNSTREAM_DEPTH% \
  --output-dir ./
  • A concrete example:
    mkdir -p /tmp/prerelease_job
    cd /tmp/prerelease_job \ \
      kinetic default ubuntu xenial amd64 \
      moveit \
      --level 0 \
      --output-dir ./
    In this example, we're testing:
  • %ROS_DISTRO%: ROS Kinetic
  • %UBUNTU_DISTRO%: Ubuntu xenial

  • %PACKAGE_TARGETED%: moveit

  • %DOWNSTREAM_DEPTH%: 0 (i.e. we're not testing if the tests in the downstream packages pass. Depending on the number of packages, the downstream testing can take long)

Customizing prerelease test commands

As you see, the configuration of prerelease test is done by a script (that was installed on your computer with python3-ros-buildfarm package). See its option with the help to get some idea for customization. --help
usage: [-h] --output-dir OUTPUT_DIR
                                     [--custom-branch [REPO_NAME:BRANCH_OR_TAG_NAME [REPO_NAME:BRANCH_OR_TAG_NAME ...]]]
                                     [--custom-repo [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME ...]]]
                                     [--pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
                                     [--exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
                                     [--level DEPENDENCY_LEVEL]
                                     config_url rosdistro_name
                                     source_build_name os_name os_code_name
                                     arch [REPO_NAME [REPO_NAME ...]]

Generate a 'prerelease' script

positional arguments:
  config_url            The url of the ROS buildfarm configuration index
  rosdistro_name        The name of the ROS distro from the index
  source_build_name     The name / key of the 'source-build' file from the
  os_name               An OS name from the build file
  os_code_name          An OS code name from the build file
  arch                  An architecture from the build file

optional arguments:
  -h, --help            show this help message and exit
  --output-dir OUTPUT_DIR
                        The output directory

Repositories in underlay workspace:
  The repositories in the underlay workspace will be built and installed as
  well as built and tested. Dependencies will be provided by binary

  REPO_NAME             A name of a 'repository' from the distribution file
                        A name of a 'repository' from the distribution file
                        followed by a colon and a branch / tag name
                        The name, type, url and branch / tag name of a repository,
                            'e.g. "common_tutorials:git:"')

Packages in overlay workspace:
  The packages in the overlay workspace will be built and tested. The
  workspace will be extended with packages from GBP repositories for
  dependencies between the overlay and underlay workspaces. All other
  dependencies will be provided by binary packages.

                        A name of a released 'package' from the distribution
                        file to be included in the overlay workspace
  --exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]
                        A name of a released 'package' from the distribution
                        file which should be excluded from the overlay
                        The depth of the depends-on tree to be included in the
                        overlay workspace (-1 implies unlimited, default: 2)

For example,

  • to run prerelease test using a pull request that's not yet merged, you can do something like this: indigo default ubuntu trusty amd64  moveit   --level 0 --custom-repo moveit:git:   --output-dir ./

What can I do when a prerelease fails?

Once the generate prerelease command is finished, it will instruct you to run the ./ shell script. This in turn runs a shell script for each step of the prerelease. If a step fails, then you can edit the files which are local (not stored inside the docker container) and then each step of the prerelease process can be rerun by executing the corresponding shell script. These are the steps that you might want to rerun:

  • checkouts source code for the repositories you specified

  • checkouts source code for the dependent packages you are going to test on top of the repositories you specified

  • builds and tests the packages in the repositories you specified

  • builds and tests the dependent packages on top of the repositories built and tested in the "build_underlay" step

If you look in the folder where you ran the prerelease then you'll also notice some folders of interest when debugging:

  • catkin_workspace: "underlay" catkin workspace in which the repositories you selected are built and tested

  • catkin_workspace_overlay: catkin workspace in which dependent packages are built; it is chained on top of the "underlay" catkin workspace

You can modify code in these two repositories and rerun one or more of the above shell scripts to iterate on the prerelease test and try to resolve issues.

Wiki: regression_tests (last edited 2017-03-24 17:54:10 by IsaacSaito)