To catch regressions in our code base, we run thousands of tests on all the packages in the ROS distributions. The tests range from small unit tests, to large simulation based integration tests. Our Jenkins server manages and schedules many of the tests. Below you find more details on each class of test that we run.

Development Tests

What gets tested?

The development tests will build and test all packages from the development branch of each repository. The development branch is specified in the source sections of the distribution.yaml rosdistro file (One from ROS Indigo | ROS Kinetic). The devel branch is chosen by the maintainer of a repository, and is often trunk/default/master or <DISTRO>-devel. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Currently the development tests are only run for Ubuntu LTS releases for amd64 due to limited resources on the build farm.

When do tests run?

Every time you commit to a development branch the corresponding development tests will get triggered within an hour. It might take a while until the job is processed if the build farm is busy though. For example, you can find the development jobs for Indigo on Jenkins.

Who gets a notification?

The person that did the commit will receive an email if the build succeeds or fails. If the development job is marked as unstable (due to warnings or failing tests) all committers since the last stable build get notified.

How long does it take to run them?

One test typically takes less than 10 minutes. If the build farm is busy rebuilding Debian packages it might take longer until the job actually gets started.

Pull Request Tests

Pull request tests are identical to development tests but they are being triggered for the branches of pull requests. Currently only GitHub repositories are supported. Commonly pull requests need to be specifically enabled (see REP 143). You can find the pull request jobs for Indigo on Jenkins.

Prerelease Tests

Before pushing out a new release of your package(s), you should run the prerelease tests on your repository. This will help you to avoid breaking the ROS Debian builds for the entire ROS community, and having to push out a bugfix release five minutes later. Starting with ROS Jade, prereleases are run locally on your machine using docker. There is a webpage to help you configure the command line tool that runs the prerelease tests on your machine: http://prerelease.ros.org/.

What gets tested?

For the selected repositories (you can select one or multiple) all packages will be built and the tests will be run. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Additionally, it can download, build, and test packages which depend on your packages. The prerelease website can help you to figure out which settings will test the right amount of packages for you. You can control the number of packages that will be tested by adjusting level of recursive dependent packages. You can additionally specify arbitrary additional packages to test. If you want to avoid building a package, for example a package which is known to be broken for another reason, you can also explicitly exclude individual packages, though this might prevent the testing of other packages. The prerelease website can help you see how different settings affect the packages that will be tested.

Which repositories can be tested?

• For repositories of catkin packages, you can choose between multiple versions:

  • version number: a specific released version of the repo/stack which is fetched from the release repository as listed in the distribution.yaml rosdistro file

  • latest: the latest released version of the repo/stack which is fetched from the release repository as listed in the distribution.yaml rosdistro file. While the distribution.yaml file might not contain a version the latest released version from the GBP repository will be used.

  • devel: the development branch which is fetched from the source repository as listed in the same file. If the source rosdistro file does not contain a version for the repository the default branch is used. Note that this file is manually generated and not part of the automatic bloom release process - you will need to seed it for your repositories with a pull request.

  • All dependent repos/stacks will always be fetched from their released version as listed in the rosdistro file

  • Here is an example "repository" entry in the distribution.yaml rosdistro file:

    •    1   rviz:  # <-- this is the repository's name
         2     doc:  # <-- this describes what to checkout for doc generation
         3       type: git
         4       url: https://github.com/ros-visualization/rviz.git
         5       version: hydro-devel
         6     release:  # <-- this is the "release" entry
         7       tags:
         8         release: release/hydro/{package}/{version}
         9       url: https://github.com/ros-gbp/rviz-release.git
        10       # ^-- url to the release repository
        11       version: 1.10.18-0  # <-- maps to "latest"
        12     source:  # <-- this is the "source" entry
        13     # ^-- maps to the "devel" prerelease option
        14       type: git
        15       url: https://github.com/ros-visualization/rviz.git
        16       version: hydro-devel
        17     status: maintained
      

When do tests run?

The prerelease tests only run when you run them on your local machine.

How long does it take to run them?

Low level repositories can take more than an hour to test, while high level repositories can be finished in 5 minutes. The test duration mainly depends on how many repositories depend on the repositories being tested. You can use the prerelease website to help you adjust the number of packages that are tested and built to avoid extremely long builds due to a large number of dependent packages being tested.

Prerelease Instructions

Setting up Your Computer for Prerelease

First, you need to install Docker. We recommend following this guide at docker.io because you'll need a newer version of docker than the default version that comes with Ubuntu 14.04. Make sure to follow the directions to add your user to the docker group so you can invoke docker without sudo.

Install the ros_buildfarm package to get the scripts and tools you need to run the prerelease:

$ sudo apt-get install python3-ros-buildfarm

$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
$ sudo apt-key adv --keyserver hkp://pool.sks-keyservers.net --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
$ sudo apt-get update

Creating a Prerelease Command

Once you've set up your computer, use the prerelease website to help you generate a command to run the prerelease: http://prerelease.ros.org/

First, the website will prompt you to select a ROS distribution. Select whichever distribution you want to target with your new package. Next, the site will guide you through a 3-step process to generate a terminal command.

1. Select Repositories. In this step, you add packages to the prerelease. You're essentially making a mini build of ROS, and you don't want to build and test every package! So in this step, you'll add your new package(s) to the list. If your package is already on the build farm, you should be able to find it in the drop-down list. If your package is brand new or otherwise doesn't appear in the list, you should use the "Add a custom repository" button and fill in the requested information. When you're done, click Next.

2. Additional Options. In this step, you can choose a custom build farm config file (for advanced users). You should be able to use the default one without issues. You can also choose an operating system to target. This can be useful to test your release on a different OS than the one you're using for local development. Once you've chosen an OS, click Next.

3. Select Dependents to Test. This step may take a bit of time to load. If your package is brand new, you can ignore the rest of this step and go to Step 4. If your package is already on the build farm and has dependencies, this step will automatically add those dependencies to your prerelease for testing, to ensure that your package's changes didn't break those packages. You can add or remove dependencies for testing as desired (but you should probably test all of them).

4. The wizard is complete. Click the Generate Command button and copy the commands into your terminal and run them.

Running the Prerelease

Wait, you still haven't done the prerelease yet! Look at the terminal output carefully. The custom command you ran just created prerelease.sh. To actually run the prerelease, run that file:

$ ./prerelease.sh

Be patient, the command takes a while. The prerelease takes place in a Docker container, and should automatically build your packages, test them, and print the results using catkin_test_results. Make sure that there are no failed tests, and your prerelease is done!

Examples

Below a few common prerelease examples are described. The ros_buildfarm documentation contains additional information and examples on prereleases: https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/jobs/prerelease_jobs.rst

A set of commands the website provides

The command set provided by the prerelease test website above would look like the following. Variables that you can configure on the website are surrounded with %%.

mkdir -p /tmp/prerelease_job
cd /tmp/prerelease_job
generate_prerelease_script.py \
  https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml \
  %ROS_DISTRO% default ubuntu %UBUNTU_DISTRO% amd64 \
  %PACKAGE_TARGETED% \
  --level %DOWNSTREAM_DEPTH% \
  --output-dir ./

• A concrete example:

  mkdir -p /tmp/prerelease_job
  cd /tmp/prerelease_job
  generate_prerelease_script.py \
    https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml \
    kinetic default ubuntu xenial amd64 \
    moveit \
    --level 0 \
    --output-dir ./

  In this example, we're testing:
• %ROS_DISTRO%: ROS Kinetic

• %UBUNTU_DISTRO%: Ubuntu xenial

• %PACKAGE_TARGETED%: moveit

• %DOWNSTREAM_DEPTH%: 0 (i.e. we're not testing if the tests in the downstream packages pass. Depending on the number of packages, the downstream testing can take long)

Customizing prerelease test commands

As you see, the configuration of prerelease test is done by a script generate_prerelease_script.py (that was installed on your computer with python3-ros-buildfarm package). See its option with the help to get some idea for customization.

generate_prerelease_script.py --help
usage: generate_prerelease_script.py [-h] --output-dir OUTPUT_DIR
                                     [--custom-branch [REPO_NAME:BRANCH_OR_TAG_NAME [REPO_NAME:BRANCH_OR_TAG_NAME ...]]]
                                     [--custom-repo [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME ...]]]
                                     [--pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
                                     [--exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
                                     [--level DEPENDENCY_LEVEL]
                                     config_url rosdistro_name
                                     source_build_name os_name os_code_name
                                     arch [REPO_NAME [REPO_NAME ...]]

Generate a 'prerelease' script

positional arguments:
  config_url            The url of the ROS buildfarm configuration index
  rosdistro_name        The name of the ROS distro from the index
  source_build_name     The name / key of the 'source-build' file from the
                        index
  os_name               An OS name from the build file
  os_code_name          An OS code name from the build file
  arch                  An architecture from the build file

optional arguments:
  -h, --help            show this help message and exit
  --output-dir OUTPUT_DIR
                        The output directory

Repositories in underlay workspace:
  The repositories in the underlay workspace will be built and installed as
  well as built and tested. Dependencies will be provided by binary
  packages.

  REPO_NAME             A name of a 'repository' from the distribution file
  --custom-branch [REPO_NAME:BRANCH_OR_TAG_NAME [REPO_NAME:BRANCH_OR_TAG_NAME ...]]
                        A name of a 'repository' from the distribution file
                        followed by a colon and a branch / tag name
  --custom-repo [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME ...]]
                        The name, type, url and branch / tag name of a repository,
                            'e.g. "common_tutorials:git:https://github.com/ros/common_tutorials:pullrequest-1"')

Packages in overlay workspace:
  The packages in the overlay workspace will be built and tested. The
  workspace will be extended with packages from GBP repositories for
  dependencies between the overlay and underlay workspaces. All other
  dependencies will be provided by binary packages.

  --pkg [PACKAGE_NAME [PACKAGE_NAME ...]]
                        A name of a released 'package' from the distribution
                        file to be included in the overlay workspace
  --exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]
                        A name of a released 'package' from the distribution
                        file which should be excluded from the overlay
                        workspace
  --level DEPENDENCY_LEVEL
                        The depth of the depends-on tree to be included in the
                        overlay workspace (-1 implies unlimited, default: 2)

For example,

• to run prerelease test using a pull request that's not yet merged, you can do something like this:

  generate_prerelease_script.py   https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml indigo default ubuntu trusty amd64  moveit   --level 0 --custom-repo moveit:git:https://github.com/130s/moveit:fix/5   --output-dir ./

What can I do when a prerelease fails?

Once the generate prerelease command is finished, it will instruct you to run the ./prerelease.sh shell script. This in turn runs a shell script for each step of the prerelease. If a step fails, then you can edit the files which are local (not stored inside the docker container) and then each step of the prerelease process can be rerun by executing the corresponding shell script. These are the steps that you might want to rerun:

• prerelease_clone_underlay.sh: checkouts source code for the repositories you specified

• prerelease_clone_overlay.sh: checkouts source code for the dependent packages you are going to test on top of the repositories you specified

• prerelease_build_underlay.sh: builds and tests the packages in the repositories you specified

• prerelease_build_overlay.sh: builds and tests the dependent packages on top of the repositories built and tested in the "build_underlay" step

If you look in the folder where you ran the prerelease then you'll also notice some folders of interest when debugging:

• catkin_workspace: "underlay" catkin workspace in which the repositories you selected are built and tested

• catkin_workspace_overlay: catkin workspace in which dependent packages are built; it is chained on top of the "underlay" catkin workspace

You can modify code in these two repositories and rerun one or more of the above shell scripts to iterate on the prerelease test and try to resolve issues.