To catch regressions in our code base, we constantly run thousands of tests on all stacks in the ROS distributions. The tests range from small unit tests, to large simulation based integration tests. Our Jenkins server manages and schedules all tests. Below you find more details on each class of test that we run.

Prerelease Tests

Before pushing out a new release of your packages, you should run the prerelease tests on your repository. This will safe you from the embarrassment of breaking the ROS Debian builds for the entire ROS community, and having to push out a bugfix release five minutes later. There is a central webpage to trigger prerelease tests from: http://prerelease.ros.org/.

• What gets tested? For the selected repos/stacks (you can select one or multiple) all packages will be built and the tests will be run. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Additionally, it downloads all repos/stacks that depend on the tested repos/stacks, and builds/tests them. Again additional dependencies which are not build from source will be installed from Debian packages.

• Which repositories can be tested?

  • For catkinized repositories, you can choose between multiple versions:

    • version number: a specific released version of the repo/stack which is fetched from the release repository as listed in the release.yaml rosdistro file

    • latest: the latest released version of the repo/stack which is fetched from the release repository as listed in the release.yaml rosdistro file

    • devel: the development branch which is fetched from the source repository as listed in the source.yaml rosdistro file. If the source rosdistro file does not contain a version for the repository the default branch is used. Note that this file is manually generated and not part of the automatic bloom release process - you will need to seed it for your repositories with a pull request.

    • All dependent repos/stacks will always be fetched from their released version as listed in the rosdistro file

  • For non-catkin stacks, you can only test the development branch that is listed in the non-catkin rosdistro file

• When do tests run? The prerelease tests only run on demand, when you request them to run at http://prerelease.ros.org/select_distro.

• How long does it take to run them? Low level repos/stacks can take more than an hour to test, while high level repos/stacks can be finished in 5 minutes. The test duration mainly depends on how many repos/stacks depend on the repos/stacks being tested.

• Where can I see inprogress or past prerelease tests? Jenkins prerelease jobs

Development Tests

• What gets tested? The development tests will build and test all packages from the development branch of each repo/stack. The development branch is specified in the source.yaml rosdistro file for catkinized repos, and in the non-catkin rosdistro file for non-catkin stacks. The devel branch is chosen by the maintainer of a repository, and is often trunk/default/master. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Currently the development tests are only run on Precise amd64 due to limited resources on the buildfarm.

• When do tests run? Every time you commit to a development branch the corresponding development tests will get triggered. It might take a while until the job is processed if the buildfarm is busy though. You can find the development jobs on Jenkins.

• Who gets a notification? The person that did the commit will receive an email if the build succeeds or fails. If the development job is marked as unstable (due to warnings or failing tests) all committers since the last stable build get notified.

• How long does it take to run them? One test typically takes less than 10 minutes.

Running regression tests on your machine

• See regression_tests/reproducing.