ROS Buildfarm Redesign Motivation

Background

A very valuable part of the ROS community is the buildfarm operated by OSRF. It is provides the functionality to make ros.org the hub for the community. It provides up to date code documentation, metadata for the wiki, binary packages, and continuous integration for the whole open source community. All of these resources need continuous operations behind the scenes to keep the system working.

The system evolved over many years of development to support these many different roles. The continuous evolution of the system means that there’s a lot of state in the system and it’s not reproducible. With only the live system development must be very slow to prevent breaking the running system, and it’s very hard to test new features to make sure that it does not break existing ones inadvertently.

New Features in Design

Reproducibility

One of the biggest challenges for users of the build and test pipeline is that they cannot necessarily reproduce the exact build environment on the running system. Without being able to run the exact same scripts due to custom environments there are errors which cannot be reproduced locally and requires iterating releases to debug on the main buildfarm.

Deployability For Testing

To support testing it’s important to be able to bring up a ROS buildfarm from scratch. This can allow us to test new features on a non-production system before deploying to the production system.

Deployability for Users

Customization

As we seek to grow ROS it’s important to support our users and their ability to customize ROS. From the beginning we’ve chosen to be completely open with liberal licensing to be commercial usage friendly. However if the infrastructure is not available for commercial users to use customized versions of ROS we are not effectively open. There are many cases where users may have requirements for their project that they build everything they deploy from the verified sources and cannot rely on external release reviews. This certainly was possible before, but if they needed to do this they lost the value of the packaging infrastructure.

Extension

There are also many use cases where ROS is used and our users want to extend the rosdistro for commercial purposes which might not be appropriate on the community buildfarm. For example a robot manufacturer might want to build and distribute firmware and drivers for their robot. For the highest quality they will want to have full control of the release pipeline for those packages, and likely have development and testing packages available internally and for their users. All of these will be compatible with upstream packages but they need the full infrastructure for these packages.

Approach

When approaching reproducing the buildfarm we needed to collect all the configuration points into defined inputs. Many of these configurations were scattered around in different places and needed to be collected into a central location for easier customization.

deployment vs job configuration

The design was separated into pure deployment scripts which are ROS agnostic and set up the underlying infrastructure in ros_buildfarm_deployment which sets up the computers for the buildfarm and installs the core applications and configures them for basic interconnectivity. However with only the deployment you will only have empty idle servers.

The rosdistro aware ros_buildfarm takes it’s own configuration which points to the rosdistro configuration and generates all the rosdistro specific jobs.

REP 143

To support the buildfarm a few more fields were required to be added to the rosdistro distribution file format for more specific information. And the information which was previously in the index file and is now duplicated in the buildfarm config has been removed.

http://www.ros.org/reps/rep-0143.html

Jenkins

We chose to continue to use Jenkins-CI as our main continuous integration framework. It continues to be the more prevalent open CI solutions available. It’s highly configurable and has a very large community.

Docker

To address the reproducability we chose to use Docker to provide deterministic environments in which to run our builds. It also provides portability by which we can support different distros and arches on the same server.

Docker images also allow users to reproduce the exact environments as on the server for better local debugging.

reprepro

We continue to use reprepro as the easiest to use debian package archive.

Apache2

Apache2 provides the web hosting for the basic web services such as hosting the documentation.