API review

Proposer: Thibault Kruse

Present at review:

• List reviewers

Description of catkin in current state (08-2012)

The introduction of catkin as a replacement for rosbuild has raised a number of discussions. The change in the approach needs to be justified by a change of requirements/constraints. This section tries to summarize the justification for catkin design decisions as discused in several mailing list threads and documentation. For the discussions, see ros-users and the ROS Buildsystem SIG

Since the workflow from rosmake/rosbuild, vanilla cmake and catkin is very different it is important to understand the implications and forces.

Prerequisites

Catkin strives to be a tool that makes setting up developer environments, building, cross compiling and packaging easier. It competes with rosbuild, vanilla cmake, and other build projects like Autotools.

To understand Catkin, several concepts are helpful in discussion (also see the catkin glossary)

The lifecycle of software development has several phases, in c++ those include:

• building:

  • based in a set of source files binary files are created (including libraries and generated code)

• installing-from-source:

  • after building a set of source and binary files are copied to a specific location

• packaging:

  • create an archive file for a package manager (e.g. apt-get)

• cross-compile:

  • is similar to building but for a different target architecture (e.g. building OSX or Windows binaries on a Linux machine)

• installing:

  • installation of a package which was created by packaging

[DT] Proposing to remove that block since it is nowhere referenced and clarify the layout of a workspace

Building and installing created files somewhere, and apart of the challenge is to select useful places to build and install to. The tree below shows an example of a filesystem tree with several location that may be relevant for discussion.

/
- usr/
-- local/
--- <system install space example>
- home/
-- <username>/
--- local/
---- <userland install space example>
--- ros/
---- build/
----- <workspace build space>
---- <stackname>/
----- build/
------ <project build space>
----- lib/
------ <in-source build space>

Environment variables

A set of environment variables is used to influence how binaries, libraries, etc. are discovered. Important in this context are:

• PATH: a list of directories where executables are looked up

• LD_LIBRARY_PATH: a list of directories where libraries are looked up

• PYTHONPATH: a list of paths where Python modules are looked up

• PKG_CONFIG_PATH: a list of paths where pkg-config searches for PKG.pc files

• CMAKE_INSTALL_PREFIX: a list of paths where CMake searches for PKG-config.cmake files used for find_package()

rosbuild (rb)

Design Goals

• rb_G1: building generates an environment which enables running binaries

• rb_G2: machine-readable meta-information
• rb_G3: Single command build of multiple interdependent software projects
• rb_G4: reducing the release effort of multiple independent packages by grouping several packages into a single release unit (stack).

• rb_G5: when running binaries after building uncompiled resources should be used directly (without copying them) so that changes are immediately effective

Design Decisions

• rb_DD1: separate release unit (stack) from build unit (package) [rb_G4]
• rb_DD2: declare package-to-package dependencies and compiler flags in manifests [rb_G2, rb_G3, rb_G4]
• rb_DD3: cmake invoked per package, in-place builds (CMAKE_BUILD_DIR == CMAKE_SOURCE_DIR) [rg_G1]
• rb_DD4: Python parsing of manifest and custom creation of a build order for packages [rb_G3, rb_DD2, rb_DD5]
• rb_DD5: rosmake wrapper to cmake for single invocation [rb_G3]
• rb_DD6: In-source build and ROS_PACKAGE_PATH for resource location in environment [rb_G1, rb_G5]

Shortcomings

• does not provide an installation-from-source step (no make install)

• packinging just packs the mixture of source and build (which contains much more files than necessary)

• compiler-flags apply uniformly to all target in a package [rb_DD2]
• does not allow multiple builds from a single source tree (due to in-source builds) [rb_DD3]
• does not support building external code on top of ROS packages (due to in-source builds) [rb_DD6]
• rosmake is very slow even when no or only a few changes have been made to the code [rb_DD3]

Catkin (ctk)

Design Goals

Same as rosbuild: rb_G1, rb_G2, rb_G3, rb_G5

Dropped: rb_G4 [DT} review after clarifying design goal

Additional Goals

• ctk_G3: Avoid long environment variables (some OS have size limit)
• ctk_G4: Cross compilation support

• ctk_G5: packaging support

• ctk_G6: Less custom tool maintenance effort than rosbuild

• ctk_G7: FHS compliant layout (REP 122)

• ctk_G8: use existing tools as most as possible, be as little as possible invasive, work well together with other projects

• ctk_G9: installation-from-source support

• ctk_G10: allow multiple builds from a single source tree
• ctk_G11: use individual compiler/linker-flags per target
• ctk_G12: play nice with external code, allow build external code on top of ROS stuff
• ctk_G13: improve performance of make cycle

Design Decisions

• ctk_DD1: cmake invoked per workspace, thereby single invocation [rb_G3, ctk_G6, ctk_G13]
• ctk_DD2: Single build and release unit (stack) [ctk_DD1]
• ctk_DD3: out-of-source build [ctk_G4, ctk_G5, ctk_G10]
• ctk_DD4: build into centralized workspace build folder [rb_G1, ctk_G3, ctk_G6, ctk_G8]

• ctk_DD5: while doing [ctk_DD3] we must be able to load Python code from build folder as well as from source folder, so a init.py script gets generated which looks in both location [rb_G1, rb_G5]

• ctk_DD6: Support for make install target [ctk_G4, ctk_G5, ctk_G7, ctk_G8, , ctk_G12]

• ctk_DD7: Declare dependencies in stack manifests [rb_G2]
• ctk_DD8: Installed package folders are not subfolders of installed stack folders [ctk_DD1]
• ctk_DD9: Source folders also in environment for uncompiled resources [rb_G5]
• ctk_DD10: Use standard cmake find_package() to find stuff [ctk_G8]
• ctk_DD11: Generate PKG-config.cmake and PKG.pc code [ctk_G12]
• ctk_DD12: Python parsing of stack and custom creation of a build order for stacks [rb_G3, rb_DD5]

Alternatives Comparison

Traditional approach with cmake

Cmake is a widely used tool that offers some flexibility in how it is used. We describe here the most common usage.

• building: cmake creates files in the project build space.

  • The dependencies are resolved by searching several paths specified as environment variable as mentioned above. All generated files go into one build folder (and subfolders below it).

• installing-from-source: cmake copies specific files (binaries and dedicated resources) to system or userland install spaces.

• packaging: what is declared in installing-from-source targets gets packaged

In this approach, if source Project C depends on source Project B, and source project B depends on source project A, then the user must know this and build and install A, then build and install B, and then build and install C for a complete build.

• benefits:
  • cleanly separated builds
  • short environment variables, since only one specific paths under CMAKE_INSTALL_PREFIX need to be added to each environment variables
• limitations:
  • users have to do a lot of error-prone repetitive work to build many projects.
  • Uninstalling software must be managed by users, ad hoc approach practically makes uninstalling impossible (stow-like solutions allow uninstalling).
  • The order for building the projects is chosen manually by the user.
  • Necessary include directories and libraries to link are not passed but must be stated explicitly for all recursive dependencies.
  • since all projects need to be installed before being able to run the application, the workflow for i.e. modifying a Python file is an extreme overhead (need to call make install again, which copies the resource to install-space).
  • Makes development work on multiple versions of the same software very cumbersome, because in source overlays are not easy to setup with vanilla cmake.

rosbuild

rosbuild was created to make a different workflow from the traditional workflow easier. rosbuild also uses cmake, but wraps the cmake command and allows not to specify cmake install targets. rosbuild uses the in-source build space for both building and source-installing. rosmake uses custom code to invoke building in dependency projects in the right order, so the user does not need to remember doing this. rosbuild requires dependencies to be defined in special manifest files for packages and stacks. System dependencies are also defined in the manifests and can be resolved with rosdep. rosbuild also features a separation of stacks and packages, where packages are atomic build units, whereas stacks are atomic release units. Releasing also commonly implies packaging for stacks that are available through package managers.

• build: rosbuild+cmake creates files in the in-source build space. Compiler and Linker flags taken from manifests of dependency packages.
• source-install: the projects are part of the environment, building is equal to installing, no install defined to move files to system or userland install spaces
• environment: Based on the ROS_PACKAGE_PATH, all stacks and packages in install and build locations can be part of the environment.
• package: all source and binaries are packaged (e.g. c++ source)
• benefits:
  • Users can call rosmake in one package, and automatically all dependency projects (and only those) are build first in the right order.
  • Users can uninstall source packages by moving or deleting folders.
  • Independent projects can be build in parallel.
  • Users can avoid the pain of defining dependencies in cmake, need only a small subset of cmake commands, using the manifest file for the rest.
  • System dependencies resolved with rosdep.
  • Meta information in manifests about packages can be used for wiki / indexing.
  • Stack/package relationship allows semantic grouping of stacks.
  • Easy lookup of c++ source in apt-get installed stacks (source in the version that runs)
  • For the main platform (Ubuntu), users only need to use and learn a small subset of cmake (e.g. no advanced target management).
• limitations:
  • Large environment variable ROS_PACKAGE_PATH needs to be maintained.
  • In-source space becomes polluted.
  • Does not support multiple builds from a single source tree.
  • Developers do not define resources to be installed, making it hard to package cleanly.
  • Cross compilation is difficult.
  • Non-rosbuild projects cannot depend on rosbuild projects.
  • Requires manifest files in addition to cmake files.
  • Manifest file version system inferior to that of cmake (no optional dependencies, no required version numbers).
  • Requires maintenance of rosbuild, rosmake, rosdep toolsets.
  • Requires compilation flags to be manually declared in manifest to be used by other rosbuild projects.
  • Debian packages do not respect FHS standard.
  • apt-get stacks bloated with non-essential files (c++ source).
  • Compiler and linker flags used to build a binary in a project are often much more than required, as package dependency does not imply we need the compiler flags, and different cmake targets may just require a subset of flags.
  • Compiler and linker flag exports in manifests must be specified for every platform, compiler, etc. else the package is not portable to those other systems.

catkin

catkin allows to build and install projects as in the traditional cmake workflow. The cmake macros however also offer a different workflow, which is the intended usage of catkin. In the intended usage, building for all workspace projects uses the workspace build space, the workspace build space is also part of the environment.

• build: cmake+catkin creates files in the workspace build space
• source-install: the workspace build space can already be in the environment, but an install command also moves selected files to system or userland install spaces. Thus, installed files can also be used from other workspaces.
• environment: system and userland install spaces, as well as workspace build space
• package: what is declared in install targets gets packaged
• benefits:
  • All projects in the workspace can be build with a single command, paralelizing build more than cmake.
  • Source-installing is not required, but possible.
  • Small environment. Easiest cross-compiling and packaging.
  • No in-source space pollution.
  • Catkin macros reduce the effort of defining dependencies compare to vanilla cmake.
  • Installation respects FHS filesystem standard.
  • Compared to rosbuild, packaging is not bloated.
• limitations:
  • Projects build process could conflict with each other if users created unusual CMakelists targets, as stacks share the same cmake namespace ("Standard" cmake is safe)
  • Uninstalling (removing files from the build folder) is less comfortable/intuitive than for rosbuild, uninstall means delete wworkspace build folder and recreate (can be sped up with ccache?).
  • The indexing information needs to be restructured for the ROS wiki, wiki contents explaining steps in terms of packages may become broken.
  • No stack-package relationship means less grouping of projects.
  • Broken stack/package relationship breaks all ros tools that relied on the assumption.
  • Catkin requires flat workspace folder layout (some developers may not like the lack of structure).
  • Compared to rosbuild, more difficult for developer to access sources of installed code (in the version that is installed).

Feature map

(+ means better, - means worse)

Use cases

Installing ROS+stacks from source

A developer checks out the source of ROS core and several other stacks into a local folder. The user then runs a make-like command. As a result, the user is able to run ROS master and ROS nodes.

Adding a stack from source

A developer has a working ROS instance. The developer checks out the source of a stack, runs a makelike command. From then on, the newly build stack is used (instead of another overlayed stack)

Viewing source of installed package for debugging

Example: A developer wants to code a node that is somewhat similar to a node that he can run, and he wants to see the code of that rather than reinvent the wheel. Other example: the developer suspects a bug in roscpp code and wants to read the code. In this example, it is also crucial that the developer sees the code that runs, not just the latest code in a repository.

Removing a previously added-from-source stack

A developer has a working ROS instance. The developer runs one or more commands. As a result, the artefact from the stack to be removed are no more used in that environment.

Cross compiling stacks

A developer creates binaries to run on several alien architectures.

Packaging stacks

A developer packages a stack or package for Debian/Ubuntu apt-get Fedora

Example:

Dave the developer would like to make a Debian package for his ROS based software daves_inspection_system for release. He's hoping to make his release also palatable to users on limited storage targets.

Dave's workspace/repository contains several packages (Currently a stack) that are targeted at several different platforms, for instance, daves_inspection_system_OCU for the OCU, and the daves_inspection_system_robot for the robot itself.

daves_inspection_system_OCU has many more dependencies than daves_inspection_system_robot and so he would like to package them separately.

daves_inspection_system_robot depends on laser_geometry, but Dave is worried about pulling in PCL as a depencency because laser_filters shares a stack. As such he would like to depend on ROS-packages rather than ROS-stacks.

Dave also hopes that the dev and run-time dependencies are separate not only in ROS, but also in the official Deb packages so that he can keep his separate too and avoid pulling in the full gcc toolchain.

Proposed solution:

Separate the set of packages into multiple stacks.

Discovering stack / package dependencies using the dependency graph

A user wants to see how packages are related to each other, other than being dependent.

Suggestions

AndrewSomerville

Additional catkin goal:

• ctk_G8: Support packaging dev, run-time, and doc separately