API review

Proposer: Tully Foote

Present at review:

• Tully Foote
• Jonathan Bohren
• Thibault Kruse
• Jack O'Quin

Introduction

This is a review of a proposal to change the naming / migration strategy from rosbuild stacks and packages to something new with catkin.

See this post for some background on this proposal.

Prerequisites

• Variant: A set of ros distribution units (e.g. rosbuild stacks)
• Meta-package: Also a set of ros distributions units

Variant and Meta-package can be synonymous, historically the word "variant" was used for large sets, whereas "meta-package" is a name used for small sets like rosbuild packages of one stack.

Rosbuild Approach

1. Packages are atomic units of build. Packages are folders having a manifest.xml. Packages depend on packages.
2. Stacks are atomic units of release. Stacks are folders having a stack.xml. Stacks depend on stacks.
3. Stack contain packages

4. Unary stacks are packages themselves and do not contain packages REP 109

Benefit

• Compared to releasing each package, the number of release units was decreased by an order of magnitude

Limitations

• package dependencies and stack dependencies need to be kept synchronized manually
• lookup of stacks from given packages was non-intuitive with the available tools, especially outside the indexed space
• stack/package subfolder layout is difficult to maintain with an FHS layout
• versioning of package-generated libraries using the stack version does not work with packages moving between stacks

Catkin-fuerte Proposal

1. Stacks are atomic units of build
2. Stacks are also atomic units of release
3. Stacks have a top-level CMakeLists.txt, a stack.xml and subfolders
4. Migration transforms rosbuild stacks into catkin stacks, and rosbuild packages into catkin subfolders.

Catkin post-fuerte Proposal

1. We will release packages individually.

2. We will have “meta-packages” which are groups of packages that can be installed by named aliases easily, but these will simply be content free packaging decisions, not source controlled content that must agree with the package dependency. Meta-packages membership is to be managed centralized like variants on servers such as Willow Garage servers (as opposed to be managed decentralized by stack folders). Multiple servers can be used together as sources (for corporate user-groups). The variants will be kept in a file distro.yaml in a github project, so that requests for changes can be made as git pull requests,

3. There will be one xml file per package (the manifest.xml) which will have all the relevant meta data, dependencies, and information such as plugin declarations and possibly interfaces in the future.

4. Other than catkin-fuerte, there will be no restrictions of how many packages can live in the same repository subtree, to allow for easier migration from rosbuild stacks. Also catkin workspaces will thus not require a flat alyout of projects in the workspace folder anymore.

5. Variants can have two data fields: description and url (both optional, description recommended)

6. We will update the PackageHeader wiki macro to list all variants in which that the package is contained. We will create a VariantHeader wiki macro which lists all the packages in the variant, and what it extends.

7. For migration purposes, we would create a variant for each stack, so that the wiki content on stacks can easily be converted into meta-information on meta-packages / variants.

PROS

1. maintainers no longer need to maintain the dual redundant dependency trees as in rosbuild (stack dependencies having to mirror package dependencies), dependencies in meta-package are reduncancy-free.
2. things the catkin-fuerte proposal did not offer:
   1. backward compatibility with the wiki
   2. backward compatibility with the package dependencies
   3. multiple packages per repo.
   4. faster build as we can parallelize the individual units (there was already a step in that direction but we now go further in granularity). [TK] please more details
   5. finer granularity for dependencies
3. packages can easily be moved to what was called a “unary stack” or a different repo (just copy the folder, that’s it)
4. packages will have their own version number:
   1. there is a direct upgrade path for providing versioned .so files (maybe that will be done in time for Groovy)
   2. finer granularity for releases
   3. version locking of packages (not through an artificial meta-unit) which enables us to release packages at the same time
5. there is no more worrying about in which stack a package is released and what the neighboring packages bring in as dependencies.
6. no more stack.yaml/main CMakeLists.txt to maintain
7. the concept of stack/variant are merged and are still here for the user for a simple install
8. ROS packages are conceptually the same as Linux packages.
9. Indexing is easier (only one type of unit to index)
10. Discovery is easier (all things in the same namespace are the same type)

CONS

1. increase of the number of Linux (and soon Mac/Windows) packages to build and offer the user by an order of magnitude
2. upgrade requires moving some info from the stack.xml to all the manifest.xml (version number, authors/maintainers) / distro.yaml
3. we still need to adapt quite a few scripts: bloom, rosrelease, roscreate-pkg, ... (but most of those were not even adapted for the catkin-fuerte solution). E.g. people should be able to release several packages at once just like they can do now with a stack (maybe with an automatic version bump). Those will be discussed individually.

Question / concerns / comments

Enter your thoughts on the API and any questions / concerns you have here. Please sign your name. Anything you want to address in the API review should be marked down here before the start of the meeting.

Thibault Kruse

Also asked in the SIG, maybe respond there, I will update here.

Questions

• Will the meta-packages be managed like variants, centralized?

  • [DT] Yes, variants are actually meta-packages themselves - both will be stored in the DISTRO.yaml on GitHub.

• Will there be an enforced 1-to-n or an n-to-m relationship between meta-packages and packages?
  • [DT] The relationship is arbitrary: each meta-package can depend on N packages, a package can be part of M meta-packages and meta-packages can also depend on other meta-packages.
• Does that mean that all bugtrackers for current stacks will have to be split up to defer tickets to the respective package?
  • [DT] There is no fixed mapping between a package (or stack) and a bugtracker. Multiple packages from a former stack can still use the same bugtracker. It would even be possible that a single package uses multiple bugtrackers (even if that looks pretty inconvenient). That is completely up the developer/maintainer. The feature request to store a bugtracker URL in the manifest looks like a good idea to guide users to the right place.
• Does that mean VCS repositories will have to be split up to have one per package (all packages to github)?
  • [DT] Absolutely not. With the large number of packages that is not feasible. If multiple packages are stored in the same repository it is a bit more difficult for a user to checkout only that package (without the other packages of that repository). But that should not be an issue.
    • [TK] Does that mean if multiple packages live in the same SCM subtree (e.g. git repo), that they will still need a common CMakelists.txt in the subtree root to be catkin compliant?
      • [DT in forum] No, catkin will change from earlier designs to allow projects in workspace to live in subfolders.
• Currently one can create an overlay of e.g. the navigation stack, and remove that overlay by removing the "navigation" folder. In the new proposal, I assume we can get a rosinstall for the meta-package "navigation" into out workspace. How about removing that meta-package from the workspace? (Will there be a more convenient way than to manually read the variant and remove the corresponding folders?)
  • [DT] rosinstall currently gets its information from a .rosinstall file and does not rely on the DISTRO.yaml file. So there is no notion of a meta-package for rosinstall.

    • [TK] As far as I understand, the script "http://packages.ros.org/cgi-bin/gen_rosinstall.py" generates rosinstall from the distro.yaml, and havs a "variant" parameter

  • [DT] In catkin workspace all packages overlay packages with the same name in parent workspaces. New packages can be added by putting them in the workspace folder and removed by deleting the folder again. With a global build folder per workspace a make clean would be necessary to remove all artifacts.

    • [TK] Several alternatives exist. rosinstall / rosws could have a remove command, that removes all the packages of a variant instead of adding them. This could be naive, or complex to ensure that a package is kept as long as there is a package in the workspace depending on it. Alternatively, if workspaces are hierarchic / there are CMakelists.txt at the top-level of atomic meta-packages (just onre repo url), removals is as easy as removing the variant/metapackage folder
      • [TK after update of catkin workspace layout]: All this is much less of a problem when dropping the flat catkin workspace layout.
• [TK]"there will be one xml file per package (the manifest.xml) which will have all the relevant meta data,": How does that relate to dependencies within the CMakeLists.txt of a package? I assume that packages will require a CMakelists to act like catkin-fuerte-stacks?
  • [DT] Each package will contain a CMakeLists.txt in the root folder which act the same as in a catkin-fuerte-stack. It used find_package() to find other packages. Some of the catkin macro are likely to change and probably some are going away (like catkin_stack()).

Concerns

• The concept of centralized meta packages depends on a centralized list, which is less flexible than decentralized models, such as meta-package.xml defining a meta-package without the restrictions of a stack.xml.
• Meta-packages do not make it easier for a developer to decide which variants he needs to satisfy package dependency. E.g. A developer checks out a package foo from source, and now he needs to find out what other packages he needs from source for them to build. Not only does he have to check all manifests.xml and make a list of dependencies, removing entries for packages he has, but then he needs to find out what variants to get for each missing dependency, and an n-to-m relationship between variants and pckages means even an automated tool could offer several meta-packages to install, from which the developer will have to choose. This unless each package is also a meta-package for which automated dependency informatio (and rosinstall data) can be generated. [After update] This is somewhat mitigated by the wiki macros, but still confusing if there is a lot of meta-package overlap.
• The proposed solution still relies on redundant information between the manifest.xml (for packaging dependencies) and the CMakelLists.txt (for build dependencies). Differences between those two will also cause more problems once versioned dependencies are added. Suggest creation of CMake Macros that read dependency information from the manifest, e.g. catkin_find_manifest_packages()

Desired Meta-package Semantics

[JOQ] This is important for developing and maintaining reusable ROS components in a distributed, federated environment.

• Meta-packages are sets containing packages and other meta-packages. The set membership graph probably needs to be acyclic to avoid Russell's paradox (involving sets that are subsets of themselves). I am not quite sure about that, but it seems likely. Plus, acyclic graphs are easier to handle.
• Store meta-package information in developer repositories, including a description, a maintainer, a review status, doc URL, a Trac URL. Some of those could be optional. An optional deprecation status would be very helpful for tick-tock. People are free to create meta-packages without those data. Then, I will know to avoid them.
  • When depending on another meta-package (i.e. subset), I would want to know if it is supported (maintained, reviewed, etc.). If not, I probably don't want to depend on it.
  • When depending on collections of packages created by other developers, I don't want to depend on unnecessary detail about the internal structure of those packages. When checking out their source code, I don't want to list every single package, just the name of the collection (e.g. navigation, common_msgs).
  • If I do depend on another meta-package and my system breaks in some future release due to changes, I know who to contact for help in resolving that issue. For maintained meta-packages (the only ones I care about), there needs to be an explicit tick-tock mechanism, so dependent meta-packages can migrate to a new version over time.
  • This was already a problem with variants, but there are very few variants, and for most the maintainers are known from context and likely to provide relatively stable names. I would prefer variants to become meta-packages with similar information, rather than the opposite.
• When releasing a collection of packages, I want the release scripts to permit releasing all the packages in a meta-package at once (regardless of whether they come from a DVCS repository).
  • This is mostly applicable to meta-packages containing only packages, and not other meta-packages.
  • Similarly, I want the pre-release tests to accept meta-packages, so I don't have to list by hand a dozen packages when preparing to release them.
• I am basically discussing semantics here. The best syntax for representing it is not clear, and probably mostly depends on the needs and convenience of developing the support scripts and other tools.
  • Possibly, most of these semantics could be provided by creating an empty package in the repository with a manifest but no code, only dependencies. I am not sure, and have no idea how well that would integrate with CMake and catkin.

Fixing FHS Problems

[JOQ] The main issue is that the Linux Filesystem Hierarchy standard does not permit putting binary executables in the share tree. Instead of $PREFIX/share/PACKAGE_NAME, those files should go in $PREFIX/lib/PACKAGE_NAME.

• That came up too late to fix in Fuerte. So all the existing catkin packages are broken.

Meeting agenda

To be filled out by proposer based on comments gathered during API review period

Conclusion

Package status change mark change manifest)

• Action items that need to be taken.

• Major issues that need to be resolved