Evaluation of the RGB-D SLAM system
This page describes how to reproduce the results of our evaluation. The installation and benchmarking steps of the below guide are concatenated in this script. Be warned, it will download huge amounts of data and put heavy computational and i/o load on your computer.
Note: you can download the trajectories estimated by RGB-D SLAM from here (for example, to compare the performance of your own system to RGB-D SLAM).
This guide was tested with these preconditions:
Ubuntu 10.10 or 11.04, 64bit Version (see your /etc/issue and the output of uname -m)
- ROS diamondback or electric (required for ORB) is installed
- Qt4 installed as default Qt version (check/change with "update-alternatives --display qmake")
- Optional: OpenGL Hardware Support, required for visualization and SIFTGPU
If you do not already have a directory for personal ROS packages, create one like this:
mkdir -p ~/ros echo 'export ROS_PACKAGE_PATH=~/ros:$ROS_PACKAGE_PATH' >> ~/.bashrc source ~/.bashrc
The remainder of this description assumes you chose ~/ros/ as your personal package path. Otherwise you need to adapt the paths below, where ~/ros is contained to fit your directory strucure.
RGB-D SLAM (+ Requirements)
Download g2o to your personal package path:
svn co https://code.ros.org/svn/ros-pkg/stacks/vslam/trunk/g2o ~/ros/g2o
Download the RGB-D SLAM source code and extract the archive to your personal package path:
wget --no-check-certificate https://www.informatik.uni-freiburg.de/~endres/rgbdslam/rgbdslam.tgz tar xzf rgbdslam.tgz -C ~/ros/
This guide was tested with the above version. If you use Ubuntu 11.10 or want to try using the latest (newer) release, you can get RGBDSLAM from the subversion repository
svn co http://alufr-ros-pkg.googlecode.com/svn/trunk/rgbdslam_freiburg/rgbdslam
Compile rgbdslam and g2o.
rosmake --rosdep-install rgbdslam
This takes quite a time (20min on an old laptop with core2 cpu T5600 at 1.83GHz). You can advance to the next section and fetch the benchmark data and tools meanwhile. Using "export ROS_PARALLEL_JOBS=-jX" with X being the number of your cpu cors at maximum, will speed the compilation up by a factor of X, but will need huge amounts of memory (~ 1GB per core).
If ros dependency resolution failed for your system for some reason, you need to install the system dependencies manually:
sudo apt-get install libglew1.5-dev libdevil-dev libsuitesparse-dev
If this fails and you can not figure out the solution, please inquire with endres(at)informatik.uni-freiburg.de
To evaluate RGB-D SLAM with the RGB-D Benchmark you need to download the benchmark tools.
svn co https://svncvpr.in.tum.de/cvpr-ros-pkg/trunk/rgbd_benchmark/rgbd_benchmark_tools ~/ros/rgbd_benchmark_tools
The bagfiles need to be downloaded separately. Execute the following script to do so (Warning: Downloads several gigabytes):
rosrun rgbdslam download_benchmark_files.sh
The bag files are compressed to save disk space. To achieve the regular performance you need to decompress them
for file in $(rospack find rgbdslam)/rgbd_benchmark/benchmark_data/*bag; do echo $file; rosbag decompress $file; done rm $(rospack find rgbdslam)/rgbd_benchmark/benchmark_data/*.orig.bag # This removes the original (compressed) files. If you want to keep them, move them to another directory
The following commands will checkout the packages required for generation of colored octomaps and compile the octomapping stack.
svn co https://alufr-ros-pkg.googlecode.com/svn/branches/octomap_mapping_color ~/ros/octomap_mapping_color svn co https://code.ros.org/svn/wg-ros-pkg/stacks/motion_planning_common/trunk ~/ros/motion_planning_common rosmake motion_planning_common rosmake --rosdep-install octomap_mapping_color
Running RGB-D SLAM on the Benchmark dataset
The following script will run the RGB-D SLAM system on all sequences in the dataset:
rosrun rgbdslam benchmark_evaluation.sh <feature-type> ~/ros/rgbdslam/rgbd_benchmark/benchmark_data/
This generates the trajectory estimation for each sequence for the given feature type. Valid feature types are SURF, SIFT, SIFTGPU, and (for ROS electric only) ORB.
To compare the generated trajectories with the respective ground truth trajectories run
rosrun rgbdslam summarize_evaluation.sh ~/ros/rgbdslam/rgbd_benchmark/benchmark_data/<feature-type>
This generates a comprehensive summary in the file ".../<feature-type>/evaluation_1.csv" file. Using Open/Libre Office, you can open it (make sure space is no delimiter) and paste the results into "spreadsheet_template.ods" which generates mean values and nice plots for accuracy and runtime
Customization of the Experiments
Using the tools above, you can quickly generate results for different settings of parameters. The parameters that change the behaviour of RGB-D SLAM can be changed in settings_for_evaluation.launch:
rosed rgbdslam settings_for_evaluation.launch
All available settings are shown and explained in rgbdslam_sample_config.launch in the "launch" folder.
Octomaps can be generated most conveniently after the above procedure in a seperate step using the trajectory estimate from RGB-D SLAM. However, you will need to download the tgz packages from the benchmark homepage. You can then use a provided script in the RGBDSLAM package to create the octomaps:
rosrun rgbdslam generate_color_octomap_from_estimate.sh <directory-extracted-from-tgz> <trajectory-file>
E.g., to generate an octomap for the "teddy" sequence using the benchmark's ground truth use
rosrun rgbdslam generate_color_octomap_from_estimate.sh rgbd_dataset_freiburg1_teddy/ rgbd_dataset_freiburg1_teddy/groundtruth.txt
Assuming you followed the procedure above and have the trajectory estimate in ~/ros/rgbdslam/rgbd_benchmark/benchmark_data/SURF, the octomap from RGB-D SLAM's estimate could be created with the command
rosrun rgbdslam generate_color_octomap_from_estimate.sh rgbd_dataset_freiburg1_teddy/ ~/ros/rgbdslam/rgbd_benchmark/benchmark_data/SURF/rgbd_dataset_freiburg1_teddy/rgbd_dataset_freiburg1_teddy.bagafter1_optimization_estimate.txt
Left: Map from Ground Truth, Right: Map from Estimate