About | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange

Hardware Requirements

To use this package you need a stereo camera system or an RGBD device that delivers point clouds with color information. Alternatively you can use a simulated version without any hardware, see cob_gazebo.

ROS API

The cob_object_detection package provides a configurable node for detecting textured objects based on local feature point detection and 3D model matching.

cob_object_detection

The cob_object_detection node takes in sensor_msgs/PointCloud2 messages generated e.g. from a stereo camera pair or an RGBD camera device and performs 6 DOF object detection based on previously trained object models. Object detection and training is triggered using services or actions. Visualization and the detection results are published on topics.

Action Goal

/object_detection/acquire_object_image/goal (cob_object_detection_msgs/AcquireObjectImageActionGoal)

• Goal for training procedure to acquire object

/object_detection/detect_object/goal (cob_object_detection_msgs/DetectObjectsActionGoal)

• Goal for executing object detection procedure

/object_detection/train_object/goal (cob_object_detection_msgs/TrainObjectsActionGoal)

• Goal for training object including the object model name

Action Result

/object_detection/acquire_object_image/result (cob_object_detection_msgs/AcquireObjectImageActionResult)

• empty

/object_detection/detect_object/result (cob_object_detection_msgs/DetectObjectsActionResult)

• List of detect object instances including a confidence and pose estimate

/object_detection/train_object/result (cob_object_detection_msgs/TrainObjectActionResult)

• empty

Action Feedback

/object_detection/acquire_object_image/feedback (cob_object_detection_msgs/AcquireObjectImageActionFeedback)

• Feedback holds number to indicate the approximate progress

/object_detection/detect_object/feedback (cob_object_detection_msgs/DetectObjectsActionFeedback)

• Feedback holds number to indicate the approximate progress

/object_detection/train_object/feedback (cob_object_detection_msgs/TrainObjectActionFeedback)

• Feedback holds number to indicate the approximate progress

Subscribed Topics

/sensor_fusion/stereo/reprojection_matrix (cob_object_perception_msgs/ReprojectionMatrix)

• Reprojection matrix to project 3D coordinates given by sensor_msgs/PointCloud2 to the 2D camera image plane. The matrix is necessary to render the detection results with a 2D bounding box.

Published Topics

/object_detection/point_cloud_2 (sensor_msgs/PointCloud2)

• Point cloud visualization of the training process and the incoming 3D data for training and detection

/object_detection/image (sensor_msgs/Image)

• 2D visualization of the training images and the object detection results by rendering the bounding boxes of the detected objects

Services

/object_detection/detect_object (cob_object_detection_msgs/DetectObjects)

• Executes object detection procedure

/object_detection/train_object (cob_object_detection_msgs/TrainObject)

• Executes object training procedure

/object_detection/acquire_object_image (cob_object_detection_msgs/AcquireObjectImage)

• Service is internally called by autonomous object training procedure

Services Called

/sensor_fusion/stereo/get_colored_pc (cob_srvs/GetPointCloud2)

• Point cloud representing a single view on the current scene image. RGB data of point cloud must represent a 2D image in order to be able to perform feature extraction.

Parameters

~object_model_type (string)

• Specifies the applied object model (MODEL_BAYES or MODEL_6D_SLAM). MODEL_BAYES applies bundle adjustment for 3D model generation and creates feature point clusters using mean shift on all affine transformations (natural and artificial) of all detected feature points. This should be your method of choice. MODEL_6D_SLAM applies FastSLAM for 3D model generation and does not do any clustering of feature points. It is the original object detection algorithm from which MODEL_BAYES emerged.

~visualization (bool)

• Specifies if visualisation is calculated published on the corresponding topics. Visualisation lets you observe the object model generation process in 3D by drawing the generated object model that has been created after incorporating successive viewpoints of the object. It also gives you an impression of the detected object models in 2D by reprojecting the bounding

~logging (bool)

• Specifies if log files are created for the training process (logs the number of features per viewpoint and the remaining number of features after clustering) and for the detection process (logs the number of detected features per object as well as the different confidence values)

~use_STAR_detector (bool)

• Specifies if STAR detector is used for feature detection. For now, this value is not working, yet, and must always be false. However it is intended to integrate an arbitrary selection of feature point detector and descriptor for each object model.

~load_model_from_srs_database (bool)

• Specifies if object models loaded form database. If there is an srs database server available, all object models will be loaded from the database.

~insert_generated_model_to_srs_database (bool)

• Specifies if object models will stored into database. After object model generation process, the object will be stored in srs object database if the database is avalible.

~train_learning_pos_x (float)

• Specifies the x position in camera coordinates of the training centre. Defining a training centre helps the training procedure to reduce the amount of superfluous data that has nothing to do with the training object and accelerates therefore the training procedure.

~train_learning_pos_y (float)

• Specifies the y position in camera coordinates of the training centre. Defining a training centre helps the training procedure to reduce the amount of superfluous data that has nothing to do with the training object and accelerates therefore the training procedure.

~train_learning_pos_z (float)

• Specifies the z position in camera coordinates of the training centre. Defining a training centre helps the training procedure to reduce the amount of superfluous data that has nothing to do with the training object and accelerates therefore the training procedure.

~train_learning_radius (float)

• Specifies the cube length in meter measured from the training centre, to determine the image are that contains the object. e.g. a value of 0.1 will generate a 3D cube with edge length 0.2 meter and centre specified by the train_learning_pos_x, train_learning_pos_y and train_learning_pos_z variables. All 3D points within this cube are potentially relevant for object model creation. All other points are discarded.

~train_affine_fp_transforms (bool)

• Determines if each training image should be artificially transformed to generate more views for object model training. This generates in general better detection results, however at the cost of a longer training time (approx. by factor 10).

~train_min_cluster_size (integer)

• After extracting 2D feature points from all image views, many feature points describe one and the same point in 3D space. To reduce the object model size, the feature points are clustered based on 3D space distance and 2D descriptor distance. The parameter specifies the minimal number of feature points that must constitute a cluster e.g. if you chose 10 degree rotational steps for object model training, each image point is approximately seen from at least 4 different viewpoints. Therefore a cluster size of 3 is reasonable to filter out weak feature points.

~train_max_feature_match_dist_in_meter (float)

• During object model generation process is is necessary to extract features on each viewing picture. if a feature is far from an other feature away, the feature cannot match.

~robot_building_models (bool)

• Flag for object model generation process. If true the algorithm try to segment the grasper from the Care-O-bot 3.

~detect_ransac_norm_inlier_dist_thresh (float)

• RanSaC is used for 3D model fitting. In order evaluate a pose estimate, RanSaC calculates the distance of detected feature points from their predited location based on the pose estimate. The parameters specifies the maximal difference in meter that is acceptable.

~detect_min_inlier_size (integer)

• RanSaC is used for 3D model fitting. This value determines the minimal number of inliers that must be determined to count as a detection hit. This threshold value must be met for any detection result and presents the first choice to increase in case of false positive detections.

Usage/Examples

The package can be used via a launch file which loads all parameters from a .yaml file and starts the object_detection node.

roslaunch cob_object_detection object_detection.launch

For including the object_detection in your overall launch file use

<include file="$(find cob_object_detection)/ros/launch/object_detection.launch" />

A sample parameter file could look like this

##########################################
# Object model type
##########################################
# MODEL_BAYES or MODEL_6D_SLAM
object_model_type: MODEL_BAYES
# Visualize detection results in 3-D
visualization: true
# Write log files
logging: true
# Feature point detector
use_STAR_detector: false
##########################################
# Parameters for srs database interface
##########################################
# load the models form srs database or from local storage
load_model_from_srs_database: true
# if the build model ends successfully the object will be automatically add th the database
insert_generated_model_to_srs_database: false
##########################################
# Parameters for object model training
##########################################
# Use u, v parameters to limit training region
# Limit the training area of the image by cutting out 
# relevant regions
# Unit [px]
train_learning_segm_min_u: 500
train_learning_segm_max_u: 1300
train_learning_segm_min_v: 400
train_learning_segm_max_v: 1000

# XYZ and Radius DEPRECATED and no longer in us and no longer in usee, 
# X Position of object during training in camera coordinates
# Unit: [m]
train_learning_pos_x: 0.0776975028338
# Y Position of object during training in camera coordinates
# Unit: [m]
train_learning_pos_y: 0.188573
# Z Position of object during training in camera coordinates
# Unit: [m]
train_learning_pos_z: 0.837447
# Radius of training area for object segmentation during trainng
# Unit: [m]

train_learning_radius: 0.1
# Apply affine transformation to source images before feature extraction
train_affine_fp_transforms: false
# Minimal cluster size for feature points to be valid
train_min_cluster_size: 3
# After applying the odometry, matching features should ideally be at the same position.
# This value specifies the maximal difference in m that is allowed.
train_max_feature_match_dist_in_meter: 0.005
# Enable PreBuildSegmentation to eliminate grasp (sdh) from images and improve transformation data 
robot_building_models: true
##########################################
# Parameters for object model detection
##########################################
# Minimal cluster size for feature points to be valid
detect_ransac_norm_inlier_dist_thresh: 0.014
# Standard deviation of inlier points
detect_min_inlier_size: 10

Definition of object coordinate system

The frame of the object is defined like the following picture.

The front side of the object is the native object front like you would place an object into a shelf. The the coordinate system is oriented to the specific object. Normally the front side show in the negative Y-direction. The Z-direction show vertical to the top.

Tutorials

Tutorials can be found on the Tutorials page.

Models

You can get model files either from https://github.com/ipa320/srs_data or from an internal Server at IPA

smb://saturn20.ipa.fhg.de/austauschipa/J0SF

To insert models into cob_object_detection, pllace the models you like to use in the folder

roscd cob_object_detection/common/files/models/ObjectModels/

Either you delete the Info.txt or you write the name correct inside the file.

rm Info.txt

The object names must be "Model_<object_name>.txt".