About | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange

Only released in EOL distros:   indigo jade

About

The rail_recognition package contains nodes for object recognition and demonstration grasp selection using models from a grasp database, handled by graspdb. The package also contains a node used for generating object models from grasp demonstrations collected with the rail_grasp_collection package. The recognizer has support for a 2D image recognizer to run first in the full recognition pipeline, significantly increasing the runtime of the point cloud recognizer by limiting the number of candidate classes to test with point cloud registration. This package also contains nodes for data collection, training, and testing of the 2D recognizer.

Nodes

model_generator

Generate object models for recognition and manipulation using demonstrations from the grasp database.

Action Goal

model_generator/generate_models (rail_pick_and_place_msgs/GenerateModelsGoal)

• Generate models from a set of grasp demonstrations or object models (specified by id in the grasp database), and store the result as new models in the database.

Action Result

model_generator/generate_models (rail_pick_and_place_msgs/GenerateModelsResult)

• Model generation result.

Published Topics

model_generator/debug_pc (sensor_msgs/PointCloud2)

• Point cloud resulting from registration, only published if the debug parameter is set to true.

model_generator/debug_poses (geometry_msgs/PoseArray)

• Grasp poses resulting from registration, only published if the debug parameter is set to true.

Parameters

/graspdb/host (string, default: "127.0.0.1")

• Grasp database host ip.

/graspdb/port (int, default: graspdb::Client::DEFAULT_PORT)

• Grasp database port.

/graspdb/user (string, default: "ros")

• Grasp database username.

/graspdb/password (string, default: "")

• Grasp database password.

/graspdb/db (string, default: "graspdb")

• Grasp database name.

debug (bool, default: false)

• Debug flag, set to true to publish debugging information.

object_recognizer

Object recognition of passed-in segmented objects, returns the recognized object with name, database id, and grasps filled in.

Action Goal

rail_recognition/recognize_object/goal (rail_manipulation_msgs/RecognizeObjectGoal)

• Recognize a segmented object passed in as the goal.

Action Result

rail_recognition/recognize_object/result (rail_manipulation_msgs/RecognizeObjectResult)

• The segmented object with recognized information (name, database id, and example grasps ordered by success rate) filled in, or simply the segmented object if recognition failed.

Parameters

/graspdb/host (string, default: "127.0.0.1")

• Grasp database host ip.

/graspdb/port (int, default: graspdb::Client::DEFAULT_PORT)

• Grasp database port.

/graspdb/user (string, default: "ros")

• Grasp database username.

/graspdb/password (string, default: "")

• Grasp database password.

/graspdb/db (string, default: "graspdb")

• Grasp database name.

object_recognition_listener

Object recognition of segmented objects, specified by object index in the segmented object list.

Action Goal

rail_recognition/recognize (rail_manipulation_msgs/RecognizeGoal)

• Recognize a segmented object passed in by index.

rail_recognition/recognize_all (rail_manipulation_msgs/RecognizeAllGoal)

• Recognize all segmented objects.

Action Result

rail_recognition/recognize (rail_manipulation_msgs/RecognizeResult)

• Object recognition success.

rail_recognition/recognize_all (rail_manipulation_msgs/RecognizeAllResult)

• Object recognition successes.

Subscribed Topics

object_topic param (rail_manipulation_msgs/SegmentedObjectList)

• Incoming segmented object data.

Published Topics

object_recognition_listener/recognized_objects (rail_manipulation_msgs/SegmentedObjectList)

• Outgoing segmented and recognized object data.

object_recognition_listener/debug (geometry_msgs/PoseArray)

• Poses for most recently recognized object, only published if the debug flag is set.

Parameters

/graspdb/host (string, default: "127.0.0.1")

• Grasp database host ip.

/graspdb/port (int, default: graspdb::Client::DEFAULT_PORT)

• Grasp database port.

/graspdb/user (string, default: "ros")

• Grasp database username.

/graspdb/password (string, default: "")

• Grasp database password.

/graspdb/db (string, default: "graspdb")

• Grasp database name.

object_topic (string, default: "/rail_segmentation/segmented_objects")

• Incoming segmented objects topic.

debug (bool, default: false)

• Debug flag, publishes grasp poses on recognized objects when true.

use_image_recognition (bool, default: true)

• Enable/disable narrowing grasp models by first using the 2D image recognizer before performing point cloud recognition.

rail_grasp_model_retriever

Grasp model retrieval from the grasp database via action server.

Action Goal

rail_grasp_model_retriever/retrieve_grasp_model/goal (rail_pick_and_place_msgs/RetrieveGraspModelGoal)

• Grasp model to be retrieved from the database.

Action Result

rail_grasp_model_retriever/retrieve_grasp_model/result (rail_pick_and_place_msgs/RetrieveGraspModelResult)

• The grasp model retrieved from the database. The point cloud and grasp poses will also be published on their respective topics when the result is sent.

Published Topics

rail_grasp_model_retriever/point_cloud (sensor_msgs/PoitnCloud2)

• Retrieved point cloud.

rail_grasp_model_retriever/poses (geometry_msgs/PoseArray)

• Retrieved grasp poses.

Parameters

/graspdb/host (string, default: "127.0.0.1")

• Grasp database host ip.

/graspdb/port (int, default: graspdb::Client::DEFAULT_PORT)

• Grasp database port.

/graspdb/user (string, default: "ros")

• Grasp database username.

/graspdb/password (string, default: "")

• Grasp database password.

/graspdb/db (string, default: "graspdb")

• Grasp database name.

metric_trainer

Generate data sets for training registration metric decision trees used in classifying pairwise point cloud registrations for the object model generation and recognition pipelines.

Action Goal

metric_trainer/train_metrics/goal (rail_pick_and_place_msgs/TrainMetricsGoal)

• Initiate metric training for all pairs of point clouds of a given type; requires user input for labeling whether each pairwise merge is a successful or not.

Action Result

metric_trainer/train_metrics/result (rail_pick_and_place_msgs/TrainMetricsResult)

• Training result, true on success. The training data itself is saved to an output file in the .ros directory.

Actions Called

metric_trainer/get_yes_no_feedback (rail_pick_and_place_msgs/GetYesNoFeedback)

• Action for collecting feedback for the success of a pairwise merge.

Published Topics

metric_trainer/base_pc (invalid message type for MsgLink(msg/type))

• Base point cloud used for pairwise registration.

metric_trainer/aligned_pc (invalid message type for MsgLink(msg/type))

• Point cloud aligned to the base point cloud.

Parameters

/graspdb/host (string, default: "127.0.0.1")

• Grasp database host ip.

/graspdb/port (int, default: graspdb::Client::DEFAULT_PORT)

• Grasp database port.

/graspdb/user (string, default: "ros")

• Grasp database username.

/graspdb/password (string, default: "")

• Grasp database password.

/graspdb/db (string, default: "graspdb")

• Grasp database name.

collect_images

Run an instance of the 2D image recognizer to save 2D images as training/test data at every segmentation call.

Subscribed Topics

rail_segmentation/segmented_objects (rail_manipulation_msgs/SegmentedObjectList)

• Segmented objects topic from which to save images.

Parameters

collect_images/saved_data_dir (string, default: ros::package::getPath("rail_recognition")/data/saveddata)

• Directory containing trained image recognizer data (not used by this node).

collect_images/images_dir (string, default: ros::package::getPath("rail_recognition")/data/images)

• Directory containing training images (not used by this node).

collect_images/test_images_dir (string, default: ros::package::getPath("rail_recognition")/data/testimages)

• Directory containing test images (not used by this node).

collect_images/new_images_dir (string, default: ros::package::getPath("rail_recognition")/data/newimages)

• Location at which to save new images.

collect_images/save_new_images (bool, default: false)

• Flag for saving new images; if true each segmented object from the collect_images/segmented_objects_topic will be saved as a 2D image in the collect_images/newimages directory.

collect_images/segmented_objects_topic (string, default: /rail_segmentation/segmented_objects)

• Topic for incoming segmented objects, used only if collect_images/save_new_images is true.

save_image_recognizer_features

Run an instance of the 2D image recognizer to build the dictionaries for the recognizer, then exit.

Subscribed Topics

rail_segmentation/segmented_objects (rail_manipulation_msgs/SegmentedObjectList)

• Segmented objects topic from which to save images.

Parameters

save_image_recognizer_features/saved_data_dir (string, default: ros::package::getPath("rail_recognition")/data/saveddata)

• Directory at which to save the dictionaries.

save_image_recognizer_features/images_dir (string, default: ros::package::getPath("rail_recognition")/data/images)

• Directory containing training images, organized by object class.

save_image_recognizer_features/test_images_dir (string, default: ros::package::getPath("rail_recognition")/data/testimages)

• Directory containing test images, organized by object class (not used by this node).

save_image_recognizer_features/new_images_dir (string, default: ros::package::getPath("rail_recognition")/data/newimages)

• Location at which to save new images (not used by this node).

save_image_recognizer_features/save_new_images (bool, default: false)

• Flag for saving new images (not used by this node).

save_image_recognizer_features/segmented_objects_topic (string, default: /rail_segmentation/segmented_objects)

• Topic for incoming segmented objects (not used by this node).

train_image_recognizer

Run an instance of the 2D image recognizer to train the neural net on the training dataset, then exit.

Subscribed Topics

rail_segmentation/segmented_objects (rail_manipulation_msgs/SegmentedObjectList)

• Segmented objects topic from which to save images.

Parameters

train_image_recognizer/saved_data_dir (string, default: ros::package::getPath("rail_recognition")/data/saveddata)

• Directory at which to save the dictionaries.

train_image_recognizer/images_dir (string, default: ros::package::getPath("rail_recognition")/data/images)

• Location of the training dataset, organized by object class.

train_image_recognizer/test_images_dir (string, default: ros::package::getPath("rail_recognition")/data/testimages)

• Directory containing test images, organized by object class (not used by this node).

train_image_recognizer/new_images_dir (string, default: ros::package::getPath("rail_recognition")/data/newimages)

• Location at which to save new images (not used by this node).

train_image_recognizer/save_new_images (bool, default: false)

• Flag for saving new images (not used by this node).

train_image_recognizer/segmented_objects_topic (string, default: /rail_segmentation/segmented_objects)

• Topic for incoming segmented objects (not used by this node).

test_image_recognizer

Run an instance of the 2D image recognizer to test the trained neural net on the test dataset, then exit. Results of the test are printed to the terminal.

Subscribed Topics

rail_segmentation/segmented_objects (rail_manipulation_msgs/SegmentedObjectList)

• Segmented objects topic from which to save images.

Parameters

test_image_recognizer/saved_data_dir (string, default: ros::package::getPath("rail_recognition")/data/saveddata)

• Directory at which to save the dictionaries.

test_image_recognizer/images_dir (string, default: ros::package::getPath("rail_recognition")/data/images)

• Directory containing training images, organized by object class (not used by this node).

test_image_recognizer/test_images_dir (string, default: ros::package::getPath("rail_recognition")/data/testimages)

• Location of the test dataset, organized by object class.

test_image_recognizer/new_images_dir (string, default: ros::package::getPath("rail_recognition")/data/newimages)

• Location at which to save new images (not used by this node).

test_image_recognizer/save_new_images (bool, default: false)

• Flag for saving new images (not used by this node).

test_image_recognizer/segmented_objects_topic (string, default: /rail_segmentation/segmented_objects)

• Topic for incoming segmented objects (not used by this node).

The Recognition Pipeline

The full recognition pipeline takes in a list of unrecognized segmented objects, such as what the rail_segmentation package provides. Each unrecognized object will first be classified by the 2D image classifier to determine a set of candidate object classes of high probability. These candidate classes will then be retrieved from the object model database, handled by graspdb, which are then used by the point cloud recognizer to provide a final object label and set of example grasps for picking up the now-recognized object.

Note that the initial 2D image recognition is an optional step. It can be disabled by setting the use_object_recognition parameter of the object_recognition_listener node to false. Disabling this step will significantly lower the runtime of the algorithm, however, as every model in the database will be considered by the point cloud recognizer.

Setting up the 2D Image Recognizer

The 2D image recognizer can be trained in a few steps. First, an image set representative of the objects in the environment must be collected. This can be accomplished by running a segmentation node, such as the one found in rail_segmentation, and the collect_images node as follows:

• roslaunch rail_recognition collect_images.launch

The images will be saved in the directory specified to the collect_images node. These must then be sorted by hand into a training dataset organized by each object class. See the data directory for an example. The collected images, or a subset of them, can also be sorted into the test dataset.

The recognizer uses a bag-of-words model with a neural network classifier. The next step is to train the dictionary for the bag-of-words, which can be done by running the node below. Note that if the recognizer is being retrained for added objects or data, the same dictionaries can be re-used if the dataset hasn't changed dramatically, and this step can be skipped.

• rosrun rail_recognition save_image_recognizer_features

Next, the neural net itself needs to be trained:

• rosrun rail_recognition train_image_recognizer

The 2D image recognizer is now trained and ready to use. For verification of its accuracy, the test dataset can be used for evaluation:

• rosrun rail_recognition test_image_recognizer

Setting up the Point Cloud Recognizer

See the tutorials section of rail_pick_and_place for details on how to set up an object model database, provide grasp demonstrations, train new object models, and refine existing object models.

Installation

To install the rail_pick_and_place package, you can install from source with the following commands:

•    1 cd /(your catkin workspace)/src
     2 git clone https://github.com/WPI-RAIL/rail_pick_and_place.git
     3 cd ..
     4 catkin_make
     5 catkin_make install
  

Startup

The rail_recognition package contains a launch file for launching the object_recognition_listener node with the various database, segmentation, and recognition parameters set, executed with:

• roslaunch rail_recognition object_recognition_listener.launch

Other major nodes in the package contain similar launch files with parameters set, listed below, where each launch file corresponds to a node of the same name.

• roslaunch rail_recognition collect_images.launch

• roslaunch rail_recognition metric_trainer.launch

• roslaunch rail_recognition model_generator.launch

• roslaunch object_recognizer.launch

• roslaunch rail_recognition rail_grasp_model_retriever.launch

The set of nodes used for training the 2D image recognizer can be run as follows:

• rosrun rail_recognition save_image_recognizer_features
  rosrun rail_recognition train_image_recognizer
  rosrun rail_recognition test_image_recognizer

Model generation can be run with an rviz plugin for visualization found in rail_pick_and_place_tools.