## For instruction on writing tutorials
## http://www.ros.org/wiki/WritingTutorials
####################################
##FILL ME IN
####################################
## for a custom note with links:
## note =
## for the canned note of "This tutorial assumes that you have completed the previous tutorials:" just add the links
## note.0=
## descriptive title for the tutorial
## title = Converting between ROS images and OpenCV images (Python)
## multi-line description to be displayed in search
## description = This tutorial describes how to interface ROS and OpenCV by converting ROS images into OpenCV images, and vice versa, using cv_bridge. Included is a sample node that can be used as a template for your own node.
## the next tutorial description (optional)
## next =
## links to next tutorial (optional)
## next.0.link= [[cv_bridge/Tutorials/Converting between ROS images and OpenCV images (Android Java)|Converting between ROS images and OpenCV images (Android Java)]]
## next.1.link=
## what level user is this tutorial for
## level= IntermediateCategory
## keywords = image, images, OpenCV, cvbridge, !CvBridge
####################################
<<TOC(3)>>

<<IncludeCSTemplate(TutorialCSHeaderTemplate)>>

== Concepts ==
ROS passes around images in its own <<MsgLink(sensor_msgs/Image)>> message format, but many users will want to use images in conjunction with OpenCV. `CvBridge` is a ROS library that provides an interface between ROS and OpenCV. `CvBridge` can be found in the [[cv_bridge]] package in the [[vision_opencv]] stack.

In this tutorial, you will learn how to write a node that uses `CvBridge` to convert ROS images into OpenCV [[https://docs.opencv.org/3.3.0/d3/d63/classcv_1_1Mat.html|cv::Mat]] format.

You will also learn how to convert OpenCV images to ROS format to be published over ROS.

{{attachment:cvbridge3.png}}

== Converting ROS image messages to OpenCV images ==
To convert a ROS image message into an `cv::Mat`, module `cv_bridge.CvBridge` provides the following function:

{{{#!python
from cv_bridge import CvBridge
bridge = CvBridge()
cv_image = bridge.imgmsg_to_cv2(image_message, desired_encoding='passthrough')
}}}
The input is the image message, as well as an optional `encoding`. The encoding refers to the destination `cv::Mat` image.

If the default value of "passthrough" is given, the destination image encoding will be the same as the image message encoding. Image encodings can be any one of the following OpenCV image encodings:

 * 8UC[1-4]
 * 8SC[1-4]
 * 16UC[1-4]
 * 16SC[1-4]
 * 32SC[1-4]
 * 32FC[1-4]
 * 64FC[1-4]

For popular image encodings, `CvBridge` will optionally '''do color or pixel depth conversions as necessary'''. To use this feature, specify the `encoding` to be one of the following strings:

 * `mono8`: `CV_8UC1`, grayscale image
 * `mono16`: `CV_16UC1`, 16-bit grayscale image
 * `bgr8`: `CV_8UC3`, color image with blue-green-red color order
 * `rgb8`: `CV_8UC3`, color image with red-green-blue color order
 * `bgra8`: `CV_8UC4`, BGR color image with an alpha channel
 * `rgba8`: `CV_8UC4`, RGB color image with an alpha channel

Note that `mono8` and `bgr8` are the two image encodings expected by most OpenCV functions.

== Converting OpenCV images to ROS image messages ==
To convert an `cv::Mat` into a ROS image message, `CvBridge` provides the following function:

{{{#!python
from cv_bridge import CvBridge
bridge = CvBridge()
image_message = bridge.cv2_to_imgmsg(cv_image, encoding="passthrough")
}}}
The use of "encoding" is slightly more complicated in this case. It does, as before, refer to the `cv::Mat`. However, `cv2_to_imgmsg()` does not do any conversions for you (use [[http://opencv.willowgarage.com/documentation/python/miscellaneous_image_transformations.html#CvtColor|CvtColor]] and [[http://opencv.willowgarage.com/documentation/python/core_operations_on_arrays.html#ConvertScale|ConvertScale]] instead). The ROS image message must always have the same number of channels and pixel depth as the `cv::Mat`. However, the special commonly used image formats above (bgr8, rgb8, etc.) will insert information into the image message about the channel ordering. In this way, future consumers will know whether the image they receive is RGB or BGR.

== An example ROS node ==
Here is a node that listens to a ROS image message topic, converts the images into an `cv::Mat`, draws a circle on it and displays the image using OpenCV. The image is then republished over ROS.

In your manifest (alternatively when using [[roscreate|roscreate-pkg]] or [[ROS/Tutorials/CreatingPackage|catkin_create_pkg]]), add the following dependencies:

{{{
sensor_msgs
opencv2
cv_bridge
rospy
std_msgs
}}}

''(Kinetic users, please see the compatibility section below.)''

{{{#!python block=pythonconverter
#!/usr/bin/env python
from __future__ import print_function

import roslib
roslib.load_manifest('my_package')
import sys
import rospy
import cv2
from std_msgs.msg import String
from sensor_msgs.msg import Image
from cv_bridge import CvBridge, CvBridgeError

class image_converter:

  def __init__(self):
    self.image_pub = rospy.Publisher("image_topic_2",Image)

    self.bridge = CvBridge()
    self.image_sub = rospy.Subscriber("image_topic",Image,self.callback)

  def callback(self,data):
    try:
      cv_image = self.bridge.imgmsg_to_cv2(data, "bgr8")
    except CvBridgeError as e:
      print(e)

    (rows,cols,channels) = cv_image.shape
    if cols > 60 and rows > 60 :
      cv2.circle(cv_image, (50,50), 10, 255)

    cv2.imshow("Image window", cv_image)
    cv2.waitKey(3)

    try:
      self.image_pub.publish(self.bridge.cv2_to_imgmsg(cv_image, "bgr8"))
    except CvBridgeError as e:
      print(e)

def main(args):
  ic = image_converter()
  rospy.init_node('image_converter', anonymous=True)
  try:
    rospy.spin()
  except KeyboardInterrupt:
    print("Shutting down")
  cv2.destroyAllWindows()

if __name__ == '__main__':
    main(sys.argv)
}}}
Don't forget to `chmod +x` your file.

<<CodeRef(pythonconverter,6,6)>> All of OpenCV is included by importing cv. In the manifest, add a dependency to `opencv2` and `cv_bridge`.

<<CodeRef(pythonconverter,9,9)>> `CvBridge` also lives in `cv_bridge`.

<<CodeRef(pythonconverter,23,26)>>  Converting an image message pointer to an OpenCV message only requires a call to the function `imgmsg_to_cv2()`. This takes in the image message, as well as the encoding of the destination OpenCV image.

You should always wrap your call to `imgmsg_to_cv2()` to catch conversion errors.

To run the node, you will need an image stream. Run a camera or play a bag file to generate the image stream. Now you can run this node, remapping the image stream topic to the "image_topic". For example, "/camera/rgb/image_color".

If you have successfully converted images to OpenCV format, you will see a `HighGui` window with the name "Image window" and your image+circle displayed.

<<CodeRef(pythonconverter,35,38)>> The edited image is converted back to ROS image message format using `cv2_to_imgmsg()` with the encoding "bgr8", so future subscribers will know the color order.

You can see whether your node is correctly publishing images over ros using either [[rostopic]] or by viewing the images using [[image_view]].

=== ROS-Kinetic Compatibility ===
Since ROS Kinetic, the default supported version is [[opencv3|OpenCV 3]]. In order to run this tutorial on Kinetic, you should find_package the version 2 in your CMakeLists.txt ([[vision_opencv|see here]]). You should ensure that OpenCV 2 is installed in your system (libopencv-dev on Ubuntu 16.04).

Then you can run the example code above. However, since OpenCV2 is not a ROS package anymore, you can't add it to catkin dependence. Hence, the list of dependent packages in your manifest should be:

{{{
sensor_msgs
cv_bridge
rospy
std_msgs
}}}
## AUTOGENERATED DO NOT DELETE
## TutorialCategory
## FILL IN THE STACK TUTORIAL CATEGORY HERE