(!) Please ask about problems and questions regarding this tutorial on answers.ros.org. Don't forget to include in your question the link to this page, the versions of your OS & ROS, and also add appropriate tags.

Running ROS across multiple REMOTE machines

Description: This tutorial expands on the previous Tutorial (Running ROS across multiple machines) to include the discussion on remote ROS networks. That tutorial focuses on the situation where the multiple machines are connected by the same local network, i.e. they all share the same public IP address (See figure 2 below). This expands that discussion to Remotely networked machines, i.e. machines that are connected to each other through separate internet hotspots or so that each machine has its own public IP address (See figure 3).

Tutorial Level: INTERMEDIATE

Next Tutorial: Defining Custom Messages


This tutorial covers issues that are outside the scope of ROS, such as Internet Protocols, networking, modem setup, ISP issues, and so on. This section introduces a quick review of these issues and why they matter.

The ROS network

As discussed in the previous Tutorial, ROS is designed for distributed computing, where a number of connected components (robots, computers, mobile device, etc.) are networked as seen here.


Figure 1: The ROS network

Since all of these component share the same modem (or hotspot), they form a network local to this modem, and therefore this network is referred to as the local ROS network, in which all connected components have the same public IP address, which is the IP address of the modem. Every remote query (from the internet) to any component of this network is blocked at the modem. This is a fundamental feature of Internet and a security protocol to protect the local components against hacking and other threats.


Figure 2: The Local ROS network

Within the local ROS network, every component is identified with its unique local IP addresses (see figure 2). These are the IP addresses used to run ROS across multiple local machines, as discussed in the previous Tutorial

The remote ROS network

In many robotics applications, such as outdoor robots, there comes a need for the robot to cover vast areas and be separated from the operator. Examples include agriculture robots, search & rescue Robots, and others. In these situations, the robot and operator might not share the same modem (or internet hotspot) and instead each connect to the internet separately, and then connect remotely to each other. This setup is referred to as the remote ROS network.


Figure 3: The Remote ROS network

Figure 2 also shows the challenge of remote ROS networks. We can no longer use local IP addresses (as suggested in the previous Tutorial) to identify components since each component resides in a separate local network. Also, we cannot use their public IP addresses either, since these IP addresses refer to their modems rather than to the components thesmselve.

For example, if the laptop in figure 2 attempts to contact the robot using the robot's public IP address, it would not be able to do so, because that IP address refers to the robot’s modem and not the robot itself, and vice versa.

Therefore, there is a need to Forward that connection from modems onto their internal components, thus facilitating the requirements of ROS and so establishing a successful remote ROS network. This brings the need for PortForwarding

PortForwarding (PF)

There has been a number of approaches to set up remote ROS networks, some include Cloud Robotics, Third-party web solutions, Android, and/or combinations. An extensive review of these efforts can be found here. This tutorial focuses on using PortForwarding as the tool to enable the Remote ROS networks. PortForwarding has its advantages and limitations over other methods. These are summarized below:

Advantages of PortForwarding

Cloud-based solution facilitate remote ROS network by transferring the contents of the ROS messages through the cloud. This would require a two-way data conversion, or rosmsg-webformat-rosmsg bridge for each rosmsg being transferred throug the cloud.

A simple Caller/Listener application contains few rosmsgs so it might require few bridges, but a complex outdoor robot application might contain hundreds, or even thousands, of rosmsgs that each would require an individual bridge to be transferred through the cloud.

  • cloudVsPF.png

Figure 4: PortForwarding vs. Cloud-based solutions for remote ROS networks

Furthermore, should there be any changes or modifications to the system (introducing a new sensor, modifying robot hardware, etc.) new bridges would be needed as well.

Port forwarding offers an alternative approach, as seen in Figure 4, it opens up a remote ROS-to-ROS connection where ALL rosmsgs in the application can be remotely transferred without any conversion or bridges needed. PF also supports any future system changes or updates.

Challenges of PortForwarding (and how to overcome them)

Unfortunately, the use of Portforwarding has its limitations, but these limitations can be managed:

  1. As shown below, PF require the beforehand knowledge of all public IP addresses of the networked components. This is further problematic if these IP addresses were dynamically changing (as set by the ISP). However, this tutorial provides a solution to manage this problem.
  2. Implementing PF is multi-layered proces that involves settings with the ISP, Modem/Routers setup, Operating System configuration, setup within ROS itself, and in that order. This makes setup and troubleshooting a challenge, especially for beginners.
  3. Not all ISPs allow or support the use of PF, those that do might require permission/special access rights.

The next section details the steps needed to configure, troubleshoot, and overcome the challenges of implementing PortForwarding for remote ROS networks.

Configuration / Troubleshooting Steps

It is important that the following steps are tackled in the given order: ISP => Modem => Operating System => ROS => Dynamic IPs. Some settings in later steps depend on the successful implementation of the earlier steps. The steps below might sound trivial/redundant to some, but for best results, it is recommended that you follow steps as recommended here. The following sections show the outline/summary of the configuration/Troublshooting steps. For detailed, line-by-line steps with screenshots and explanations, refer to the attachedPaper. (henceforth, this will be referred to as "The Paper")

STEP 1: ISP Settings

Configuration Steps:

  1. Verify with your provider/network Admin that PF is supported/allowed/enabled on your network.
  2. Optional (but recommended): Obtain a fixed Public IP address for your modem.

Verification Steps (verify that PF is allowed on your network):

  1. Nothing to be verified here, apart from confirmation from your provider/admin that PF is allowed/enabled.
  2. Once Verified, proceed to STEP 2

STEP 2: Modem/Router Settings

At this stage, your ISP has enabled/allowed Portforwarding, next we need to configure the modem so it can allow remote queries to be forwarded to your robot (or other components from the ROS network, see figures 1 & 3)

Configuration Steps:

  1. On your browser/App, Login to your modem homage, navigate to firewall settings => enable Port Forwarding, DMZ, and other filtering settings

  2. Optional (but recommended): Set the local IP address and port range (see the Paper for details)

  3. Repeat this step for every modem of every machine of this remote network (see figure 3)


Figure 5: Modem Configuration for PortForwarding

Verification Steps (verify that your modem/router successfully allowed PF):

  1. First insure that your connection is up, open a browser and connect to the internet
  2. Select a Port number between 1024 and 65000
  3. Host a netcat chat session on the selected port (nc -l portNumber)
  4. On the browser, open a PortChecker website and check the status of the portNumber you selected

  5. If the status is OPEN, then this step is completed and you have successfully enabled PF on your modem
  6. Once Verified, you can proceed to STEP 3

STEP 3: Operating System (Linux) Settings

At this stage, your ISP has allowed/enabled Portforwarding, and your modem/router was properly configured to forward queries to your machine. Next we need to configure your system so it can identify and interact with all members of the remote ROS network (See figures 1 & 3), as per ROS networking requirements. The following steps are to be conducted on every member of the remote ROS network (robot, laptop, etc.).

For now, let us assume that all Public/Local IP addresses are fixed. Fixed public IPs can be granted by the ISP, fixed local IPs are defined/reserved in the modem's home page. Dynamic IPs will be covered later.

Configuration Steps:

  1. In the hosts file, define all public/local IP addresses and hostnames for all components in the ROS network.

  2. Optional (but recommended): Define as many definitions as possible, be it for local/remote networks, through fixed/mobile modems, etc. (see Figure 6)


Figure 6: Hosts definitions for multiple network configurations

Verification Steps (verify that hosts were defined properly for PF):

To verify that hostnames and IP addresses have been correctly configured, we will use three tests; the Ping test, the SSH test, and the netcat test. These tests are chosen to verify that ROS network requirements has been meet, namely: 1) each member can ID all other members in the network by their IP Public addresses (Ping), 2) each member can can access other members in the network (SSH), and 3) each member can act as server and/or client within the network (netcat).

  • Ping Test

  • Open the Terminal in one of the members of the network, say the home computer
  • Run: ping otherMemberHostName, as defined in the hosts file, (example: ping turtlebot)
  • If Ping test OK, run the same test from other side
  • If OK for all members, then proceed to the SSH test.
  • SSH Test

  • Open the Terminal in one of the members of the network, say the home computer
  • SSH from this machine to the robot, using its hostname or IP address (as defined in the hosts file)
  • If SSH test OK, run the same test from other side
  • If OK for all members, then proceed to the netcat test.
  • netcat Test

  • Open the Terminal in one of the members of the network, say the home computer
  • Start a netcat session from the home computer (select a portNumber, and run: nc -l portNumber)
  • On the laptop, start a netcat session on that same port, and exchange few quick messages
  • If messages successfully read and received on both sides, then netcat test OK.
  • Repeat process from the robot's computer (host the session there), and verify it is also OK
  • If all members verified can work as host and/or client, then proceed to STEP 4.

STEP 4: ROS Settings

At this stage, your ISP has enabled/allowed PF, your modem/router is properly configured, and each member of your network has been configured to recognize and interact with all other members of the network remotely through PortForwarding. Next, we need to configure ROS within each machine to facilitate the remote ROS nework. The Previous Tutorial begins from this point. Configuration Steps:

  1. In the ROSmaster member of the network, open a terminal ...

  2. Run this command: export ROS_MASTER_URI=ROSmaster_local_IP:11311 (replace ROSmaster_local_IP with the local IP address of the ROSmaster machine)

  3. In all other components of the network, open a terminal ...
  4. Run this command: export ROS_MASTER_URI=ROSmaster_public_IP:11311 (replace ROSmaster_public_IP with the public IP address of the ROSmaster machine)

Now, every machine in the remote network knows which machine is the ROSmaster, which is another requirement of ROS networking requirements.

Verification Steps:

Once again, there will be three tests conducted to verify that everything is configured properly, and once again, these tests are to be run on every member of the remote ROS network.

  • The Minimal Network test (to verify ROSmaster is recognized by everyone)

  • To very that each member can recognize the ROSmaster, After insuring all previous STEPS are successful, go to the ROSmaster machine, and run: roscore

  • This will start a ROS session, that should be recognized by all members of the remote ROS network.
  • In any member of the network, open a terminal and run: rostopic list

  • If all OK, then you should see a list of ROS topics, from the session started by the ROSmaster. There might be a delay of the outcome of the command due to networking
  • repeat for all members of the remote network
  • If OK, Proceed to the Talker/Listener test.
  • The Talker/Listener Test (to verify all members can act as server/client)

  • With the session still running, start a Talker from any member of the network.
  • In any other member of the network, start a Listener, and see if messages are sent/received successfully
  • Repeat the test, but reverse the Talker and Listener
  • Repeat for all members of network,
  • If OK for all members, then proceed to the Robot Interaction tests.
  • Robot Interaction Tests (to verify with actual robot)

If all is OK up to this point, then congratulations, the remote ROS network is functioning through PortForwarding! Next, you will need to verify it further by testing it with a real robot. Using Turtlebot's interaction as an example.

  • At the robot's machine, bring-up the robot (or you can ssh to it from your home computer and then bring it up)
  • From the home computer, launch the visualization, keyboard teleop, and other components.
  • Using the keyboard of your home computer, teleoperate the robot.
  • If all goes well, the robot should receive the commands through the remote network and move accordingly
  • Also, the robot should return back its visualization (and even its life footage) through the remote network.

STEP 4: Managing Dynamically Changing IPs

Some ISPs may not provide fixed Public IPs to subscribers. Instead, they would provide a range of values for their IPs, such as 173.17.XXX.XXX, where XXX could range from 000 to 255. This would obviously complicate the setup and configuration steps discussed above as steps 2 through 4 would need to be revisited every time the IP addresses are changed. The Solution is to automated this step through the use of environment scripts and cloud solutions (ironic!), as seen in figure 7.


Figure 7: Auto-Updating IPs to overcome Dynamic IP addresses

Current IP definitions (similar to those in the hosts files) would be stored in a cloud-storage facility such as Dropbox or other. Each member would have access to the file and can read it /write to it. Environment scripts in the ROSmaster would check the current IP values and update configuration autonomously through out the network. The degree of autonomy can be defined by the user.

Configuration Steps (Auto-Updating IP addresses)

The actual scripts and further discussion can be found in the Paper (and its accompanying Github page), the following is the Auto-update algorithm and logic.

  1. Upon starting up the ROSmaster machine, the master-script is run, which does the following:
    1. Checks current Public IP address of the machine
    2. Compares the IP address with the values stored in the hosts file in the cloud.

    3. If values are the same, then exit (no changes/updates needed)
    4. If values are different, then update PF settings
      1. update hosts values

      2. Update ROSmaster_Public_IP and ROSmaster_Local_IP
      3. Once values are changed in file on storage facilities, all members in network are notified.
      4. This triggers environment scipts in all member machines to update values accordingly.

The degree of Autonomy (completely autonomous or User-verified) can be defined by tweeking the scripts (and commenting/uncommenting some specific lines)

Wiki: ROS/Tutorials/MultipleRemoteMachines (last edited 2020-01-08 09:30:42 by judejtng)