About | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange

Unit Testing with RT

Description: This explains how to create unit tests with RT, also for rostest

Tutorial Level: INTERMEDIATE

Next Tutorial: Using actionlib

Creating a unit test with RT

The COMMON LISP implementation used for roslisp is SBCL. SBCL ships with a minimalist testing framework called RT. The ROS package roslisp_testing extends this to provide functionality to write unit tests that are compliant with the rostest testing tool.

roslisp_testing is also used to test roslisp.

Interactive use

Tests can be created for interactive use using the RT macros.

As an example, we will define the fibonacci sequence function and unit test it:

CL-USER> (ros-load:load-system "roslisp_testing" "roslisp-testing")
CL-USER> (in-package roslisp-testing)
ROSLISP-TESTING> (defun fib-trec (n)
  "Tail-recursive Fibonacci number function"
  (labels ((calc-fib (n a b)
                     (if (= n 0)
                       a
                       (calc-fib (- n 1) b (+ a b)))))
    (calc-fib n 0 1)))

ROSLISP-TESTING> (deftest fibtrec-test
    (list
      (fib-trec 0)
      (fib-trec 1)
      (fib-trec 2)
      (fib-trec 3)
      (fib-trec 4)
      (fib-trec 5))
  (1 1 1 2 3 5))

This defines the function and a single unit test for it. We can run the test using RT by calling:

(do-test 'fibtrec-test)
Test FIBTREC-TEST failed
Form: (LIST (FIB-TREC 0) (FIB-TREC 1) (FIB-TREC 2) (FIB-TREC 3)
            (FIB-TREC 4) (FIB-TREC 5))
Expected value: (1 1 1 2 3 5)
Actual value  : (0 1 1 2 3 5).
NIL
(FIBTREC-TEST 0
 (LIST (FIB-TREC 0) (FIB-TREC 1) (FIB-TREC 2) (FIB-TREC 3) (FIB-TREC 4)
       (FIB-TREC 5))
 ((1 1 1 2 3 5)) ((0 1 1 2 3 5)))

As you can see we messed up the test case, expecting the result for (FIB-TREC 0) to be 1, while it is defined as 0. The first return value NIL shows that there have been errors, the next result lists for each testcase the name, the time the test took, the tested function body, the expected result and the actual result.

So we redefine the test, and run it again. Note you can run the last test that was defined by just calling (do-test) without arguments.

ROSLISP-TESTING> (deftest fibtrec-test
    (list
      (fib-trec 0)
      (fib-trec 1)
      (fib-trec 2)
      (fib-trec 3)
      (fib-trec 4)
      (fib-trec 5))
  (0 1 1 2 3 5))

ROSLISP-TESTING> (do-test)
FIBTREC-TEST
(FIBTREC-TEST 0)

So this time no error occured, all tests passed, which is why we get non-NIL as first result, and a list of all tests with the time they took.

Tests expecting errors

Now maybe we want our fibonacci funtion to be robust as well, so that it returns a SIMPLE-ERROR when called with negative numbers. So we define a unit test for that. RT does not provide explicit handling of errors as expected values, but roslisp-testing defines a test fixture for that.

ROSLISP-TESTING> (defun fib-trec (n)
  "Tail-recursive Fibonacci number function"
  (when (< n 0) (error "fibonacci numbers only defined for positive integers: ~a" n))
  (labels ((calc-fib (n a b)
                     (if (= n 0)
                       a
                       (calc-fib (- n 1) b (+ a b)))))
    (calc-fib n 0 1)))
ROSLISP-TESTING> (deftest fibtrec-negative-test
     (type-of
      (with-fixture roslisp-testing:error-caught ()
        (fib-trec -1)))
   simple-error)

Note we give the new unit test a different name. We can now call all unit tests:

ROSLISP-TESTING> (do-tests)
Doing 2 pending tests of 2 tests total.
 FIBTREC-TEST FIBTREC-NEGATIVE-TEST
No tests failed.
T
((FIBTREC-NEGATIVE-TEST 12) (FIBTREC-TEST 0))
12

Multiple tests

RT remembers all tests that were loaded so far, to make it forget tests, call function (rem-all-tests)

Writing fixtures

Fixtures are pieces of code that are called before and after tests. in LISP, those are macros, and you can use the def-fixture and with-fixture macros to use fixtures in an explicit way. Note you can as well use defmacro and give your macro a name that makes it plain that it is a fixture.

Writing suites

Once you start keeping tests in files for later regression testing, RT reaches its limits in organizing many testcases. roslisp-testing extends RT with the concept of suites. This way, you can define multiple test cases in a file, store them in a suite, and load a different file with other test cases.

Testcase files should generally look like this:

;; mypackage-suite1.lisp
(in-package :test-mypackage)

(rem-all-tests)

(deftest foo
  ...)

(deftest bar
  ...)

(create-suite "mypackage-suite1")

;; mypackage-suite2.lisp
(in-package :test-mypackage)

(rem-all-tests)

(deftest floo
  ...)

(deftest barl
  ...)

(create-suite "mypackage-suite2")

The package declaration of such a test package should minimally contain this:

;; package.lisp
(in-package :cl-user)

(defpackage :test-mypackage
  (:documentation "tests for my package")
  (:use
   :cl
   #+sbcl :sb-rt
   #-sbcl :rtest
   :gtest-adapter
   ))

Also extend your ROS manifest.xml to include roslisp_testing.

Using suites with rostest

Test cases that are organized in suites as above can be used with rostest. All you need to do is to create an executable file that calls all your tests and transforms the results into the gtest format. For tests written with RT as in this tutorial, a wrapper function exists that does all that for you.

Given the example above:

;; mypackage-allsuites.lisp
(in-package :test-mypackage)

(defun mypackage-test-main ()
  "runs the test suites and writes results to xml as gtest would"
  (rt-do-suites->gtest  '("mypackage-suite1"
                          "mypackage-suite2")
                       sb-ext:*posix-argv*))

An executable file which calls mypackage-test-main is compatible with the rostest framework.

Creating suite scripts

The following is a simple example for testing using scripts. Create a file name simpletest in a ROS package, maybe neatly in a subfolder test.

simpletest:

#!/usr/bin/env sh
"true";exec /usr/bin/env rosrun roslisp_runtime run-roslisp-script.sh --script "$0" "$@"

(ros-load:load-system "roslisp_testing" "roslisp-testing")

(in-package :roslisp-testing)

;; simple test to show ho to use rt-test
;; call this with (do-test 'addition)
(deftest test-addition
    ;; the test
    (+ 21 21)
  ;; the expected result
  42)

;; another simple test. Note the expected result argument does not get
;; evaluated, therefore (list 1 2) and '(1 2) do not work!!!
(deftest test-append
    ;; the test
    (append '(1) '(2))
  ;; the expected result
  (1 2))

(create-suite "simple-suite")

(rt-do-suites->gtest  '("simple-suite")
                      sb-ext:*posix-argv*)

The first two lines are the lisp script shebang lines. We then load roslisp-testing to have access to the test functions. We define two simple tests, and create a suite. Finally the file does the gtest conversion if the script is called with a gtest command line argument.

Make this file executable and run it:

$ chmod u+x simpletest
$ ./simpletest 
No --gtest_output=xml:filename given in args NIL
Doing 2 pending tests of 2 tests total.
 TEST-ADDITION TEST-APPEND
No tests failed.

As you can see, the function warns that no gtest output xml argument was given, but performs the tests anyway. You can also run the test and generate xml:

$ ./simpletest --gtest_output=xml:foo.xml
...
$ cat foo.xml 
<testsuites>
<testsuite name="simple-suite" tests="2" failures="0" errors="0" time="0.0">
<testcase classname="TEST-APPEND" name="TEST-APPEND" status="run" time="0.0">
</testcase>
<testcase classname="TEST-ADDITION" name="TEST-ADDITION" status="run" time="0.0">
</testcase>
<system-out><![CDATA[Doing 2 pending tests of 2 tests total.
 TEST-ADDITION TEST-APPEND
No tests failed.]]></system-out>
<system-err><![CDATA[]]></system-err>
</testsuite>
</testsuites>

As you can see an xml file was generated summarizing the results in gtest compatible format.

Suites in built executables

It is also possible to use LISP executables defined in clean .lisp files and build as explained in the roslisp tutorials. For that, instead of calling (rt-do-suites->gtest), define a main function in the test file:

...
(defun simple-test-main ()
  (rt-do-suites->gtest  '("simple-suite")
                        sb-ext:*posix-argv*))

And build your executable using this as main function.

You would also need the dependency to roslisp-testing in your manifest and .asd file in that case.

Invoke using rostest

With rostest you can automize invocations of tests. We will create an small package only for the purpose of demonstrating how it is done:

$ roscreate-pkg simpletest_pkg
$ cd simpletest_pkg
$ mkdir test

place the script file simpletest from the previous section inside test, and do not forget to make it executable.

Not we do not need to depend on roslisp or on roslisp-testing, as we just create a script and not a compiled executable.

Then create a file named simple-rostest.launch in the test folder:

<launch>
<test test-name="simple_tests" pkg="simpletest_pkg" type="simpletest"/>
</launch>

Rostest will manage to find the executable file within the package.

To run from the command line:

$ rostest simpletest_pkg simple-rostest.launch 
... logging to ....log
[ROSUNIT] Outputting test results to .../.ros/test_results/simpletest_pkg/TEST-rostest__test_simple-rostest.xml

testsimple_tests ... ok

[ROSTEST]-----------------------------------------------------------------------

[simpletest_pkg.simple_tests/TEST-APPEND][passed]
[simpletest_pkg.simple_tests/TEST-ADDITION][passed]

SUMMARY
 * RESULT: SUCCESS
 * TESTS: 2
 * ERRORS: 0
 * FAILURES: 0

rostest log file is in ....log

And, as the rostest documentation suggests, you can also add this to your CMakelists.txt. Add the line:

rosbuild_add_rostest(test/simple-rostest.launch)

and then call the test using make:

$ roscd simpletest_pkg
$ make test

Other test frameworks

RT is a minimalist testing framework, there are other COMMON LISP testing frameworks available freely, such as FiveAM and stefil. Wrapping those for rostest just means to transform their test results to the gtest XML format.

Helper functions exist to make this somewhat easier. There are testsuite result structures defined in roslisp-testing.

(defstruct gtestfailure
  type ;; string
  message ;; string
  )

(defstruct gtestcase
  classname ;; string
  name ;; string
  time ;; float
  failures ;; list of gtestfailure
  )

(defstruct gtestsuite
  name ;; string
  time ;; float
  testcases ;; list of gtestcase
  sysout ;; string
  syserr ;; string
  )

The function (defun run-suites-write-gtest-file (posix-args suite-fun-list transform-fun &key (no-output nil))...) provides all the common code to create the gtest compatible xml. You need to provide a list of suite-functions, that is functions which will run your suite and return some result object of a type of your choice, and a transform function which will transform this result object into a structure of type gtestsuite.

The function run-suites-write-gtest-file will then parse the posix-arg list for the gtest parameter, run your suite functions and capture stdout and stderr messages, transform the results and serialize them into gtest-compatible xml.