In our last article, as part of our investigation into different Graphical User Interface (GUI) options for the next European Rover Challenge (ERC) we looked at a proof of concept for using QML and Qt5 with ROS. In this article we will continue with that proof of concept by creating a custom QML component that streams a ros sensor_msgs/Video topic, and adding it to the window we created in the previous article.

Setting up our Catkin Packages

1. In qt-creator reopen the workspace project you used for the last tutorial.
2. For this project we need an additional ROS package for our shared library that will contain our custom QML Video Component. We need this so the qt-creator design view can deal with our custom component. In the project window, right click on the “src” folder, and select “add new”.
3. Select “ROS>Package” and then fill in the details so they match the screenshot below. We’ll call this package “ros_video_components” and  the Catkin dependencies are “qt_build roscpp sensor_msgs image_transport” 
4. Click “next” and then “finish”
5. Open up the CMakeLists.txt file for the ros_video_components package, and replace it with the following file.

   ##############################################################################
   # CMake
   ##############################################################################
   
   cmake_minimum_required(VERSION 2.8.3)
   project(ros_video_components)
   
   ##############################################################################
   # Catkin
   ##############################################################################
   
   # qt_build provides the qt cmake glue, roscpp the comms for a default talker
   find_package(catkin REQUIRED COMPONENTS qt_build roscpp sensor_msgs image_transport)
   include_directories(include ${catkin_INCLUDE_DIRS})
   # Use this to define what the package will export (e.g. libs, headers).
   # Since the default here is to produce only a binary, we don't worry about
   # exporting anything.
   catkin_package(
       CATKIN_DEPENDS qt_build roscpp sensor_msgs image_transport
       INCLUDE_DIRS include
       LIBRARIES RosVideoComponents
   )
   
   ##############################################################################
   # Qt Environment
   ##############################################################################
   
   # this comes from qt_build's qt-ros.cmake which is automatically
   # included via the dependency ca ll in package.xml
   find_package(Qt5 COMPONENTS Core Qml Quick REQUIRED)
   
   ##############################################################################
   # Sections
   ##############################################################################
   
   file(GLOB QT_RESOURCES RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} resources/*.qrc)
   file(GLOB_RECURSE QT_MOC RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} FOLLOW_SYMLINKS include/ros_video_components/*.hpp)
   
   QT5_ADD_RESOURCES(QT_RESOURCES_CPP ${QT_RESOURCES})
   QT5_WRAP_CPP(QT_MOC_HPP ${QT_MOC})
   
   ##############################################################################
   # Sources
   ##############################################################################
   
   file(GLOB_RECURSE QT_SOURCES RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} FOLLOW_SYMLINKS src/*.cpp)
   
   ##############################################################################
   # Binaries
   ##############################################################################
   add_library(RosVideoComponents ${QT_SOURCES} ${QT_RESOURCES_CPP} ${QT_FORMS_HPP} ${QT_MOC_HPP})
   qt5_use_modules(RosVideoComponents Quick Core)
   target_link_libraries(RosVideoComponents ${QT_LIBRARIES} ${catkin_LIBRARIES})
   target_include_directories(RosVideoComponents PUBLIC include)
   
   

   Note: This code is based on the auto generated CMakeList.txt file provided by the qt-create ROS package.
   This is similar to what we did for the last example, but with a few key differences

   catkin_package(
       CATKIN_DEPENDS qt_build roscpp sensor_msgs image_transport
       INCLUDE_DIRS include
       LIBRARIES RosVideoComponents
   )
   

   This tells catkin to export the RosVideoComponents build target as a library to all dependencies of this package.

   Then in this section we tell catkin to make a shared library target called “RosVideoComponents” that links the C++ source files with the Qt MOC/Header files, and the qt resources. Rather than a ROS node.

   add_library(RosVideoComponents ${QT_SOURCES} ${QT_RESOURCES_CPP} ${QT_FORMS_HPP} ${QT_MOC_HPP})
   qt5_use_modules(RosVideoComponents Quick Core)
   target_link_libraries(RosVideoComponents ${QT_LIBRARIES} ${catkin_LIBRARIES})
   target_include_directories(RosVideoComponents PUBLIC include)
   

6. Next we need to fix our package.xml file, the qt-creator plugin has a bug where it puts all the ROS dependecies in one build_depends and run_depends tag, rather than putting them seperatly. You need to seperate them like so:

     <buildtool_depend>catkin</buildtool_depend>
     <buildtool_depend>catkin</buildtool_depend>
     <build_depend>qt_build</build_depend>
     <build_depend>roscpp</build_depend>
     <build_depend>image_transport</build_depend>
     <build_depend>sensor_msgs</build_depend>
     <build_depend>libqt4-dev</build_depend>
     <run_depend>qt_build</run_depend>
     <run_depend>image_transport</run_depend>
     <run_depend>sensor_msgs</run_depend>
     <run_depend>roscpp</run_depend>
     <run_depend>libqt4-dev</run_depend>
   

7. Again we need to create src/ resources/ and include/ros_video_components folders in the package folder.
8. We also need to make some changes to our gui project to depend on the library we generate. Open up the CMakeLists.txt file for the gui package and replace the following line:

   find_package(catkin REQUIRED COMPONENTS qt_build roscpp sensor_msgs image_transport)

   with

   find_package(catkin REQUIRED COMPONENTS qt_build roscpp sensor_msgs image_transport ros_video_components)

9. Then add the following lines to the gui package’s package.xml,

   <build_depend>ros_video_components</build_depend>
   <run_depend>ros_video_components</run_depend>
   

10. In the file browser create src and include/ros_video_components folders in the ros_video_components folder.

Building the Video Streaming Component

When we are using the rover the primary purpose the GUI serves in most ERC tasks is displaying camera feed information to users. Thus it felt appropriate to use streaming video from ROS as a proof of concept to determine if QML and Qt5 would be an appropriate technology choice.

We will now look at building a QML component that subscribes to a ROS image topic, and displays the data on screen.

1. Right click on the src folder of the ros_video_components folder, and select “Add New.”
2. We first need to create a class for our qt component so select,  “C++>C++ Class”
3. We’ll call our class “ROSVideoComponent” and it has the custom base class “QQuickPaintedItem.” We’ll also need to select that we want to “Include QObject” and adjust the path of the header file so the compiler can find it. Make sure your settings match those in this screenshot:
   
4. Open up the header file you just created and update it to match the following

    
   #ifndef ROSVIDEOCOMPONENT_H
   #define ROSVIDEOCOMPONENT_H
   
   #include <QQuickPaintedItem>
   #include <ros/ros.h>
   #include <image_transport/image_transport.h>
   #include <sensor_msgs/Image.h>
   #include <QImage>
   #include <QPainter>
   
   class ROSVideoComponent : public QQuickPaintedItem {
       // this marks the component as a Qt Widget
       Q_OBJECT
       
       public:
           ROSVideoComponent(QQuickItem *parent = 0);
   
           void paint(QPainter *painter);
           void setup(ros::NodeHandle * nh);
   
       private:
           void receiveImage(const sensor_msgs::Image::ConstPtr & msg);
   
           ros::NodeHandle * nh;
           image_transport::Subscriber imageSub;
           // these are used to store our image buffer
           QImage * currentImage;
           uchar * currentBuffer;
           
           
   };
   
   #endif // ROSVIDEOCOMPONENT_H
   

   Here, QQuickPaintedItem is a Qt class that we can override to provide a QML component with a custom paint method. This will allow us to render our ROS video frames.
   Also in the header file we have a setup function which we use to initialise our ROS subscriptions since we don’t control where the constructor of this class is called, and our conventional ROS subscriber callback.

5. Open up the ROSVideoComponent.cpp  file add change it so it looks like this:

    
   #include <ros_video_components/ROSVideoComponent.hpp>
   
   ROSVideoComponent::ROSVideoComponent(QQuickItem * parent) : QQuickPaintedItem(parent), currentImage(NULL), currentBuffer(NULL) {
   
   }
   

   Here we use an intialiser list to call our parent constructor, and then initialise our currentImage and currentBuffer pointers to NULL. The latter is very important as we use it to check if we have received any ROS messages.

6. Next add a “setup” function:

   void ROSVideoComponent::setup(ros::NodeHandle *nh) {
       image_transport::ImageTransport imgTrans(*nh);
       imageSub = imgTrans.subscribe("/cam0", 1, &ROSVideoComponent::receiveImage, this, image_transport::TransportHints("compressed"));
       ROS_INFO("setup");
   }
   

   This function takes in a pointer to our ROS NodeHandle, and uses it to create a subscription to the “/cam0” topic.  We use image transport, as is recomended by ROS for video streams, and direct it to call the recieveImage callback.

7. And now we implement  said callback:

   void ROSVideoComponent::receiveImage(const sensor_msgs::Image::ConstPtr &msg) {
       // check to see if we already have an image frame, if we do then we need to delete it
       // to avoid memory leaks
       if(currentImage) {
           delete currentImage;
       }
   
       // allocate a buffer of sufficient size to contain our video frame
       uchar * tempBuffer = (uchar *) malloc(sizeof(uchar) * msg->data.size());
       
       // and copy the message into the buffer
       // we need to do this because the QImage api requires the buffer we pass in to continue to exist
       // whilst the image is in use, but the msg and it's data will be lost once we leave this context.
       memcpy(tempBuffer, msg->data.data(), msg->data.size());
       
       // we then create a new QImage, this code below matches the spec of an image produced by the ros gscam module
       currentImage = new QImage(tempBuffer, msg->width, msg->height, QImage::Format_RGB888);
       
       ROS_INFO("Recieved");
       
       // We then swap out of buffer to avoid memory leaks
       if(currentBuffer) {
           delete currentBuffer;
           currentBuffer = tempBuffer;
       }
       // And re-render the component to display the new image.
       update();
   }
   

8. Finally we override the paint method

   
   void ROSVideoComponent::paint(QPainter *painter) {
       if(currentImage) {
           painter->drawImage(QPoint(0,0), *(this->currentImage));
       }
   }
   

9. We now have our QML component, and you can check that everything is working as intended by building the project (hammer icon in the bottom right or the IDE or using catkin_make). In order to use it we must add it to our qml file, but first since we want to be able to use it in qt-creator’s design view we need to add a plugin class.
10. Right click on the src folder and select “Add New” again.
11. Then select “C++>C++ Class.”
12. We’ll call this class OwrROSComponents, and use the following settings:
13. Replace the header file so it looks like this

    #ifndef OWRROSCOMPONENTS_H
    #define OWRROSCOMPONENTS_H
    
    #include <QQmlExtensionPlugin>
    
    class OWRRosComponents : public QQmlExtensionPlugin {
        Q_OBJECT
        Q_PLUGIN_METADATA(IID "bluesat.owr")
    
        public:
            void registerTypes(const char * uri);
    };
    
    #endif // OWRROSCOMPONENTS_H
    

14. Finally make the OwrROSComponents.cpp file look like this

    #include "ros_video_components/OwrROSComponents.hpp"
    #include "ros_video_components/ROSVideoComponent.hpp"
    
    void OWRRosComponents::registerTypes(const char *uri) {
        qmlRegisterType<ROSVideoComponent>("bluesat.owr",1,0,"ROSVideoComponent");
    }
    

15. And now we just need to add it our QML and application code. Lets do the QML first. At the top of the file (in edit view) add the following line:

    import bluesat.owr 1.0
    

16. And just before the final closing bracket add this code to place the video component below the other image

    ROSVideoComponent {
       // @disable-check M16
       objectName: "videoStream"
       id: videoStream
       // @disable-check M16
       anchors.bottom: parent.bottom
       // @disable-check M16
       anchors.bottomMargin: 0
       // @disable-check M16
       anchors.top: image1.bottom
       // @disable-check M16
       anchors.topMargin: 0
       // @disable-check M16
       width: 320
       // @disable-check M16
       height: 240
    }
    

    This adds our custom “ROSVideoComponent” who’s type we just registered in the previous steps to our window.

    Note: the @disable-check M16 prevents qt-creator from getting confused about our custom component, which it doesn’t detect properly. This is an unfortunate limit of using cmake (catkin) rather than qt’s own build system.

17. Then because Qt’s runtime and qt-creator use different search paths we also need to register the type on the first line of our MainApplication::run() function

    qmlRegisterType<ROSVideoComponent>("bluesat.owr",1,0,"ROSVideoComponent");
    

18. Finally we need to add the following lines to the end of our run function in main application to connect our video component to our NodeHandle

    ROSVideoComponent * video = this->rootObjects()[0]->findChild<ROSVideoComponent*>(QString("videoStream"));
    video->setup(&nh);
    

    And the relevant #include

    #include <ros_video_components/ROSVideoComponent.hpp>
    

19. To test it publish a video stream using your preferred ros video library.
    For example if you have the ROS gscam library setup and installed you could run the following to stream video from a webcam:

    export GSCAM_CONFIG="v4l2src device=/dev/video0 ! videoscale ! video/x-raw,width=320,height=240 ! videoconvert"
    rosrun gscam gscam __name:=camera_1 /camera/image_raw:=/cam0

Conclusion

So in our previous post we learnt how to setup Qt and QML in ROS’s build system, and get that all working with the Qt-Creator IDE. Then this time we built on that system to develop a widget that takes ROS video data and renders it to the screen, demonstrating how to integrate ROS’s message system into a Qt/QML environment.

The code in this tutorial forms the basis of BLUEsat’s new rover user interface, which is currently in active development. You can see the current progress on our github, where there should be a number of additional widgets being added in the near future.  If you want to learn more about the kind of development we do at BLUEsat or are a UNSW student interested in joining feel free to send an email to info@bluesat.com.au.

Acknowledgements

Some of the code above is based on a Stack Overflow answer by Kornava, about how to created a custom image rendering component, which can be found here.