Gstreamer QT Overlay
|
Template:Eval SDK Download, Demo Image download and Contact Us buttons |
Overview
GstQtOverlay is a GStreamer plug-in that renders QT graphics on top of a video stream. This rendering occurs directly into the GStreamer video frame buffer, rather than in physical graphics memory, with the resultant video having the QT graphics image merged into the video image. This brings together the best of the two worlds:
- Modularity and extensibility of GStreamer
- Beauty and flexibility of QT
With the RidgeRun GstQtOverlay element, GUIs will be available not only at the video display, but also embedded in file recordings and network streaming, for example.
GstQtOverlay makes heavy use of OpenGL in order to achieve high performance overlay rendering. The pixel blitting and memory transfers are done by GPU and DMA hardware accelerators, allowing real time operation at HD resolutions. As a consequence, the CPU remains free for other processing.
Graphics are modeled using QT's QML, which enables fast, powerful and dynamic GUI implementation. QML is loaded at runtime so there is no need to recompile the GstQtOverlay plug-in in order to change the GUI. Having QML independent of the GStreamer element speeds up development time, eases quick prototyping and reduces time-to-market without impacting performance. A single GstQtOverlay element is able to handle many different GUIs.
Architecture
GstQtOverlay element's architecture is summarized in Figure 1.
- GStreamer
This layer provides the necessary classes to expose GstQtOverlay as a GStreamer element. As such, GstQtOverlay can be linked into any compatible GStreamer pipeline and will participate in the standard pipeline lifecycle. This includes among other, caps and allocator negotiation, and pipeline state changes.
- QT/QML
On top of GStreamer, a QT powered class is instantiated in order to manage graphic resources in a decoupled way. Here, QT event loop is specially processed in order to have both Glib's and QT's event sources active. Finally, a QML engine is created to read and render the user's QML file.
- HAL
The Hardware Abstraction Layer (or simply HAL) is a thin layer that acts as an adapter between QT and OpenGL in order to leverage the platform specific GPU utilities and maximize performance. This layer is the only one that is HW dependent and will be conditionally built via configuration parameters. The HAL ensures that data transfers are HW accelerated so that there is no performance penalty.
- OpenGL
The OpenGL layer will render QT content into the image data using the GPU's parallel power. The blitting to merge the QT image onto a video frame is performed into a copy of the frame image, so that the integrity of the original data is preserved.
Supported platforms
Currently, the following platforms support an Autotools based standalone GstQtOverlay plug-in installation:
- PC: both x86 and x64.
- Nvidia Jetson boards: TX1, TX2, and AGP Xavier.
Also, GstQtOverlay plug-in can be installed using Yocto. Some of the platforms tested with Yocto:
- IMX6-based boards.
- IMX8-based boards.
More platforms are under development. If you wish to sponsor the plug-in for a custom platform or accelerate development for another SoC not listed above, please contact us at support@ridgerun.com
Building GstQtOverlay
GstQtOverlay can be built and installed GstQtOverlay either on Stand-alone or through Yocto. In this section, the two methods are explained and detailed in order to add GstQtOverlay to your system.
Stand-alone build
GstQtOverlay can be built and installed on stand-alone easily using Autotools. This method is recommended for PC-based platforms and Nvidia Jetson boards.
Dependencies
Before starting with the installation, make sure your system has the following dependencies already installed.
QT5 packages
sudo apt-get install \ qtdeclarative5-dev \ qtbase5-dev-tools \ qml-module-qtquick-dialogs \ qml-module-qtquick2 \ qml-module-qtquick-controls \ qt5-default
OpenGL packages
Warning: If you have a platform with Nvidia or ATI GPU, you mesa packages could override your drivers.
sudo apt-get install \ libx11-dev \ xorg-dev \ libglu1-mesa-dev \ freeglut3-dev \ libglew1.5 \ libglew1.5-dev \ libglu1-mesa \ libglu1-mesa-dev \ libgl1-mesa-glx \ libgl1-mesa-dev
GStreamer packages
sudo apt-get install \ gstreamer1.0-x \ libgstreamer1.0-dev \ libgstreamer-plugins-base1.0-dev
Autotools packages
sudo apt install \ autotools-dev \ autoconf
Build steps
When you purchase GstQtOverlay, you will get a Gitlab repository with the source code inside. You need it to build it in your system and add it to your GStreamer plug-ins catalogue. So, in a temporary path, clone the repository using:
git clone git://git@gitlab.com/RidgeRun/orders/${CUSTOMER_DIRECTORY}/gst-qt-overlay.git
Where ${CUSTOMER_DIRECTORY}
contains the name of your customer directory given after the purchase.
Afterwards, find the path where GStreamer looks for plug-ins and libraries (LIBDIR
). Take as a reference to the following table:
Platform | LIBDIR path |
---|---|
PC 32-bits/x86 | /usr/lib/i386-linux-gnu/ |
PC 64-bits/x86 | /usr/lib/x86_64-linux-gnu/ |
Nvidia Jetson | /usr/lib/aarch64-linux-gnu/ |
According to the table, define an environment variable which stores the path accordingly to your system. For example, for an x64 PC, it will be:
LIBDIR=/usr/lib/x86_64-linux-gnu/
Then, do the installation using:
cd gst-qt-overlay ./autogen.sh ./configure --libdir $LIBDIR make sudo make install
Finally, verify the plug-in installation, running:
gst-inspect-1.0 qtoverlay
It should show:
Factory Details: Rank none (0) Long-name Qt Overlay Klass Generic Description Overlays a Qt interface over a video stream Author <http://www.ridgerun.com> Plugin Details: Name qtoverlay Description Overlay a Qt interface over a video stream. ...
Build using Yocto
RidgeRun offers a Yocto layer containing RidgeRun commonly used packages. You can download this package from Meta-Ridgerun Repository.
It contains a recipe to build GstQtOverlay but you need also to purchase GstQtOverlay with full source code, from GstQtOverlay Purchase.
Dependencies
For building GstQtOverlay using Yocto, make sure you have installed the following packages on your target platform:
gstreamer1.0 gstreamer1.0-plugins-base gstreamer1.0-plugins-bad qtbase qtbase-native qtdeclarative imx-gpu-viv libffi gstreamer1.0-plugins-imx
You will also require some extra environment variables before running a GStreamer pipeline with GstQtOverlay. Please, refer to Examples to see how it works.
Build steps
Adding meta-ridgerun to your Yocto build
First you need to copy meta-ridgerun in your Yocto sources directory, where $YOCTO_DIRECTORY
is the location of your Yocto project:
cp -r meta-ridgerun $YOCTO_DIRECTORY/sources/
Then add the RidgeRun meta-layer to your bblayers.conf file. First, go to the build configuration directory
cd $YOCTO_DIRECTORY/build/conf/
open the bblayers.conf file and add the RidgeRun meta layer path $YOCTO_DIRECTORY/sources/meta-ridgerun to BBLAYERS.
Building GstQtOverlay
Once you have access to the repository, open gst-qt-overlay_xxx.bb
in $YOCTO_DIRECTORY/sources/meta-ridgerun/recipes-multimedia/gstreamer/. Make sure to edit gst-qt-overlay_xxx.bb
accordingly to your plug-in version given by the purchase.
Then, modify the following line in SRC_URI
with the correct GstQtOverlay URL by fitting ${CUSTOMER_DIRECTORY}
with your own.
git://git@gitlab.com/RidgeRun/orders/${CUSTOMER_DIRECTORY}/gst-qt-overlay.git;protocol=ssh;branch=${SRCBRANCH};name=base \
Besides, make sure you have added your ssh key to your GitLab account. The recipe will fetch the repository using your SSH key; thus it is necessary to add the key in Gitlab page.
For adding the SSH key go to Settings -> SSH Keys and attach your key. If you do not have any SSH key yet, you can generate and get it using:
ssh-keygen -t rsa -C "your_email@example.com" cat ~/.ssh/id_rsa.pub
Finally, build the recipe:
bitbake gst-qt-overlay
SHH Issue
If your list of known hosts on the PC does not have the GitLab key, you will have fetch errors when trying to build GstQtOverlay recipe.
One easy way to add the key is when cloning the repository for the first time from Gitlab, it will ask if you want to add the key to your list of known hosts.
Example:
git clone git@gitlab.com:RidgeRun/orders/${CUSTOMER_DIRECTORY}/gst-qt-overlay.git
Interacting with the GUI
Supported Capabilities
Currently, the GstQtOverlay element supports the following formats at the Sink and Source pads respectively. More formats may be supported in the future, subject to platform capabilities.
Input
RGBA.
Output
RGBA.
GStreamer Properties
GStreamer properties are used to customize the QT image overlay and to interact with QML.
qml property
The qml property specifies the location of the QML source file to use. This path can be either relative or absolute and must point to a valid/existing file at the moment the pipeline starts up. Some examples include:
gst-launch-1.0 videotestsrc ! qtoverlay qml=/mnt/gui/sources/main.qml ! videoconvert ! autovideosink
gst-launch-1.0 videotestsrc ! qtoverlay qml=../../sources/main.qml ! videoconvert ! autovideosink
If not specified, the element will attempt to load main.qml from the current working directory.
qml-attribute property
QML attributes may be configured at runtime using the qml-attribute property. The property syntax is as follows:
<item name>.<attribute name>:<value>
Item name refers to the QML objectName attribute of an item, rather than its ID. The following QML contains an item named labelMain:
Label { text: "Way cool QT imaging with GStreamer!" font.pixelSize: 22 font.italic: true color: "steelblue" objectName: "labelMain" }
For example, to modify the text attribute of a label named labelMain to display GStreamer with QT imaging is way cool! you would run:
gstd-client element_set qtoverlay qml-attribute "labelMain.text:GStreamer with QT imaging is way cool!"
The qml-attribute will recursively traverse the objects tree to find all the items in the hierarchy with the given objectName.
The objectName attribute is not required to be unique, so multiple items can be modified as once by assigning them the same objectName.
qml-action property
NOT IMPLEMENTED YET
External events can also trigger actions on the QML by using the qml-action property. Similarly, the syntax goes as the following:
<item name>.<method name>()
Again, the qml-action will recursively traverse the objects tree to find all the items in the hierarchy with the given objectName.
For example, consider the following QML snippet:
import QtQuick 2.0 Item { objectName:i1 function sayHi() { console.log("Hello World!") } }
To invoke sayHi you would run:
gstd-client element_set qtoverlay qml-action "i1.sayHi()"
At this time, passing parameters to the invoked functions is not supported. Instead, first set an attribute value, then invoke the function which can access the attribute. There are future plans to support function parameters via Variants or some similar mechanism.
Examples
For these examples, create a file named: "main.qml" with the following content:
import QtQuick 2.0 Item { id: root transform: Rotation { origin.x: root.width/2; origin.y: root.height/2; axis { x: 1; y: 0; z: 0 } angle: 180 } Text { text: "Way cool QT imaging with GStreamer!" font.pointSize: 30 color: "Black" objectName: "labelMain" } }
PC examples
Here is an example to run and test GstQtOverlay:
Display a text overlay
gst-launch-1.0 videotestsrc ! qtoverlay qml=main.qml ! autovideosink
You can stream video over a network and save videos in a file as well.
IMX6 examples
Before running the examples below, make sure you have setup the environment correctly.
export QT_EGLFS_IMX6_NO_FB_MULTI_BUFFER=1 export QT_QPA_PLATFORM=eglfs export DISPLAY=:0.0
Now, you are able to run the following examples:
Display a text overlay
gst-launch-1.0 videotestsrc is-live=true ! 'video/x-raw, width=640, height=480, framerate=30/1' ! queue ! imxg2dvideotransform ! \ queue ! qtoverlay qml=/main.qml ! queue ! imxeglvivsink qos=false sync=false enable-last-sample=false
Saving a video with a text overlay
gst-launch-1.0 videotestsrc is-live=true ! 'video/x-raw, width=640, height=480, framerate=30/1' ! queue ! imxg2dvideotransform ! \ queue ! qtoverlay qml=/main.qml ! queue ! imxipuvideotransform input-crop=false ! \ capsfilter caps=video/x-raw,width=640,height=480,format=NV12 ! imxvpuenc_h264 bitrate=4000 gop-size=15 idr-interval=15 ! \ capsfilter caps=video/x-h264 ! mpegtsmux alignment=7 ! queue ! filesink location=test.mp4 -e
Network streaming with a text overlay
Send stream to a host with IP address $IP
On the IMX6: video source
gst-launch-1.0 videotestsrc is-live=true ! 'video/x-raw, width=640, height=480, framerate=30/1' ! queue ! imxg2dvideotransform ! \ queue ! qtoverlay qml=/main.qml ! queue ! imxipuvideotransform input-crop=false ! \ capsfilter caps=video/x-raw,width=640,height=480,format=NV12 ! imxvpuenc_h264 bitrate=4000 gop-size=15 idr-interval=15 ! \ capsfilter caps=video/x-h264 ! mpegtsmux alignment=7 ! queue ! udpsink async=false sync=false host=$IP port=5012
On the host: video receptor
gst-launch-1.0 udpsrc address=$IP port=5012 ! queue ! tsdemux ! identity single-segment=true ! queue ! decodebin ! \ queue ! fpsdisplaysink sync=false -v
FAQ
- Where can I get GstQtOverlay?
You can purchase it at RidgeRun Online Store. You can contact our engineering team at support@ridgerun.com or post your queries at our Contact Us link.
- Is the source code delivered with the purchase?
Yes. After the purchase the complete source code is delivered.
- Can I use external events to interact with my GUI?
Yes. See GStreamer Properties
- Am I limited to a single QML?
No. As with any regular QML powered application, events can fire up the load of other QML files. This allows the user to build state-pattern like apps with ease.
- Can I use mouse and/or keyboard events?
Not yet. At least not out-of-the-box.
- Is QML enough for my GUI?
Yes (most likely). QML support is great, there's a lot of documentation and examples available. QML is competitive with HTML5 in cross-platform app development.
- Can I create my own custom QML components?
Yes. See the QT QML module guide.
- Can I use animations?
Yes.
- Does my HW support GstQtOverlay?
Check Supported platforms