Qualcomm Robotics RB5/RB6 - GStreamer pipelines

Qualcomm Robotics RB5/RB6
Introduction
SoM Overview Why the RB5/RB6? Carrier Boards Qualcomm Robotics RB5 developer kit Qualcomm Robotics RB6 platform
Software Development Support
Boot Devices Thundercomm SDK Downloading SDK Installing SDK Generating Image and flashing Linaro Getting Started Flashing Bootloader Flashing Boot Image and Rootfs Special / Debug Modes Setting up board in EDL Mode Setting up board in Fastboot Mode
Development in the Board
Getting into the Board Using ADB Using Serial Console Using SSH Device Setup WiFi setup HDMI setup Update Ubuntu Packages
Build from source
Base Image Customize Image
Capture Subsystem
Hardware Capture Components Software Capture Components Building your own camera sensor driver
Image Processing and Acceleration Software
GPU Profiling Fast CV Installing Hexagon SDK Installing Fast CV Building your own Fast CV application
GStreamer Pipelines
Capture and Display Video Encoding Video Decoding
AI Acceleration
Neural Processing SDK Downloading Requirements Setup SDK Environment Install qtimlesnpe Running Inception v3 Running MobileNetV2 + SSD TensorFlow Example pipeline TensorFlow to DLC Conversion
Sensors
IMU
Demos
Drone Demo Smart Camera
Contact Us

GStreamer is a multimedia framework based in pipelines that link together a variety of media processing systems. It enables quick prototyping of multimedia projects . In this section, we will see some examples of pipelines to capture from the camera on the Qualcomm Robotics RB5/RB6 (QRB5165) and how to work with it.

The following subsections will show example pipelines:

• The Capture and Display section shows pipelines to capture from the main camera and display it.
• The Video Encoding section shows pipelines on how to capture from the main camera, encode the video and save it in a file.
• The Video Decoding section shows pipelines on how to decode video files and display them.

Note

NOTE: All the pipelines described in these sections were tested using GStreamer version 1.20.5.

Performance Measurement

The performance measurements are extracted with the GStreamer pipeline running in the system.

CPU Usage

• For the CPU percentage utilization, we use the top command in Ubuntu by running the following:

top

And then pressing: Shift+i. This will give the average CPU usage per core. For the measurement, we compute the average over 30 samples of the utilization values for the CPU cores.

FPS

• For the FPS, we use we use the gst-perf plugin from RidgeRun. The plugin computes the FPS mean. For the measurement, we compute the average over 30 samples of the FPS mean values. The following example shows a pipeline using the perf element:

gst-launch-1.0 qtiqmmfsrc ! "video/x-raw(memory:GBM),format=NV12,width=1920,height=1080,framerate=30/1" ! perf ! waylandsink -v

Latency

To measure latency, we use two methods. The first one is a measure of glass-to-glass latency, and the other one measures the time it takes a buffer to go from the source to the sink of the pipeline.

• For the first method, we use a stopwatch. The camera from the QRB5/RB6 is setup to look at the stopwatch and we measure the time difference from the actual stopwatch time to the time displayed in the monitor. We also use this method for encoding by checking the last time saved in video by the pipeline and comparing it to the one of the stopwatch.

• For the second method, we use GStreamer's tracers, thar provide tracinf information of GStreamer applications. We will use the latency tracer, this metric shows the time needed by a buffer to travel from the src element to the sink one of the pipeline. For the measurement, we compute the average over 30 samples of the latency value. The following example shows a pipeline using the latency tracer:

GST_DEBUG="GST_TRACER:7" GST_TRACERS=latency gst-launch-1.0 qtiqmmfsrc ! "video/x-raw(memory:GBM),format=NV12,width=1920,height=1080,framerate=30/1" ! waylandsink -v