Qualcomm Robotics RB5/RB6 - Capture and Display GStreamer Pipelines

Qualcomm Robotics RB5/RB6
Introduction
SoM Overview Why the RB5/RB6? Carrier Boards Qualcomm Robotics RB5 developer kit Qualcomm Robotics RB6 platform
Software Development Support
Boot Devices Thundercomm SDK Downloading SDK Installing SDK Generating Image and flashing Linaro Getting Started Flashing Bootloader Flashing Boot Image and Rootfs Special / Debug Modes Setting up board in EDL Mode Setting up board in Fastboot Mode
Development in the Board
Getting into the Board Using ADB Using Serial Console Using SSH Device Setup WiFi setup HDMI setup Update Ubuntu Packages
Build from source
Base Image Customize Image
Capture Subsystem
Hardware Capture Components Software Capture Components Building your own camera sensor driver
Image Processing and Acceleration Software
GPU Profiling Fast CV Installing Hexagon SDK Installing Fast CV Building your own Fast CV application
GStreamer Pipelines
Capture and Display Video Encoding Video Decoding
AI Acceleration
Neural Processing SDK Downloading Requirements Setup SDK Environment Install qtimlesnpe Running Inception v3 Running MobileNetV2 + SSD TensorFlow Example pipeline TensorFlow to DLC Conversion
Sensors
IMU
Demos
Drone Demo Smart Camera
Contact Us

In this section we present some GStreamer pipelines to capture from MIPI interface on the two onboard cameras in the Qualcomm Robotics RB5 development kit (QRB5165). This is also applicable for the RB6.. We also measure key performance indicator for each pipeline. Check our GStreamer Pipelines section to find more information about how we extracted the performance metrics presented in this section. For the latency measurement, we will use the glass-to-glass method.

Main Camera

For the pipelines in this section, we used the IMX577 camera.

1920x1080@30

The pipeline below captures and renders the video to the display connected to the Qualcomm Robotics RB5/RB6. In this pipeline the resolution is of 1920x1080 while the framerate is of 30fps. Table 1 shows the performance metrics for this pipeline.

gst-launch-1.0 qtiqmmfsrc camera=0 ! "video/x-raw(memory:GBM),format=NV12,width=1920,height=1080,framerate=30/1" ! waylandsink sync=true

Table 1: Performance of pipeline at 1920x1080@30 for IMX577.

Operation Mode	CPU (%)	FPS	latency (ms)
Max performance	14.9	29.941	102

1280x720@60

The pipeline below captures and renders the video to the display connected to the Qualcomm Robotics RB5/RB6. In this pipeline the resolution is of 1280x720 while the framerate is of 60fps. Table 2 shows the performance metrics for this pipeline.

gst-launch-1.0 qtiqmmfsrc camera=0 ! "video/x-raw(memory:GBM),format=NV12,width=1280,height=720,framerate=60/1" ! waylandsink sync=true

Table 2: Performance of pipeline at 1280x720@60 for IMX577.

Operation Mode	CPU (%)	FPS	latency (ms)
Max performance	18	59.99	81

Tracking Camera

For the pipelines in this section, we used the OV9282 camera.

640x480@30

The pipeline below captures and renders the video to the display connected to the Qualcomm Robotics RB5/RB6. In this pipeline the resolution is of 640x480 while the framerate is of 30fps. Table 3 shows the performance metrics for this pipeline.

gst-launch-1.0 qtiqmmfsrc camera=1 ! "video/x-raw(memory:GBM),width=640,height=480,framerate=30/1" ! waylandsink sync=true

Table 3: Performance of pipeline at 640x480@30 for OV9282.

Operation Mode	CPU (%)	FPS	latency (ms)
Max performance	13.2	29.86	117

1280x800@60

The pipeline below captures and renders the video to the display connected to the Qualcomm Robotics RB5/RB6. In this pipeline the resolution is of 1280x800 while the framerate is of 60fps. Table 4 shows the performance metrics for this pipeline.

gst-launch-1.0 qtiqmmfsrc camera=1 ! "video/x-raw(memory:GBM),width=1280,height=800,framerate=60/1" ! waylandsink sync=true

Table 4: Performance of pipeline at 1280x800@60 for OV9282.

Operation Mode	CPU (%)	FPS	latency (ms)
Max performance	13.7	29.82	88

Display and encoding

Thw following pipeline uses the IMX577 camera as a source to capture and encode video. We are displaying RAW video in the monitor and at the same time, we are using HW accelerated encoding to create and save a mp4 file. For more information and pipelines about encoding, you can check our Video Encoding section. Table 5 shows the performance metrics for this pipeline.

gst-launch-1.0 -e -v qtiqmmfsrc name=qmmf qmmf.video_0 ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! h264parse ! mp4mux \
    ! queue ! filesink location="/data/two_streams.mp4" qmmf.video_1 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! waylandsink

Table 5: Performance of pipeline for display and encoding.

Operation Mode	CPU (%)	FPS	latency (ms)
Max performance	16.8	29.997	98.1