Jump to content

Video Transformation for the Dragonwing EVK Board

From RidgeRun Developer Wiki


Follow us on: YouTube Twitter LinkedIn Email Share this page

Share This Page

Qualcomm Dragonwing Technical Guide logo
Qualcomm Dragonwing Technical Guide

Qualcomm Dragonwing Technical Guide
Table of Contents [Sticky]




The IQ-9075 provides enables hardware accelerated video transformations and compositions through qti and gl GStreamer elements. Use qtivtransform for supported platform transformations and GStreamer GL for GPU-based operations. This page provides example pipelines along with average and limit performance results. Use it as reference to build high performance video conversion and composition pipelines.


Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help .

Video Transformation

The following elements were tested on the Ubuntu Image (24.04 LTS) provided, and with the packages installed in the GStreamer Section. They were tested for both their average performance as well as their limit performance for different supported formats. The white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 options for the qtiqmmfsrc element were implemented to output a clearer picture but are not strictly necessary for the correct functionality of the pipeline.

Output image generated without any adjustments.
Output image generated after the aforementioned tunings.
Output images that work as a reference for the following experiments.

Qualcomm Hardware Elements

  • qtivtransform
    • Resizes, colorspace converts, flips and rotates video.
    • Example pipeline for qtivtransform element:
gst-launch-1.0 -e qtiqmmfsrc camera=0 white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 ! "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1" ! qtivtransform rotate=1 ! v4l2h264enc capture-io-mode=4 output-io-mode=5 extra-controls="controls,video_bitrate=12000000,video_bitrate_mode=0;" ! h264parse ! mp4mux ! filesink location=qtivtransform-example.mp4
  • Resulting image:
Output generated by the qtivtransform element.
  • qtivcomposer
    • Mix together multiple video streams
    • Example pipeline for qtivcomposer:
gst-launch-1.0 -e qtivcomposer name=comp \
  sink_0::position="<0, 0>" \
  sink_0::dimensions="<960, 1080>" \
  sink_0::zorder=0 \
  sink_1::position="<960, 0>" \
  sink_1::dimensions="<960, 1080>" \
  sink_1::zorder=1 ! \
"video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1" ! waylandsink sync=false qtiqmmfsrc camera=0 ! "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1" ! queue ! comp.sink_0 videotestsrc is-live=true ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! comp.sink_1
  • Resulting image:
Output generated by the qtivcomposer element.
  • qtivoverlay
    • Generic plugin to extract meta like ROI from image buffer and overlaying that data on top of that buffer
    • Example pipeline for qtivoverlay:
gst-launch-1.0 -e qtiqmmfsrc camera=0 white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 ! "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1" ! qtivoverlay images="{(structure)\"Logo,path=logorr.bgra,resolution=<135,134>,destination=<50,50,135,134>;\"}" ! waylandsink sync=false
  • Resulting image:
Output generated by the qtivoverlay element.
  • qtivsplit
    • Split single video stream into multiple streams
    • Example pipeline for qtivsplit:
gst-launch-1.0 -e qtiqmmfsrc camera=0 white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 ! "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1" ! qtivsplit name=split split.src_0 ! "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1" ! queue ! waylandsink sync=false split.src_1 ! "video/x-raw,format=NV12,width=640,height=360,framerate=30/1" ! queue ! v4l2h264enc capture-io-mode=4 output-io-mode=5 extra-controls="controls,video_bitrate=4000000,video_bitrate_mode=0;" ! h264parse ! mp4mux ! filesink location=qtivsplit-example.mp4
  • Resulting images:
Output generated by the qtivcomposer element (Left is the lower resolution image and on the right is the 1080p image.)
  • Performance measurements for the qtivtransform. For these measurements, the copy element was necessary to tackle issues with how the imagefreeze element interacts with the video transformation, thus the following pipeline was used for limit performance testing:
gst-launch-1.0 videotestsrc pattern=black num-buffers=1 ! "video/x-raw,format={FORMAT},width={WIDTH},height={HEIGHT}" ! imagefreeze ! queue ! copy ! queue ! qtivtransform flip-horizontal=1 ! perf print-cpu-load=true ! fakesink sync=0
Format Resolution Average Performance Limit Performance
%MEM RSS (KB) GPU Utilization (%) CPU (%) FPS Mean FPS
RGBA 1280x720 0.2 101544 2.39 17 478.680 459.720
RGBA 1920x1080 0.2 100448 3.76 17 328.767 342.768
RGBA 3840x2160 0.2 100536 5.27 16 124.819 125.379
NV12 1280x720 0.2 100960 5.21 16 491.269 478.451
NV12 1920x1080 0.2 100568 2.86 15 400.543 372.558
NV12 3840x2160 0.2 100428 3.74 15 208.700 202.787

GL Transformation Elements

  • glcolorconvert
    • Converts between color spaces using OpenGL shaders
    • Example pipeline for glcolorconvert:
gst-launch-1.0 -e qtiqmmfsrc camera=0 white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 ! 'video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1' ! queue max-size-buffers=4 leaky=downstream ! glupload ! glcolorconvert ! 'video/x-raw(memory:GLMemory),format=RGBA' ! gldownload ! videoconvert ! waylandsink sync=false
  • Resulting image:
Output generated by the glcolorconvert element.
  • glvideoflip
    • Flip video on the GPU
    • Example pipeline for glvideoflip:
gst-launch-1.0 -e qtiqmmfsrc camera=0 white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 ! 'video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1' ! queue max-size-buffers=4 leaky=downstream ! glupload ! glcolorconvert ! 'video/x-raw(memory:GLMemory),format=RGBA' ! glvideoflip video-direction=vert ! gldownload ! videoconvert ! waylandsink sync=false
  • Resulting image:
Output generated by the glvideoflip element.
  • gltransformation
    • Transform video on the GPU
    • Example pipeline for gltransformation:
gst-launch-1.0 -e qtiqmmfsrc camera=0 white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 ! 'video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1' ! queue max-size-buffers=4 leaky=downstream ! glupload ! glcolorconvert ! gltransformation rotation-z=45 ! gldownload ! waylandsink sync=false
  • Resulting image:
Output generated by the gltransformation element.
  • glupload
    • Uploads data into OpenGL. This element is essential when working with GPU processing. As seen in previous examples, the glupload element transforms video buffers into GLMemory so they can be processed by OpenGL-based elements.
  • gldownload
    • Downloads data from OpenGL. This element is essential for retrieving GPU-processed video buffers back into system memory. As seen in previous examples, gldownload converts GLMemory buffers into regular video buffers so they can be processed by non-OpenGL elements or displayed by standard sinks.
  • glvideomixer
    • OpenGL video_mixer bin
    • Example pipeline for glvideomixer:
gst-launch-1.0 -e glvideomixer name=mix sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=1080 sink_0::zorder=0 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=1080 sink_1::zorder=1 ! gldownload ! videoconvert ! waylandsink sync=false qtiqmmfsrc camera=0 white-balance-mode=6 focus-mode=1 iso-mode=0 exposure-mode=0 manual-exposure-time=43333333 ! "video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1" ! queue ! glupload ! glcolorconvert ! "video/x-raw(memory:GLMemory),format=RGBA" ! mix.sink_0 gltestsrc pattern=smpte ! "video/x-raw(memory:GLMemory),format=RGBA,width=1920,height=1080,framerate=30/1" ! queue ! mix.sink_1
  • Resulting image:
Output generated by the glvideomixer element.


  • Performance measurements for gltransformation. The following pipeline was used for limit performance testing:
gst-launch-1.0 videotestsrc pattern=black num-buffers=1 ! "video/x-raw,format={FORMAT},width={WIDTH},height={HEIGHT}" ! imagefreeze ! queue ! glupload ! glcolorconvert ! gltransformation rotation-z=45 ! perf print-cpu-load=true ! fakesink
Format Resolution Average Performance Limit Performance
%MEM RSS (KB) GPU Utilization (%) CPU (%) FPS Mean FPS
RGBA 1280x720 0.3 118932 2.31 20 901.855 921.017
RGBA 1920x1080 0.4 145700 4.35 20 778.891 564.690
RGBA 3840x2160 0.7 259168 5.59 16 222.039 227.266
NV12 1280x720 0.2 103060 3.19 21 381.466 287.482
NV12 1920x1080 0.3 111500 5.99 18 286.795 267.213
NV12 3840x2160 0.4 146604 7.16 17 188.280 198.075

FAQ

Which element should scale or rotate video?
The element choice depends on the required input/output caps, mainly colorspace and memory type. The qtivtransform element supports camera-native formats like NV12, and Qualcomm GBM buffers, which enables zero-copy when the elements downstream use this same type of memory, for example the hardware-accelerated encoder. The gltransformation element is useful when the buffers are already in GLMemory or when more operations in this memory are required downstream.

Related pages


Qualcomm Dragonwing Technical Guide logo
Qualcomm Dragonwing Technical Guide
Back to top ↑


Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=Qualcomm_Dragonwing_Technical_Guide/Multimedia_Support/Video/Video&oldid=86946"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.