GStreamer Pan Tilt Zoom and Rotate Element - iMX8 Pipelines

Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help.

GStreamer pipelines

The following pipelines will send the resulting video stream via UDP to another computer where it can be displayed.

Important: The i.MX8 uses the GstOpenGL support. It requires EGL and a valid /dev/fbX device. If you are developing, connecting a display will leverage a valid /dev/fbX device. When going to production, the graphics drivers should be patched to leverage the /dev/fbX without a screen connected.

On receiver computer

PORT=5004
gst-launch-1.0 udpsrc port=$PORT ! decodebin ! videoconvert ! identity silent=false ! autovideosink -v

On board

Same input and output resolution with zoom level 1:

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=640,height=480'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

You can change the zoom level by changing "zoom-level".

Magnification (zoom level > 1):

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=640,height=480'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr zoom-level=2 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

You can perform translations relative to the center of the input image by setting the "pan-level" and "tilt-level" properties or perform translations relative to the current position with "pan-delta" and "tilt-delta". The element supports input and output of any resolution and aspect ratio.

Translate right (-1 < pan level < 0):

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr pan-level=-1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

Translate left (0 < pan level < 1):

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr pan-level=1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

Translate down (-1 < tilt level < 0):

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr tilt-level=-1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

Translate up (0 < tilt level < 1):

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr tilt-level=1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

The property "normalize-translation" (true by default) lets you switch the translation normalization ON and OFF. When translation normalization is true the minimum and maximum translation values are mapped to -1 and 1. On the other hand, when it is set to false, the pan and tilt levels reflect pixels on the input image.

Turn off translation normalization:

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr normalize-translation=false ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

The property "output-reference" (true by default) lets you change the coordinate system used as reference for translations. This is specially useful after performing a rotation. If you want to move over the input image axis, it must be changed to false.

Set input coordinate system as reference for translations:

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr output-reference=false ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

To rotate the input modify the "rotate-level" property (degrees). You can also perform rotation relative to the current angle with "rotate-delta":

Rotate counter-clockwise (rotate level < 0):

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr rotate-level=-100 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

Rotate clockwise (rotate level > 0):

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr rotate-level=100 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST

Gstd pipelines

Run the element with gstd to change parameters dynamically:

CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1'
IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1'
OUT_CAPS='video/x-raw,width=320,height=240'
PORT=5004
HOST=<IP address of the computer>

gstd&
gstd-client pipeline_create p0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr name=ptzr0 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
gstd-client pipeline_play p0

Change propierties:

gstd-client element_set p0 "ptzr0 zoom-level 0.0" 
gstd-client element_set p0 "ptzr0 zoom-level 2.0" 
gstd-client element_set p0 "ptzr0 zoom-level 1.0" 
gstd-client element_set p0 "ptzr0 tilt-level -1.0" 
gstd-client element_set p0 "ptzr0 tilt-level 1.0" 
gstd-client element_set p0 "ptzr0 tilt-level 0.0" 
gstd-client element_set p0 "ptzr0 pan-level -1.0" 
gstd-client element_set p0 "ptzr0 pan-level 1.0" 
gstd-client element_set p0 "ptzr0 pan-level 0.0" 
gstd-client element_set p0 "ptzr0 tilt-delta -1.0" 
gstd-client element_set p0 "ptzr0 tilt-delta 1.0" 
gstd-client element_set p0 "ptzr0 tilt-delta 0.0" 
gstd-client element_set p0 "ptzr0 pan-delta -1.0" 
gstd-client element_set p0 "ptzr0 pan-delta 1.0" 
gstd-client element_set p0 "ptzr0 pan-delta 0.0" 
gstd-client element_set p0 "ptzr0 rotate-level -90.0" 
gstd-client element_set p0 "ptzr0 rotate-level 90.0" 
gstd-client element_set p0 "ptzr0 rotate-level 0.0" 
gstd-client element_set p0 "ptzr0 rotate-delta -90.0" 
gstd-client element_set p0 "ptzr0 rotate-delta 90.0" 
gstd-client element_set p0 "ptzr0 rotate-delta 0.0" 
gstd-client element_set p0 "ptzr0 normalize-translation false" 
gstd-client element_set p0 "ptzr0 normalize-translation true" 
gstd-client element_set p0 "ptzr0 output-reference false" 
gstd-client element_set p0 "ptzr0 output-reference true"

Stop the pipeline:

gstd-client pipeline_stop p0
gstd-client pipeline_delete p0

Performance

Performance measured by using (framerate):

WIDTH=1280
HEIGHT=720
gst-launch-1.0 videotestsrc num-buffers=1 ! imagefreeze ! "video/x-raw,format=RGBA,width=$WIDTH,height=$HEIGHT" ! queue ! glupload ! glptzr name=ptzr0 zoom-level=3 tilt-level=0.5 pan-level=0.5 rotate-level=30 ! gldownload ! perf print-cpu-load=1 ! queue ! fakesink

CPU:

gst-launch-1.0 videotestsrc is-live=1 num-buffers=100 ! "video/x-raw,format=RGBA,width=$WIDTH,height=$HEIGHT" ! queue ! glupload ! glptzr name=ptzr0 zoom-level=3 tilt-level=0.5 pan-level=0.5 rotate-level=30 ! gldownload ! perf print-cpu-load=1 ! queue ! fakesink

and an i.MX8 MP