GStreamer Pan Tilt Zoom and Rotate Element - iMX8 Pipelines
| GStreamer PTZR |
|---|
 |
| GstPTZR Basics |
| User Guide |
| Getting Started |
| Examples |
| Contact Us |
|
GStreamer pipelines
The following pipelines will send the resulting video stream via UDP to another computer where it can be displayed.
Important: The i.MX8 uses the GstOpenGL support. It requires EGL and a valid /dev/fbX device. If you are developing, connecting a display will leverage a valid /dev/fbX device. When going to production, the graphics drivers should be patched to leverage the /dev/fbX without a screen connected.
On receiver computer
PORT=5004 gst-launch-1.0 udpsrc port=$PORT ! decodebin ! videoconvert ! identity silent=false ! autovideosink -v
On board
Same input and output resolution with zoom level 1:
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=640,height=480' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
You can change the zoom level by changing "zoom-level".
Magnification (zoom level > 1):
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=640,height=480' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr zoom-level=2 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
You can perform translations relative to the center of the input image by setting the "pan-level" and "tilt-level" properties or perform translations relative to the current position with "pan-delta" and "tilt-delta". The element supports input and output of any resolution and aspect ratio.
Translate right (-1 < pan level < 0):
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr pan-level=-1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
Translate left (0 < pan level < 1):
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr pan-level=1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
Translate down (-1 < tilt level < 0):
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr tilt-level=-1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
Translate up (0 < tilt level < 1):
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr tilt-level=1 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
The property "normalize-translation" (true by default) lets you switch the translation normalization ON and OFF. When translation normalization is true the minimum and maximum translation values are mapped to -1 and 1. On the other hand, when it is set to false, the pan and tilt levels reflect pixels on the input image.
Turn off translation normalization:
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr normalize-translation=false ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
The property "output-reference" (true by default) lets you change the coordinate system used as reference for translations. This is specially useful after performing a rotation. If you want to move over the input image axis, it must be changed to false.
Set input coordinate system as reference for translations:
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr output-reference=false ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
To rotate the input modify the "rotate-level" property (degrees). You can also perform rotation relative to the current angle with "rotate-delta":
Rotate counter-clockwise (rotate level < 0):
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr rotate-level=-100 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
Rotate clockwise (rotate level > 0):
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gst-launch-1.0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr rotate-level=100 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST
Gstd pipelines
Run the element with gstd to change parameters dynamically:
CAPTURE_CAPS='video/x-raw,width=640,height=480,framerate=30/1' IN_CAPS='video/x-raw,format=RGBA,width=640,height=480,framerate=30/1' OUT_CAPS='video/x-raw,width=320,height=240' PORT=5004 HOST=<IP address of the computer> gstd& gstd-client pipeline_create p0 videotestsrc is-live=true num-buffers=-1 ! $IN_CAPS ! imxg2dvideotransform ! $IN_CAPS ! glupload ! glptzr name=ptzr0 ! glcolorconvert ! gldownload ! $OUT_CAPS ! vpuenc_h264 bitrate=1000 ! mpegtsmux alignment=7 ! udpsink port=$PORT host=$HOST gstd-client pipeline_play p0
Change propierties:
gstd-client element_set p0 "ptzr0 zoom-level 0.0" gstd-client element_set p0 "ptzr0 zoom-level 2.0" gstd-client element_set p0 "ptzr0 zoom-level 1.0" gstd-client element_set p0 "ptzr0 tilt-level -1.0" gstd-client element_set p0 "ptzr0 tilt-level 1.0" gstd-client element_set p0 "ptzr0 tilt-level 0.0" gstd-client element_set p0 "ptzr0 pan-level -1.0" gstd-client element_set p0 "ptzr0 pan-level 1.0" gstd-client element_set p0 "ptzr0 pan-level 0.0" gstd-client element_set p0 "ptzr0 tilt-delta -1.0" gstd-client element_set p0 "ptzr0 tilt-delta 1.0" gstd-client element_set p0 "ptzr0 tilt-delta 0.0" gstd-client element_set p0 "ptzr0 pan-delta -1.0" gstd-client element_set p0 "ptzr0 pan-delta 1.0" gstd-client element_set p0 "ptzr0 pan-delta 0.0" gstd-client element_set p0 "ptzr0 rotate-level -90.0" gstd-client element_set p0 "ptzr0 rotate-level 90.0" gstd-client element_set p0 "ptzr0 rotate-level 0.0" gstd-client element_set p0 "ptzr0 rotate-delta -90.0" gstd-client element_set p0 "ptzr0 rotate-delta 90.0" gstd-client element_set p0 "ptzr0 rotate-delta 0.0" gstd-client element_set p0 "ptzr0 normalize-translation false" gstd-client element_set p0 "ptzr0 normalize-translation true" gstd-client element_set p0 "ptzr0 output-reference false" gstd-client element_set p0 "ptzr0 output-reference true"
Stop the pipeline:
gstd-client pipeline_stop p0 gstd-client pipeline_delete p0
Performance
Performance measured by using (framerate):
WIDTH=1280 HEIGHT=720 gst-launch-1.0 videotestsrc num-buffers=1 ! imagefreeze ! "video/x-raw,format=RGBA,width=$WIDTH,height=$HEIGHT" ! queue ! glupload ! glptzr name=ptzr0 zoom-level=3 tilt-level=0.5 pan-level=0.5 rotate-level=30 ! gldownload ! perf print-cpu-load=1 ! queue ! fakesink
CPU:
gst-launch-1.0 videotestsrc is-live=1 num-buffers=100 ! "video/x-raw,format=RGBA,width=$WIDTH,height=$HEIGHT" ! queue ! glupload ! glptzr name=ptzr0 zoom-level=3 tilt-level=0.5 pan-level=0.5 rotate-level=30 ! gldownload ! perf print-cpu-load=1 ! queue ! fakesink
and an i.MX8 MP
| Resolution | CPU Load (overall) | Framerate |
|---|---|---|
| 720p | 15 | 106.6 |
| 1080p | 22 | 74.7 |
| 4K | 29 (overuse) | 31.8 |
The CPU consumption might be lower depending on the framerate and camera connection.
Limitations
The support of GstOpenGL requires EGL to work on i.MX 8. EGL requires a display connected. It is possible to workaround this issue:
- Asserting the HDMI connected pin (hardwire to enable it)
- Connecting a display
- Connecting an HDMI to VGA/RCA converter
- Modifying the graphics driver