GStreamer example pipelines
Introduction
The GStreamer plug-in for the RidgeRun Video Stabilization library aims to be easy to use, accepting various video sources and supporting sensors coming from the library. This section will show some pipelines that can be used as examples for both stabilization elements provided by the plug-in:
rvstabilize: Video stabilization using IMU data.rvsdigital: Digital Image Stabilization (DIS) using optical flow over the video stream.
As a note, the rvstabilize element requires a live source, requiring an actual camera to work. Synthetic video sources like video recordings and test sources will not work for testing.
The following video was generated using our GStreamer element rvstabilize with a Xavier NX and a BMI160:
Electronic Video Stabilizer Pipelines
This section contains some examples of the video stabilization library element. To define the properties, please check the Defining Properties section for details.
Webcams and V4L2-compliant Cameras (non-Bayer cameras)
For V4L2-compliant cameras that already produce demosaic videos in formats like YUYV, I420, RGBA, NV12, etc. It includes webcams.
Output to Display
gst-launch-1.0 rrv4l2src ! \
video/x-raw,width=640,height=480 ! videoconvert ! queue ! \
\
rvstabilize undistort=fisheyeopencv imu-sensor=bmi160 \
imu-sensor-device=/dev/i2c-8 imu-data-order="zyx" \
stabilizer-smooth-constant=0.2 undistort-fov-scale=0.7 \
undistort-intrinsic-matrix="<845.63, 0.0, 348.764, 0.0, 849.12470406, 273.80647627, 0.0, 0.0, 1.0>" \
\
! queue ! videoconvert ! ximagesink sync=true
Output to Recording
gst-launch-1.0 rrv4l2src ! \
video/x-raw,width=640,height=480 ! videoconvert ! queue ! \
\
rvstabilize undistort=fisheyeopencv imu-sensor=bmi160 \
imu-sensor-device=/dev/i2c-8 imu-data-order="zyx" \
stabilizer-smooth-constant=0.2 undistort-fov-scale=0.7 \
undistort-intrinsic-matrix="<845.63, 0.0, 348.7640, 0.0, 849.124704060, 273.806476270, 0.0, 0.0, 1.0>" \
\
! queue ! videoconvert ! x264enc ! mp4mux ! \
filesink location=recording.mp4 -e
Tested environment:
- NVIDIA Jetson Xavier NX with Jetpack 5.1.2
- BMI160 Sensor Module Brick over I2C
- Webcam Trust Trino HD
Output to display using Horizon Lock with VQF integrator
ORIENTATION=xYz
CAMERA_OFFSET=26000
gst-launch-1.0 -e rrv4l2src device=/dev/video0 io-mode=2 ! \
'image/jpeg,width=3840,height=2160,framerate=30/1' ! \
nvjpegdec ! queue ! \
nvvidconv ! video/x-raw,width=1920,height=1080 ! queue ! \
rvstabilize enable=true \
undistort=fisheyecuda imu-sensor=icm42600 \
imu-data-order=$ORIENTATION \
stabilizer-smooth-constant=0.5 \
undistort-fov-scale=0.75 \
camera-timestamp-offset=${CAMERA_OFFSET} \
undistort-intrinsic-matrix="<1373.4375194145464, 0.0, 962.9389865762337, 0.0, 1384.791853291431, 518.9402787012929, 0.0, 0.0, 1.0>" \
imu-frequency=200 \
imu-sensor-device=icm42688 \
enable-accelerometer=1 \
integrator=vqf \
smoothing=fixed-horizon-algorithm ! \
video/x-raw,width=1920,height=1080 ! queue ! nvvidconv ! \
xvimagesink display=:0 sync=false
Tested environment:
- NVIDIA Jetson Orin Nano with Jetpack 6.2
- ICM 42688 IMU over SPI
- 4K USB Camera
MIPI Cameras Connected to a Jetson
For Argus-compliant cameras whose outputs can be processed by the ISP to get NV12 or I420 outputs.
Output to Display
gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! \
video/x-raw,width=1920,height=1080,format=RGBA ! queue ! \
\
rvstabilize undistort=fisheyecuda imu-sensor=bmi160 \
imu-sensor-device=/dev/i2c-8 imu-data-order="zyx" \
stabilizer-smooth-constant=0.2 undistort-fov-scale=0.7 \
undistort-intrinsic-matrix="<595.429, 0.0, 962.382, 0.0, 596.052, 543.105, 0.0, 0.0, 1.0>" \
\
! queue ! nveglglessink sync=true
Output to Recording
gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! \
video/x-raw,width=1920,height=1080,format=RGBA ! queue ! \
\
rvstabilize undistort=fisheyeopencv imu-sensor=bmi160 \
imu-sensor-device=/dev/i2c-8 imu-data-order="zyx" \
stabilizer-smooth-constant=0.2 undistort-fov-scale=0.7 \
undistort-intrinsic-matrix="<595.429, 0.0, 962.382, 0.0, 596.052, 543.105, 0.0, 0.0, 1.0>" \
\
! queue ! nvvidconv ! nvv4l2h264enc ! h264parse ! mp4mux ! \
filesink location=recording.mp4 -e
Tested environment:
- NVIDIA Jetson Xavier NX with Jetpack 5.1.2
- BMI160 Sensor Module Brick over I2C
- Camera Sony IMX477
Output to Recording using Horizon Lock with VQF integrator
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=60/1' ! \
queue ! \
nvvidconv flip-method=2 ! \
'video/x-raw,format=RGBA,width=960,height=540,framerate=60/1' ! \
tee name=t \
\
t. ! queue ! \
nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! \
nvv4l2h264enc insert-sps-pps=true iframeinterval=60 bitrate=4000000 ! \
h264parse ! qtmux ! \
filesink async=false location=$HOME/unstabilized_output.mp4 \
\
t. ! queue ! \
rvstabilize enable=true \
camera-timestamp-offset=30000 \
imu-sensor=icm42600 \
imu-sensor-device=icm42688 \
imu-frequency=1000 \
imu-sampling-rate=1000 \
imu-data-order=YXz \
enable-accelerometer=true \
integrator=vqf \
time-constant=3.0 \
stabilizer=fixed-horizon-algorithm \
stabilizer-smooth-constant=0.2 \
undistort=fisheyecuda \
undistort-fov-scale=1.0 \
hw-device-index=0 \
undistort-intrinsic-matrix="<1015.68, 0.00, 500.11, 0.00, 1018.54, 326.49, 0.00, 0.00, 1.00>" \
undistort-distortion-coefficients="<-0.25, 0.80, -3.83, 6.62>" \
\
! queue ! \
nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! \
nvv4l2h264enc insert-sps-pps=true iframeinterval=60 bitrate=4000000 ! \
h264parse ! qtmux ! \
filesink location=$HOME/vqf_stabilized_demo.mp4
Tested environment:
- NVIDIA Jetson Xavier NX with Jetpack 5.1.5
- ICM 42688 IMU over SPI
- Camera Sony IMX477
MIPI Cameras Connected to an RB5
Output to Recording
gst-launch-1.0 qtiqmmfsrc camera=0 ! 'video/x-raw(memory:GBM),width=1920,height=1080,framerate=30/1' ! qtivtransform ! queue ! rvstabilize undistort=brownconradyopencl imu-sensor=rb5imu imu-data-order=YZX stabilizer-smooth-constant=0.4 undistort-intrinsic-matrix="<643.20694632087793, 0.0, 978.87633334351062, 0.0, 643.83993260967520, 552.07822173285456, 0.0, 0.0, 1.0>" undistort-fov-scale=0.9 imu-frequency=50 undistort-distortion-coefficients="<0.018701028199217536, -0.025668731023403092, -0.0014139681623240199, 0.0011170491392074704, 0.0050457195013934846>" ! queue ! qtivtransform ! qtic2venc ! queue ! h264parse ! mp4mux ! filesink location=rvs_test.mp4 -e
Tested environment:
- Qualcomm RB5 developer kit with LU.UM.3.3.1.r1-27400 (Ubuntu 20.04.3 LTS).
- ICM-42688-P IMU Sensor (comes integrated with the RB5 devkit).
- SONY IMX577 camera module (comes with the RB5 devkit).
Defining Properties
The element is configurable through properties. This section will cover some of the mandatory properties and possible values.
imu-sensor
Sets the model of the sensor connected to the platform:
imu-sensor=bmi160
Possible values:
- bmi160 (default)
- rb5imu
- icm42600
- icm45605
- external (.so external modules)
imu-frequency
Sets the frequency of reading of the sensor:
imu-frequency=200
Possible values:
- Integer number indicating Hertz (Hz)
The recommended frequency is often four times the framerate. Some sensors may be limited by the I2C speed. Usually, the maximum for the BMI160 is between 200-300 Hz, so make sure the value you set is available for the sensor.
imu-sensor-device
Sets the path to the device. It depends on the connection with the sensor. For the BMI160, it is I2C.
imu-sensor-device=/dev/i2c-8
The Jetson Xavier NX has the sensor connected to the /dev/i2c-8 if connected as illustrated in BMI160 Setup.
For the ICM42688, it is the IIO identifier.
imu-sensor-device=icm42688
See how it is connected in ICM42688 Setup.
In the case of external sensors, it corresponds to the path to the .so shared object file.
In the case of RB5, you can leave the property to its default value.
imu-data-order
Sets the orientation mapping of the axes of the sensor w.r.t. the axes of the image reference system. It consists of a string with three characters. Please, check the IMU Calibration for reference.
imu-data-order="zyx"
Default: "XYZ"
undistort
Selects the implementation of the undistort in terms of the hardware acceleration.
undistort=fisheyecuda
Possible values:
- fisheyeopencv (default)
- fisheyeopencl
- fisheyecuda
- brownconradyopencv
- brownconradyopencl
Check Supported Platforms, Sensors, and Backends for reference.
undistort-fov-scale
Scales the field-of-view as a sort of zoom to the image. It multiplies the scale factor by the focal length to create a zoom effect. For example, 0.7 will zoom in the image, whereas 3.0 will zoom out the image.
undistort-fov-scale=0.7
Possible values:
- floating-point numbers. Default: 1.5
undistort-intrinsic-matrix
Sets the camera intrinsic matrix for better mapping between the image and world reference systems.
undistort-intrinsic-matrix="<845.62932614f, 0.f, 348.76379855f, 0.f, 849.12470406f, 273.80647627f, 0.f, 0.f, 1.f>"
Possible values:
- array of nine values
stabilizer-smooth-constant
Sets the constant of smoothness of the stabilization filter.
stabilizer-smooth-constant=0.2
Possible values:
- Floating-point numbers between 0 and 1.
Others
There are other properties. Please, check them in Video Stabilizer Element.
Digital Video Stabilizer Pipelines
This section contains example pipelines for the rvsdigital element. The element accepts RGBA input and outputs RGBA video. It exposes properties to select the DIS backend, smoothing window, crop margin, half-resolution analysis, and optical-flow feature tracking settings.
Default OpenCV Backend
The following example uses the default OpenCV backend. It captures from a V4L2 camera, converts the input to RGBA, applies digital stabilization, and displays the stabilized output:
gst-launch-1.0 -e v4l2src device=/dev/video0 ! \
'video/x-raw,width=1280,height=720,framerate=30/1' ! \
videoconvert ! 'video/x-raw,format=RGBA' ! queue ! \
rvsdigital enable=true \
dis-backend=opencv \
dis-smoothing-frames=5 \
dis-crop-margin=0.07 \
dis-half-resolution=true \
of-feature-max-corners=1000 \
of-feature-redetect-interval=3 \
of-flow-num-levels=6 ! \
queue ! videoconvert ! autovideosink sync=false
To record the stabilized output instead of displaying it:
gst-launch-1.0 -e v4l2src device=/dev/video0 ! \
'video/x-raw,width=1280,height=720,framerate=30/1' ! \
videoconvert ! 'video/x-raw,format=RGBA' ! queue ! \
rvsdigital enable=true \
dis-backend=opencv \
dis-smoothing-frames=5 \
dis-crop-margin=0.07 \
dis-half-resolution=true ! \
queue ! videoconvert ! x264enc ! h264parse ! mp4mux ! \
filesink location=digital_stabilized_opencv.mp4
The following example reads from a video file, applies digital stabilization with the OpenCV backend, and saves the stabilized result to a new file:
gst-launch-1.0 -e filesrc location=sample.mp4 ! \
decodebin ! videoconvert ! videoscale ! \
'video/x-raw,format=RGBA,width=640,height=480' ! queue ! \
rvsdigital enable=true \
dis-backend=opencv \
dis-smoothing-frames=5 \
dis-crop-margin=0.07 \
dis-half-resolution=true \
of-feature-max-corners=1000 \
of-feature-redetect-interval=3 \
of-flow-num-levels=6 ! \
queue ! videoconvert ! x264enc ! h264parse ! mp4mux ! \
filesink location=digital_stabilized_opencv.mp4
NVIDIA Jetson CUDA Backend
The following example uses a Jetson MIPI camera through nvarguscamerasrc, converts the stream to RGBA, runs rvsdigital with the CUDA backend, and displays the result:
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! \
nvvidconv ! 'video/x-raw,format=RGBA,width=1920,height=1080' ! queue ! \
rvsdigital enable=true \
dis-backend=cuda \
dis-smoothing-frames=5 \
dis-crop-margin=0.07 \
dis-half-resolution=true \
of-feature-max-corners=1000 \
of-feature-redetect-interval=3 \
of-flow-num-levels=6 ! \
queue ! nvvidconv ! nveglglessink sync=false
To record the stabilized output on Jetson:
gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! \
nvvidconv ! 'video/x-raw,format=RGBA,width=1920,height=1080' ! queue ! \
rvsdigital enable=true \
dis-backend=cuda \
dis-smoothing-frames=5 \
dis-crop-margin=0.07 \
dis-half-resolution=true \
of-feature-max-corners=1000 \
of-feature-redetect-interval=3 \
of-flow-num-levels=6 ! \
queue ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! \
nvv4l2h264enc insert-sps-pps=true bitrate=4000000 ! \
h264parse ! mp4mux ! filesink location=digital_stabilized_cuda.mp4
Tested environment:
- NVIDIA Jetson platform with CUDA support enabled in the RidgeRun Video Stabilization build.
- MIPI camera supported by
nvarguscamerasrc.
Digital Stabilizer Properties
The most commonly used rvsdigital properties are:
- dis-backend: backend used by DIS. Options reported by the element:
opencv,opencl,cuda. Default:opencv. - dis-smoothing-frames: number of frames used for smoothing. Default:
5. - dis-crop-margin: crop margin used to avoid black regions after the stabilization warp. Default:
0.07. - dis-half-resolution: enables half-resolution processing for optical-flow analysis. Default:
true. - of-feature-max-corners: maximum number of corners to track. Default:
1000. - of-feature-redetect-interval: interval for refreshing tracked features. Default:
3. - of-flow-num-levels: number of Pyramid Lucas-Kanade levels. Default:
6. - enable: enables or disables stabilization processing. If disabled, the element copies the input frame to the output. Default:
true.
Implementation note: opencl is reported by the backend enum, but the current rvsdigital element start path initializes the OpenCV and CUDA backends.
For Further information about the properties of the rvsdigital element, check Digital Video Stabilizer Element.
