GStreamer Video Stabilizer for NVIDIA Jetson Boards - User Guide
| GStreamer Video Stabilizer |
|---|
 |
| Video Stabilization Basics |
| User Guide |
| Getting Started |
| Examples |
| Performance |
| Contact Us |
Algorithm Overview
GstNvStabilize can be linked-in into a video pipeline like any other element. It receives RGBA frames and, based on the user configuration, outputs stabilized RGBA frames as well. It does so by estimating the motion between sequential frames and warping the current image so that the region of interest (ROI) remains as stable as possible. The following figure exemplifies this process:
In the animation above, the user specifies the ROI (red rectangle). The algorithm will compute an affine transformation matrix such that, when applied, the region within the rectangle will remain as stable as possible. The animation shows how, for the given movement, the compensation matrix estimates that a horizontal and vertical translation is enough to minimize movement. This, however, is for simplicity and visualization purposes, affine matrices can provide up to 6 degrees of freedom including:
- Horizontal and vertical translations
- Anisotropic scaling (scaling width and height independently)
- Rotation
- Shearing
Configuring the Stabilization
There are three parameters exposed to configure the algorithm:
- Temporal image filter size: Amount of buffers that are accounted for temporal filtering. The higher the more stable the resulting image but at the cost of higher latency.
- Region of interest: The size and/or region of the resulting stabilized image.
- Compensation limit: If the movement is too big, the compensation may be such that the resulting image starts displaying black borders. The limit inhibits this from happening, but may be disabled for debugging purposes.
The following sections elaborate in more detail on these parameters.
Temporal Filter Size
This parameter controls the number of images that are used on the internal smoothing filter. A higher value will stabilize more the resulting sequence but will introduce more latency to the pipeline.
This parameter is controlled as a Gstreamer property. The following table summarizes the possible values.
| Property | Value |
|---|---|
| GStreamer property name | queue-size |
| Type | Integer |
| Default value | 5 |
| Allowed values | 1 - 6 |
As an example, the following pipeline configures an internal filter of the maximum of 6 frames:
gst-launch-1.0 autovideosrc ! videoconvert ! nvstabilize '''queue-size=6''' ! videoconvert ! autovideosink
Region of Interest
This parameter controls the resulting size and region of the stabilized image (the red rectangle in the animation above). There are two ways to control this setting:
- Via the crop-margin GStreamer property. Use this if you want the resulting cropped region scaled to the same size as the input image.
- Via input/output image sizes in the GStreamer caps. Use this if you want the resulting cropped region to retain its unscaled resolution.
The caps will always have priority over the crop-margin property. Said in other words, the property is only taken into account if the input and output caps have the same size. If no caps are explicitly specified in the pipeline, GstNvStabilize will try to negotiate equal input and output image sizes.
Crop Region Property
When the negotiated input and output image sizes are equal (the default), the algorithm will use the crop margin configured in the GStreamer property.
The crop-margin property specifies the percentage (normalized from 0 to 1) of the original image to crop from the image borders. The resulting region will then be scaled to match the input size. The following figure describes graphically this concept for two different crop margins of 0.125 and 0.25 respectively.
From the diagram, it can be seen that the allowed values range from 0 to 0.5. The following table describes them with further detail:
| Property | Value |
|---|---|
| GStreamer property name | crop-margin |
| Type | Double |
| Default value | 0.07 |
| Allowed values | 0 - 0.5 (exclusive) |
As an example, the following pipeline configures a crop-margin of 0.125.
gst-launch-1.0 autovideosrc ! videoconvert ! nvstabilize '''crop-margin=0.125''' ! videoconvert ! autovideosink
Input/Output Caps
If the input and output caps differ in the image size, the crop-margin property will be ignored and, instead, the ROI will be defined by the output caps. The resulting image will not be scaled up. An additional benefit is that the resulting size can be of a different aspect ratio than the input. The following figure shows this concept:
As of now, the ROI will be automatically set to the center of the original image. In the future, this may be parametrized. Please contact us if you are interested in sponsoring this effort.
The following table summarizes the valid configurations:
| Property | Value |
|---|---|
| Allowed input caps | video/x-raw,format=RGBA,width=[1 - MAX INT],height=[1 - MAX INT] |
| Allowed output caps | video/x-raw,format=RGBA,width=[1 - INPUT WIDTH],height=[1 - INPUT HEIGHT] |
| Default value | INPUT = OUTPUT |
As an example, the following pipeline takes a 1280x720 ROI from the original 1920x1080 image:
gst-launch-1.0 autovideosrc ! videoconvert ! '''video/x-raw,width=1920,height=1080''' ! nvstabilize ! '''video/x-raw,width=1280,height=720''' ! videoconvert ! autovideosink
Plug-in Details
The following listing shows the output of the gst-inspect-1.0 command.
gst-inspect-1.0 nvstabilize
Factory Details:
Rank none (0)
Long-name NV Video Stabilizer
Klass filter
Description Video Stabilizer for NVIDIA Jetson boards
Author Michael Gruner <michael.gruner@ridgerun.com>
Plugin Details:
Name nvstabilize
Description Video Stabilizer for NVIDIA Jetson boards
Filename /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvstabilize.so
Version 0.4.0
License Proprietary
Source module gst-nvstabilize
Binary package RidgeRun
Origin URL https://www.ridgerun.com/
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseTransform
+----GstVideoFilter
+----GstNvstabilize
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)RGBA, (string)RGBx }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)RGBA, (string)RGBx }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "nvstabilize0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events
flags: readable, writable
Boolean. Default: true
queue-size : Queue size
flags: readable, writable
Integer. Range: 1 - 6 Default: 5
crop-margin : Max width/height to crop (0 = disabled, negative = inferred from caps)
flags: readable, writable
Double. Range: -1.797693e+308 - 0.5 Default: 0.07
In order to see how to use those capabilities, you can go to the GStreamer pipelines examples section: GStreamer-Pipelines