GStreamer / Add Support for Video Timestamps

< >

RidgeRun Video Stabilization Library

Table of Contents [Sticky]

• Basics and Foundation
  • What is Video Stabilization
  • Video Stabilization Process using IMU
  • Stabilization Algorithms with IMU
• Getting Started
  • Supported Platforms, Sensors and Backends
  • Evaluating the Library
  • Getting the Code
  • Building the Library
• API Reference
  • Library Architecture
  • Adding New Sensors
  • Adding Algorithms
  • Adding Stabilization Algorithm
  • Adding Backends
  • API Documentation
  • Examples Guidebook
• Library Integration for IMU
  • Preparing the Video
  • Preparing the IMU Measurements
  • Measurement Integration
  • Interpolation
  • Computing the Stabilization
  • Video Undistortion
  • Example Application
• GStreamer
  • Add Support for Video Timestamps
  • Add Support for IMU Sensors
  • Video Stabilizer Element
  • Example Pipelines
• Useful Links
  • BMI160 Setup
  • Performance
• Contact Us

Introduction

GStreamer requires all the elements and bins of a pipeline to be synchronized with a common clock. The clock is usually proposed and negotiated during the auto-negotiation process at startup. However, the most common clock is the system clock, which can be non monotonic and can be affected by NTP changes.

Supported Platforms

The RidgeRun Video Stabilization library aims to be platform agnostic. Nevertheless, it integrates optimizations for OpenCL and CUDA-enabled devices.

Apart from that, the RidgeRun Video Stabilization library also requires changes to the GStreamer live video sources, such as v4l2src, nvarguscamerasrc and qtiqmmfsrc, due to the requirement of the capture timestamp, as described in Library Integration for IMU/Preparing the Video.

With that in mind, the GStreamer support for the RidgeRun Video Stabilization library includes:

• NVIDIA Jetson: Nano, TX2, Xavier and Orin
• NXP: i.MX 8, i.MX 95
• Qualcomm: RB5

The supported resolution on these platforms depends on the hardware capabilities. You can check the performance in Performance.

Video Sources

This section includes the procedure to add support to the different video sources. All of them require the installation of the gstcameradrivermeta.

Installing the GstCameraDriverMeta

The gstcameradrivermeta is a metaclass for GStreamer that encapsulates the camera timestamps and sequence numbers. It is mandatory for the stabilization process because of its reliability on monotonic timestamps.

The meta is attached to each frame produced by video sources until reaching the video stabilization element, which decodes the meta and uses the timestamps to synchronize the IMU sensor with the current frame.

The installation procedure can be found here: Installing the GstCameraDriverMeta

v4l2src

We have a port of the Video For Linux 2 (v4l2) GStreamer plug-ins with support from GStreamer 1.14 up to 1.18 and 10-bit bayer for MIPI cameras. It is possible to evaluate a non-production version of the video source using a patched version that is available in this repo.

Note

This element is a reference implementation. Please adapt it to your GStreamer version based on the changes from this repo before deploying to production.

The installation procedure can be found here: Install port for v4l2src GStreamer plug-in

Supported devices

The following devices have been tested:

• USB web cameras
• MIPI cameras: imx477 and imx219

nvarguscamerasrc

The NVIDIA Argus Camera Source (nvarguscamerasrc) has also been patched for GStreamer 1.16 in both Jetpack 5.x and Jetpack 6.x. Similar to the v4l2src, it retrieves the start and end of frame timestamps and translates them into the system monotonic clock.

The installation procedure can be found here: Installing nvarguscamerasrc patched

Supported devices

The following devices have been tested:

• MIPI cameras: imx477 and imx219

qtiqmmfsrc

We are developing a Yocto recipe to ease the process

Compiling the element involves applying a patch on the sources of the Yocto building folder. You can look at the changes in this repo.

< >