GStreamer Usage

🚧 Documentation is under development

The Video Stitching for Embedded Systems guide is currently under active development. Some sections may be incomplete or change without notice.

Questions? Contact RidgeRun or email to support@ridgerun.com.

Video Stitching for Embedded Systems

< >

Contents Pin

• Getting Started
  • Requirements
  • How to Get the Code
  • Build and Install
  • Three-Camera Rectilinear Quick Start
• Architecture
  • Design Overview
  • Processing Backends
  • Input Processing Flows
  • Remapping, Blending, and Polygon Masks
  • GStreamer Element Architecture
• Configuration and Calibration
  • Configuration Overview
  • Input Requirements and Camera Layout
  • Calibration Workflow and Existing Tools
  • Rectilinear, Fisheye, and Mask Configuration
• GStreamer Usage
  • Plugin and Element Overview
  • Input and Output Caps
  • Pipeline Integration
  • Troubleshooting
• Performance
  • Benchmark Methodology
  • Tested Platform Configurations
  • Results by Platform and Configuration
  • Known Limitations
• Examples
  • Three-Camera Rectilinear Example
  • Fisheye Example
  • GstGL Pipeline Example
  • Output Videos and Reference Results
• FAQ
• Contact Us

GStreamer Usage

The GStreamer plugin provides rrstitcher and rrglstitcher for combining multiple camera streams into a panorama. Both elements use a calibration JSON file to define the camera arrangement.

GStreamer 1.20 or newer is required.

Plugin and Element Overview

The plugin is provided by libgstrrstitcher.so and registers two elements:

Element	Input/output memory	Backend	Typical use
rrstitcher	System-memory video/x-raw	LibPanorama System, CUDA, or OpenGL	Pipelines using CPU-accessible frames
rrglstitcher	video/x-raw(memory:GLMemory)	OpenGL	Pipelines that keep frames in GL memory

rrstitcher selects the LibPanorama backend with the backend property. The default value is auto. rrglstitcher always uses the OpenGL backend and is available only when GStreamer GL and LibPanorama OpenGL support are enabled.

Both elements expose one src pad and request sink pads named sink_0, sink_1, and so on.

The calibration-file property must be set before requesting the sink pads:

rrstitcher name=stitcher calibration-file=calibration.json

The sink pad order must match the camera order in the calibration file. For a three-camera calibration, the complete set of pads is:

sink_0
sink_1
sink_2

Warning

The camera order in the pipeline must match the order used during calibration.

For a local build, inspect the elements and their available properties with:

GST_PLUGIN_PATH=builddir/gst gst-inspect-1.0 rrstitcher
GST_PLUGIN_PATH=builddir/gst gst-inspect-1.0 rrglstitcher

If rrglstitcher is not required, it can be disabled at build time with:

-Dgstreamer-gl=disabled

Input and Output Caps

Element	Sink caps	Source caps
rrstitcher	video/x-raw,format=RGBA	video/x-raw,format=RGBA
rrglstitcher	video/x-raw(memory:GLMemory),format=RGBA,texture-target=2D	video/x-raw(memory:GLMemory),format=RGBA,texture-target=2D

All camera inputs must use the same resolution and framerate.

The panorama dimensions are calculated from the calibration and the negotiated input resolution, so the output size can be different from the dimensions of a single camera input.

Use videoconvert when an input is not already RGBA.

For rrglstitcher, system-memory frames can be uploaded with glupload:

video/x-raw,format=RGBA
  → glupload
  → video/x-raw(memory:GLMemory),format=RGBA,texture-target=2D
  → rrglstitcher

Use gldownload after rrglstitcher when the downstream element requires system-memory frames.

Pipeline Integration

The examples below use a two-camera calibration.json with 1280x720 RGBA inputs at 30 FPS. Add another branch ending in stitcher.sink_N for each additional camera in the calibration.

Video Files

Decode each video and convert it to the input format expected by the Stitcher:

gst-launch-1.0 \
  rrstitcher name=stitcher calibration-file=calibration.json \
    ! queue ! videoconvert ! autovideosink \
  filesrc location=camera0.mp4 ! decodebin ! videoconvert \
    ! 'video/x-raw,format=RGBA,width=1280,height=720,framerate=30/1' \
    ! queue ! stitcher.sink_0 \
  filesrc location=camera1.mp4 ! decodebin ! videoconvert \
    ! 'video/x-raw,format=RGBA,width=1280,height=720,framerate=30/1' \
    ! queue ! stitcher.sink_1

The input recordings must correspond to the camera order used during calibration.

For downstream encoding, replace the display branch with the encoder and muxer required by the application. For example:

stitcher.
  → videoconvert
  → encoder
  → muxer
  → filesink

Live Cameras

On Linux, v4l2src can be used for V4L2 capture devices:

gst-launch-1.0 \
  rrstitcher name=stitcher calibration-file=calibration.json \
    ! queue ! videoconvert ! autovideosink \
  v4l2src device=/dev/video0 do-timestamp=true \
    ! videoconvert \
    ! 'video/x-raw,format=RGBA,width=1280,height=720,framerate=30/1' \
    ! queue ! stitcher.sink_0 \
  v4l2src device=/dev/video1 do-timestamp=true \
    ! videoconvert \
    ! 'video/x-raw,format=RGBA,width=1280,height=720,framerate=30/1' \
    ! queue ! stitcher.sink_1

The cameras should be synchronized when stitching moving scenes. Assigning timestamps when frames arrive does not correct differences in the physical capture time between cameras.

OpenGL Pipeline

Use rrglstitcher when the surrounding pipeline uses GL textures.

The following example uploads system-memory RGBA inputs to GL memory before the Stitcher:

GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 \
  rrglstitcher name=stitcher calibration-file=calibration.json \
    ! gldownload ! videoconvert ! autovideosink \
  videotestsrc \
    ! 'video/x-raw,format=RGBA,width=1280,height=720,framerate=30/1' \
    ! glupload \
    ! 'video/x-raw(memory:GLMemory),format=RGBA,texture-target=2D' \
    ! queue ! stitcher.sink_0 \
  videotestsrc \
    ! 'video/x-raw,format=RGBA,width=1280,height=720,framerate=30/1' \
    ! glupload \
    ! 'video/x-raw(memory:GLMemory),format=RGBA,texture-target=2D' \
    ! queue ! stitcher.sink_1

If upstream and downstream elements already work with GstGLMemory, the upload and download steps are not required.

On headless systems, the GL environment depends on the platform. Systems with surfaceless EGL support may also require:

export GST_GL_WINDOW=surfaceless

Troubleshooting

Symptom	Checks
No such element	Make sure the plugin was built and that the directory containing libgstrrstitcher.so is available through GST_PLUGIN_PATH. Check the element with gst-inspect-1.0. If rrglstitcher is missing, verify that GStreamer GL and LibPanorama OpenGL support were enabled.
Calibration error or sink pad cannot be requested	Check that calibration-file points to a valid calibration JSON file and is set before requesting pads. Connect the complete set of sink_N pads required by the calibration.
not-negotiated or mismatched input caps	Make sure every input is RGBA and uses the same width, height, and framerate. For rrglstitcher, verify that every branch provides GstGLMemory with 2D textures.
GL context or texture error	Verify that GStreamer is using an EGL-backed GL context and that the upstream buffers use 2D textures. Use glupload and gldownload when crossing between system and GL memory.
Panorama is incorrect or streams appear out of step	Verify the camera order, calibration file, input resolution, framerate, and capture synchronization. The sink pad order must match the camera indexes used during calibration.

For additional negotiation and runtime information, enable the Stitcher debug categories:

GST_DEBUG=rrstitcher:6,rrglstitcher:6

Check the first GStreamer error reported when the pipeline fails.

Video Stitching for Embedded Systems

Back to top ↑ < >