Jump to content

Examples

From RidgeRun Developer Wiki

🚧 Documentation is under development

The Video Stitching for Embedded Systems guide is currently under active development. Some sections may be incomplete or change without notice.

Questions? Contact RidgeRun or email to support@ridgerun.com.

Video Stitching for Embedded Systems logo
Video Stitching for Embedded Systems

Contents Pin





Examples

This page provides example pipelines for running the Stitcher with recorded camera inputs.

The examples assume that the Stitcher is already installed and that a valid calibration file is available for the input cameras.

Three-Camera Rectilinear Example

This example stitches three recorded videos using the rrstitcher element.

The example requires:

  • Three videos captured from the calibrated cameras.
  • A calibration file for the same three-camera setup.
  • All inputs using the same resolution and framerate.

For example: 

export VIDEO_0=/path/to/camera-0.mp4
export VIDEO_1=/path/to/camera-1.mp4
export VIDEO_2=/path/to/camera-2.mp4
export CALIBRATION_FILE=/path/to/calibration.json

The camera order must match the camera indices in the calibration file:

  • VIDEO_0 → stitcher.sink_0
  • VIDEO_1 → stitcher.sink_1
  • VIDEO_2 → stitcher.sink_2

The following example uses 1920x1080 inputs at 30 FPS and saves the stitched output to rrstitcher-output.mp4: 

gst-launch-1.0 -e \
    rrstitcher name=stitcher calibration-file="$CALIBRATION_FILE" \
        stitcher. ! queue ! videoconvert ! \
        x264enc speed-preset=veryfast tune=zerolatency bitrate=8000 ! \
        h264parse ! mp4mux ! filesink location=rrstitcher-output.mp4 \
    filesrc location="$VIDEO_0" ! decodebin ! videoconvert ! \
        "video/x-raw,format=RGBA,width=1920,height=1080,framerate=30/1" ! \
        queue ! stitcher.sink_0 \
    filesrc location="$VIDEO_1" ! decodebin ! videoconvert ! \
        "video/x-raw,format=RGBA,width=1920,height=1080,framerate=30/1" ! \
        queue ! stitcher.sink_1 \
    filesrc location="$VIDEO_2" ! decodebin ! videoconvert ! \
        "video/x-raw,format=RGBA,width=1920,height=1080,framerate=30/1" ! \
        queue ! stitcher.sink_2

The Stitcher calculates the panorama dimensions from the calibration and the negotiated input resolution.


Warning
If the video order does not match the camera order used for calibration, the resulting panorama will be incorrect.


Platform-specific hardware decoders can be used instead of decodebin when available.

For example, on NVIDIA Jetson an H.264 input can use: 

filesrc location="$VIDEO_0" ! qtdemux ! h264parse ! \
    nvv4l2decoder ! nvvidconv ! \
    "video/x-raw,format=RGBA,width=1920,height=1080,framerate=30/1" ! \
    queue ! stitcher.sink_0

The same decoder chain can be used for the other camera inputs.

GstGL Pipeline Example

The rrglstitcher element operates on GstGLMemory. Each decoded input is uploaded with glupload before being connected to the Stitcher.

Using the same three input videos and calibration file, the following pipeline saves the stitched output to rrglstitcher-output.mp4: 

GST_GL_PLATFORM=egl \
GST_GL_API=gles2 \
gst-launch-1.0 -e \
    rrglstitcher name=stitcher calibration-file="$CALIBRATION_FILE" \
        stitcher. ! queue ! gldownload ! \
        "video/x-raw,format=RGBA" ! videoconvert ! \
        x264enc speed-preset=veryfast tune=zerolatency bitrate=8000 ! \
        h264parse ! mp4mux ! filesink location=rrglstitcher-output.mp4 \
    filesrc location="$VIDEO_0" ! decodebin ! videoconvert ! \
        "video/x-raw,format=RGBA,width=1920,height=1080,framerate=30/1" ! \
        queue ! glupload ! \
        "video/x-raw(memory:GLMemory),format=RGBA" ! \
        queue ! stitcher.sink_0 \
    filesrc location="$VIDEO_1" ! decodebin ! videoconvert ! \
        "video/x-raw,format=RGBA,width=1920,height=1080,framerate=30/1" ! \
        queue ! glupload ! \
        "video/x-raw(memory:GLMemory),format=RGBA" ! \
        queue ! stitcher.sink_1 \
    filesrc location="$VIDEO_2" ! decodebin ! videoconvert ! \
        "video/x-raw,format=RGBA,width=1920,height=1080,framerate=30/1" ! \
        queue ! glupload ! \
        "video/x-raw(memory:GLMemory),format=RGBA" ! \
        queue ! stitcher.sink_2

The glupload elements convert the system-memory inputs into the GstGLMemory buffers expected by rrglstitcher. The stitched output is downloaded back to system memory with gldownload before encoding it to H.264.

When the input is already available as GstGLMemory, the corresponding glupload stage is not required.

For headless systems, the GL window backend can also be configured before running the pipeline: 

export GST_GL_PLATFORM=egl
export GST_GL_API=gles2
export GST_GL_WINDOW=surfaceless

For additional information about the GStreamer elements and their input requirements, see GStreamer Usage.

Fisheye Pipeline Example

The fisheye example requires the use of the RidgeRun Projector to project the fisheye camera image into an equirectangular plane. Use the following input videos for reference:

These cameras have the following projector configurations:

Camera 0:

  • Radius: 750
  • Lens: 187
  • Center X: 993
  • Center Y: 762
  • Rotation X: 0.0
  • Rotation Y: 0.0
  • Rotation Z: -89.6

Camera 1:

  • Radius: 750.3
  • Lens: 186
  • Center X: 1044.0
  • Center Y: 776
  • Rotation X: 0.0
  • Rotation Y: 0.0
  • Rotation Z: 88.7

And the homography calibration file is the following (named cal.json): 

{
    "homographies": [
        {
            "images": {
                "target": 1,
                "reference": 0
            },
            "matrix": {
                "h00": 1,
                "h01": 0,
                "h02": 0.0,
                "h10": 0,
                "h11": 1,
                "h12": 0,
                "h20": 0,
                "h21": 0,
                "h22": 1
            }
        }
    ]
}

The stitching pipeline: 

gst-launch-1.0 -e rrglstitcher name=stitcher calibration-file="cal.json"                                \
     filesrc location=360-s0.mp4 ! qtdemux ! h264parse ! decodebin ! queue !                            \
           videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload !       \
           rrglfisheyetoeqr radius=750 lens=187 center_x=993 center_y=762 rot_x=0 rot_y=0 rot_z=-89.6 ! \
           queue ! stitcher.sink_0                                                                      \
    filesrc location=360-s1.mp4 ! qtdemux ! h264parse ! decodebin ! queue !                             \
           videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload !       \
           rrglfisheyetoeqr radius=750 lens=186 center_x=1044 center_y=776 rot_x=0 rot_y=0 rot_z=88.7 ! \
           queue ! stitcher.sink_1                                                                      \
    stitcher. ! queue ! gldownload ! queue ! videoconvert ! videorate !                                 \
           video/x-raw,format=I420,framerate=30/1 !                                                     \
           x264enc speed-preset=fast tune=zerolatency bitrate=10000 bframes=0 key-int-max=30 !          \
           h264parse ! mp4mux faststart=true !                                                          \
    filesink location="fisheye-stitcher-hq.mp4"

The pipeline uses the GstGL elements for better performance on Qualcomm platforms. It first performs the projection and then, performs the stitching.

The output video looks similar to the following one:


Notice
The mask can be further calibrated. This demo represents the projection and stitching capabilities only.


Output Videos and Reference Results

The following rectilinear examples use recorded camera inputs and their corresponding calibration files. Each example saves the stitched panorama to an MP4 file.

Three-Camera Rectilinear Example

This example stitches three camera recordings using rrglstitcher.

Input Files

Download: Input videos and calibration can be obtained here.

The dataset contains: 

rectilinear-3cam/
├── homography.json
├── video-s0.mp4
├── video-s1.mp4
└── video-s2.mp4

The calibration uses video-s1.mp4 as camera 0. Connect the recordings in this order:

Stitcher pad Video
sink_0 video-s1.mp4
sink_1 video-s0.mp4
sink_2 video-s2.mp4

Calibration

Save the following calibration as homography.json in the dataset directory:

Three-camera homography.json 

{
    "homographies": [
        {
            "images": {
                "target": 1,
                "reference": 0
            },
            "matrix": {
                "h00": 0.7791246794668455,
                "h01": 0.03918377289321621,
                "h02": 1017.9359103479511,
                "h10": -0.05523859597285526,
                "h11": 0.9667746037848984,
                "h12": 60.094912688347264,
                "h20": -0.00012609656784964485,
                "h21": 1.4214840486600683e-05,
                "h22": 1.0
            }
        },
        {
            "images": {
                "target": 2,
                "reference": 0
            },
            "matrix": {
                "h00": 1.3913727066002217,
                "h01": -0.050979440703839037,
                "h02": -1391.5441892163778,
                "h10": 0.11215295744983422,
                "h11": 1.2533894933660923,
                "h12": -191.87470867506212,
                "h20": 0.00019946084594618663,
                "h21": -3.912626846464952e-05,
                "h22": 1.0
            }
        }
    ]
}

Run the Example

Set the paths to the recordings and calibration: 

export ASSET_DIR=/path/to/rectilinear-3cam

export INPUT_0="$ASSET_DIR/video-s1.mp4"
export INPUT_1="$ASSET_DIR/video-s0.mp4"
export INPUT_2="$ASSET_DIR/video-s2.mp4"

export CALIBRATION_FILE="$ASSET_DIR/homography.json"

Run the pipeline: 

gst-launch-1.0 -e rrglstitcher name=stitcher calibration-file="$CALIBRATION_FILE" \
    filesrc location="$INPUT_0" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_0 \
    filesrc location="$INPUT_1" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_1 \
    filesrc location="$INPUT_2" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_2 \
    stitcher. ! queue ! gldownload ! queue ! videoconvert ! videorate ! \
        video/x-raw,format=I420,framerate=30/1 ! \
        x264enc speed-preset=fast tune=zerolatency bitrate=10000 bframes=0 key-int-max=30 ! \
        h264parse ! mp4mux faststart=true ! \
        filesink location="rectilinear-3cam.mp4"

The inputs are converted to RGBA at 30 FPS, uploaded to GL memory, and then passed to rrglstitcher. The stitched panorama is downloaded to system memory and encoded to H.264.

Reference Output

The stitched video is saved as rectilinear-3cam.mp4 in the current directory and should look like the following screenshot:

The reference output video can be seen here.

Six-Camera Rectilinear Example

This example stitches six recordings using rrglstitcher and the calibration supplied with the six-camera dataset.

Input Files

Download: Input videos and calibration can be obtained here.

The dataset contains: 

mosaic_static_vid1_1m/
├── homography.json
├── mosaic_static_vid1_1m_s0.mp4
├── mosaic_static_vid1_1m_s1.mp4
├── mosaic_static_vid1_1m_s2.mp4
├── mosaic_static_vid1_1m_s3.mp4
├── mosaic_static_vid1_1m_s4.mp4
└── mosaic_static_vid1_1m_s5.mp4

Connect the recordings in this order:

Stitcher pad Video
sink_0 mosaic_static_vid1_1m_s4.mp4
sink_1 mosaic_static_vid1_1m_s3.mp4
sink_2 mosaic_static_vid1_1m_s0.mp4
sink_3 mosaic_static_vid1_1m_s1.mp4
sink_4 mosaic_static_vid1_1m_s2.mp4
sink_5 mosaic_static_vid1_1m_s5.mp4

Calibration

Use the homography.json included with this dataset. It corresponds to the six-camera order shown above.

Six-camera homography.json 

{
    "homographies": [
        {
            "images": {
                "target": 1,
                "reference": 0
            },
            "matrix": {
                "h00": 0.7490261895239074,
                "h01": 0.04467113632580552,
                "h02": 1018.9828151317821,
                "h10": -0.05577820485200396,
                "h11": 0.9590844935041531,
                "h12": 61.08068248533324,
                "h20": -0.00014412069693060743,
                "h21": 1.7581178118418628e-05,
                "h22": 1.0
            }
        },
        {
            "images": {
                "target": 2,
                "reference": 0
            },
            "matrix": {
                "h00": 0.8203959419915308,
                "h01": 0.19134092629013782,
                "h02": 927.0948457177544,
                "h10": -0.0179915273643625,
                "h11": 1.034698464498257,
                "h12": -680.8085473782533,
                "h20": -0.00014834561743950648,
                "h21": 8.652704821052748e-05,
                "h22": 1.0
            }
        },
        {
            "images": {
                "target": 3,
                "reference": 0
            },
            "matrix": {
                "h00": 1.0378054016500433,
                "h01": 0.03676895233913665,
                "h02": -5.558535987201656,
                "h10": -0.006201947768703567,
                "h11": 1.0395215780726415,
                "h12": -621.329917462604,
                "h20": -1.2684476455313587e-05,
                "h21": 5.7273947504607006e-05,
                "h22": 1.0
            }
        },
        {
            "images": {
                "target": 4,
                "reference": 0
            },
            "matrix": {
                "h00": 1.3394576098442972,
                "h01": -0.15576318123486257,
                "h02": -1230.4161628261922,
                "h10": 0.21296156399303987,
                "h11": 1.25689163996496,
                "h12": -1130.7835925176462,
                "h20": 0.00014172905668812799,
                "h21": 4.9304913992162066e-05,
                "h22": 1.0
            }
        },
        {
            "images": {
                "target": 5,
                "reference": 0
            },
            "matrix": {
                "h00": 1.3197060186315637,
                "h01": -0.10518566348433173,
                "h02": -1264.5768270277113,
                "h10": 0.1467783274677278,
                "h11": 1.1524649023229194,
                "h12": -227.0179395401691,
                "h20": 0.00019864314625771476,
                "h21": -0.00010857278904972765,
                "h22": 1.0
            }
        }
    ]
}

Run the Example

Set the paths to the recordings and calibration: 

export ASSET_DIR=/path/to/mosaic_static_vid1_1m

export INPUT_0="$ASSET_DIR/mosaic_static_vid1_1m_s4.mp4"
export INPUT_1="$ASSET_DIR/mosaic_static_vid1_1m_s3.mp4"
export INPUT_2="$ASSET_DIR/mosaic_static_vid1_1m_s0.mp4"
export INPUT_3="$ASSET_DIR/mosaic_static_vid1_1m_s1.mp4"
export INPUT_4="$ASSET_DIR/mosaic_static_vid1_1m_s2.mp4"
export INPUT_5="$ASSET_DIR/mosaic_static_vid1_1m_s5.mp4"

export CALIBRATION_FILE="$ASSET_DIR/homography.json"

Run the pipeline: 

gst-launch-1.0 -e rrglstitcher name=stitcher calibration-file="$CALIBRATION_FILE" \
    filesrc location="$INPUT_0" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_0 \
    filesrc location="$INPUT_1" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_1 \
    filesrc location="$INPUT_2" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_2 \
    filesrc location="$INPUT_3" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_3 \
    filesrc location="$INPUT_4" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_4 \
    filesrc location="$INPUT_5" ! qtdemux ! h264parse ! decodebin ! queue ! \
        videoconvert ! videorate ! video/x-raw,format=RGBA,framerate=30/1 ! queue ! glupload ! \
        queue ! stitcher.sink_5 \
    stitcher. ! queue ! gldownload ! queue ! videoconvert ! videorate ! \
        video/x-raw,format=I420,framerate=30/1 ! \
        x264enc speed-preset=fast tune=zerolatency bitrate=10000 bframes=0 key-int-max=30 ! \
        h264parse ! mp4mux faststart=true ! \
        filesink location="rectilinear-6cam.mp4"

The six inputs are converted to RGBA at 30 FPS and uploaded to GL memory before stitching.

Reference Output

The stitched video is saved as rectilinear-6cam.mp4 in the current directory and should look like the following screenshot:

The reference output video can be seen here.


Video Stitching for Embedded Systems logo
Video Stitching for Embedded Systems
Back to top ↑



Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=Video_Stitching_for_Embedded_Systems/Examples&oldid=90735"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.