User Guide On Gstreamer plugins

Image Stitching for NVIDIA®Jetson™
Before Starting
Image Stitching Basics
Stitching Projections
Overview
Stitcher Projector Typical Applications Limitations and known issues
Getting Started
Evaluating the Stitcher Getting the code Building Image Stitching
User Guide
Quick Start Guide Controlling the Stitcher Calibration GStreamer Additional details
Resources
3D bases Panoramic data 360 data
Examples
Basic pipelines Using Gstd Other pipelines
Spherical Video
About Equirectangular Projection Projector Building Projector Stitcher Calibration Tool
Performance
Orin Xavier Nano
Contact Us

This page provides a basic description of the parameters to build a cudastitcher pipeline.

Cuda-Stitcher

To build a cudastitcher pipeline use the following parameters:

• homography-list

List of homographies as a JSON formatted string without spaces or newlines. The homography list can be store in a JSON file and used in the pipeline with the following format:

homography-list="`cat <json-file> | tr -d "\n" | tr -d " "`"

The JSON file section provides a detailed explanation of the JSON file format for the homography list.

• stitcher.sink

The indices of the stitcher's sinks (sink_0, for example) are mapped directly to the image index we use in the homography list.

• camera-matrix

Definition of the camera matrix (only if undistort is necessary). The format is the following:

"{"fx": double, "fy": double,  "cx": double,  "cy": double}"

• distortion-model

Distortion model to use (only if undistort is necessary). The options are: (0)brown-conrady or (1)fisheye.

• distortion-parameters

Definition of the distortion matrix (only if undistort is necessary). The format is the following:

"{"k1": double, "k2": double,  "p1": double,  "p2": double,  "k3": double,  "k4": double,  "k5": double,  "k6": double}"

JSON file

The homography list is a JSON formatted string that defines the transformations and relationships between the images. Here we will explore (with examples) how to create this file in order to stitch the corresponding images.

Case: 2 Images

2 Images Stitching Example

---

Let's assume we only have 2 images (with indices 0 and 1). These 2 images are related by a homography which can be computed using the Homography Estimation Tool. The computed homography transforms the Target image from the Reference image perspective.

This way, to fully describe a homography, we need to declare 3 parameters:

• Matrix: the 3x3 transformation matrix.
• Target: the index of the target image (i.e. the image to be transformed).
• Reference: the index of the reference image (i.e. the image used as a reference to transform the target image).

Having this information, we build the Homography JSON file:

{
    "homographies":[
        {
            "images":{
                "target":1,
                "reference":0
            },
            "matrix":{
                "h00": 1, "h01": 0, "h02": 510,
                "h10": 0, "h11": 1, "h12": 0,
                "h20": 0, "h21": 0, "h22": 1
            }
        }
    ]
}

With this file, we are describing a pair of images (0 and 1), where the given matrix will transform the image 1 based on 0.

Case: 3 Images

3 Images Stitching Example

---

Similar to the 2 images case, we use homographies to connect the set of images. The rule is to use N-1 homographies, where N is the number of images.

One panoramic use case is to compute the homographies for both left (0) and right (2) images, using the center image (1) as the reference. The hoography list JSON file would look like this:

{
    "homographies":[
        {
            "images":{
                "target":0,
                "reference":1
            },
            "matrix":{
                "h00": 1, "h01": 0, "h02": -510,
                "h10": 0, "h11": 1, "h12": 0,
                "h20": 0, "h21": 0, "h22": 1
            }
        },
        {
            "images":{
                "target":2,
                "reference":1
            },
            "matrix":{
                "h00": 1, "h01": 0, "h02": 510,
                "h10": 0, "h11": 1, "h12": 0,
                "h20": 0, "h21": 0, "h22": 1
            }
        }
    ]
}

Your case

You can create your own homography list, using the other cases as a guide. Just keep in mind these rules:

1. N images, N-1 homographies: if you have N input images, you only need to define N-1 homographies.
2. Reference != Target: you can't use the same image as a target and as a reference for a given homography.
3. No Target duplicates: an image can be a target only once.
4. Image indices from 0 to N-1: if you have N images, you have to use consecutive numbers from 0 to N-1 for the target and reference indices. It means that you cannot declare something like target: 6 if you have 6 images; the correct index for your last image is 5.

Basic Pipelines

The following tool will provide simple pipelines according to the selected elements.

The generated pipelines use our perf element to measure the performance of the stitcher. It can be downloaded from this repository, otherwise, the element can be removed from the pipeline without any problem. In case of performance issues, consider executing the /usr/bin/jetson_clocks binary.

Inputs

Select the input type Cameras Video files

Color space RGBA GRAY8

---

Output

Save video
Show on screen

Video format mp4 ts

Generate pipe

Cuda-Undistort

Please follow the Camera Calibration section