Image Stitching for NVIDIA Jetson/Resources/360 data

Image Stitching for NVIDIA®Jetson™
Before Starting
Image Stitching Basics
Stitching Projections
Overview
Stitcher Projector Typical Applications Limitations and known issues
Getting Started
Evaluating the Stitcher Getting the code Building Image Stitching
User Guide
Quick Start Guide Controlling the Stitcher Calibration GStreamer Additional details
Resources
3D bases Panoramic data 360 data
Examples
Basic pipelines Using Gstd Other pipelines
Spherical Video
About Equirectangular Projection Projector Building Projector Stitcher Calibration Tool
Performance
Orin Xavier Nano
Contact Us

Triple Fisheye Videos

The collection of videos for this setup corresponds to shots in motion where the camera's array was installed over the roof of a car. The shots were divided into clips of 30 seconds or 1 minute to make them manageable and can be found in the dataset folder Triple_Fisheye. Each clip group consists of three MP4 videos, one for each of the cameras, and is saved in its folder named triple_fisheye_vid<index>_<duration>, where an index is a number indicating the clip group id and the duration that can be the 30s or 1m. The following table provides the list of clip groups available for the three fisheye cameras setup:

`

Scenario	Clip Group	Description
Triple fisheye 360 video	triple_fisheye_vid1_30s	Drive down a residential street along a park.
Triple fisheye 360 video	triple_fisheye_vid2_30s	Turn into a residential street.
Triple fisheye 360 video	triple_fisheye_vid3_30s	Drive through a roundabout.
Triple fisheye 360 video	triple_fisheye_vid1_1m	Drive down a residential.

Table 1. 360 triple videos clip groups

Each folder contains three video files named as the folder with a suffix indication of the sensor id from s0 to s2.

Triple videos results

Using the 360 Calibration tool the following projection parameters and homography matrix can be obtained.

To watch the stitching result of any of the clip groups, save the above parameters and homography matrix in a JSON file named 360-triple.json and run the following commands. This pipeline will save the stitching result in a 360_triple_stitched_video.mp4 file.

Dual Fisheye Videos

The collection of videos for this setup consists of shots in motion where the camera's array was installed on top of a box and carried on through a park scenario. The shots were divided into clips with durations of 30 seconds or 1 minute and can be found in the dataset folder Dual_Fisheye. Each clip group of videos consists of MP4 two videos, one for each camera, and saved in a folder named dual_fisheye_vid<index_<duration>, the index indicates the clip group id, and the duration that can be 30s or 1m. The following table provides the list of clips groups available for the dual fisheye cameras setup:

`

Scenario	Clip Group	Description
Dual fisheye 360 video	dual_fisheye_vid1_30s
Dual fisheye 360 video	dual_fisheye_vid2_30s
Dual fisheye 360 video	dual_fisheye_vid3_30s
Dual fisheye 360 video	dual_fisheye_vid1_1m

Table 2. 360 dual videos clip groups

Each folder contains two video files named as the folder with a suffix indicating the sensor id s0 or s1.

Dual videos results

Using the 360 Calibration tool the following projection parameters and homography matrix can be obtained.

{
    "projections": [
        {
            "0": {
                "radius": 750.0,
                "lens": 187.0,
                "center_x": 993.0,
                "center_y": 762.0,
                "rot_x": 0.0,
                "rot_y": 0.0,
                "rot_z": -89.6,
                "fisheye": true
            }
        },
        {
            "1": {
                "radius": 750.3,
                "lens": 186.0,
                "center_x": 1044.0,
                "center_y": 776.0,
                "rot_x": 0.0,
                "rot_y": 0.0,
                "rot_z": 88.7,
                "fisheye": true
            }
        }
    ],
    "homographies": [
        {
            "images": {
                "target": 1,
                "reference": 0
            },
            "matrix": {
                "h00": 1,
                "h01": 0,
                "h02": 0.0,
                "h10": 0,
                "h11": 1,
                "h12": 0,
                "h20": 0,
                "h21": 0,
                "h22": 1
            }
        }
    ]
}

To watch the stitching result of any of the clip groups, save the above parameters and homography matrix in a JSON file named 360-dual.json and run the following commands. This pipeline will save the stitching result in a 360_dual_stitched_video.mp4 file.

S0_C_X=993
S0_C_Y=762
S0_rad=750
S0_R_X=0
S0_R_Y=0
S0_R_Z=-89.6
S0_LENS=187

S1_C_X=1044
S1_C_Y=776
S1_rad=750
S1_R_X=0
S1_R_Y=0
S1_R_Z=88.7
S1_LENS=186

CONFIG_FILE="360-dual.json"
HOMOGRAPHIES="`cat $CONFIG_FILE | tr -d "\n" | tr -d "\t" | tr -d " "`"

GST_DEBUG=WARNING gst-launch-1.0 -e -v \
	cudastitcher name=stitcher homography-list=$HOMOGRAPHIES \
	filesrc location=360-s0.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! queue ! nvvidconv ! rreqrprojector center-x=$S0_C_X center-y=$S0_C_Y radius=$S0_rad rot-y=$S0_R_Y rot-z=$S0_R_Z lens=$S0_LENS name=proj0 !  queue ! stitcher.sink_0 \
	filesrc location=360-s1.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! queue ! nvvidconv !  rreqrprojector center-x=$S1_C_X center-y=$S1_C_Y radius=$S1_rad rot-y=$S1_R_Y rot-x=$S1_R_Y rot-z=$S1_R_Z lens=$S1_LENS name=proj1 !  queue ! stitcher.sink_1 \
	stitcher.  ! queue ! nvvidconv ! nvv4l2h264enc bitrate=30000000 ! h264parse ! queue !  qtmux ! filesink location=360_dual_stitched_video.mp4