Image Stitching

Image Stitching for NVIDIA®Jetson™
Before Starting
Image Stitching Basics
Stitching Projections
Overview
Stitcher Projector Typical Applications Limitations and known issues
Getting Started
Evaluating the Stitcher Getting the code Building Image Stitching
User Guide
Quick Start Guide Controlling the Stitcher Calibration GStreamer Additional details
Resources
3D bases Panoramic data 360 data
Examples
Basic pipelines Using Gstd Other pipelines
Spherical Video
About Equirectangular Projection Projector Building Projector Stitcher Calibration Tool
Performance
Orin Xavier Nano
Contact Us

Image stitching is a process where multiple input images with overlap are merged in order to create a larger image. The image stitching algorithm can be divided into the following stages:

• Feature Matching
• Homographies
• Warping
• Blending

Feature matching

This stage consists in matching features between the input images in order to obtain the best arrangement possible before stitching.

SIFT is an algorithm invariant to image scale and rotation, its objective is to extract keypoints and compute the descriptors for each point in an image.

Descriptors of the found keypoints are matched between two images by matching each feature of one image with the set of features of the second image. The match is true only if the features in both images match each other.

Features of objects with different viewpoints from two adjacent images are matched in order to compute a transformation that match the viewpoint of objects in two images.

Homographies

A homography in the context of image stitching describes a relation between two images, that relation transforms a target image based on a reference image. This provides a way to specify the arrangement and adjustments between images for stitching.

These adjustments and transformations are represented in a homography matrix that synthesizes the mapping between two image planes. For a two-dimension homography, the matrix looks like the following, where x and y are the original coordinates while x' and y' are the new ones.

${\begin{pmatrix}x'\\y'\\1\\\end{pmatrix}}={\begin{pmatrix}h_{00}&h_{01}&h_{02}\\h_{10}&h_{11}&h_{12}\\0&0&1\\\end{pmatrix}}\cdot {\begin{pmatrix}x\\y\\1\\\end{pmatrix}}$

Homography decomposition

This mapping mentioned above can be decomposed into translation, rotation, shear, and scale. each one represented by its own transformation matrix.

From that original homography matrix, the translation can be factored out like so:

${\begin{pmatrix}h_{00}&h_{01}&h_{02}\\h_{10}&h_{11}&h_{12}\\0&0&1\\\end{pmatrix}}={\begin{pmatrix}1&0&h_{02}\\0&1&h_{12}\\0&0&1\\\end{pmatrix}}\cdot {\begin{pmatrix}h_{00}&h_{01}&0\\h_{10}&h_{11}&0\\0&0&1\\\end{pmatrix}}$

Where $h_{02}$ and $h_{12}$ are the x and y axis translations respectively.

The reduced resulting matrix ${\begin{pmatrix}h_{00}&h_{01}\\h_{10}&h_{11}\\\end{pmatrix}}$ can then be decomposed into three more matrices representing rotation, shear and scale; however that process is not as intuitive as the previous one; therefore it is not recommended to manually define the homography matrix unless the transformation is only a translation.

Even with simple homographies it is highly recommended to use the included calibration tool since it is easier and yields better results.

Warping

This stage is divided into alignment and compositing, Each image is transformed with various geometrical transformations set by the calculated homography in the calibration. These transformations maintain the image content but deform the image to fit the viewpoint of the reference image, matching the objects in the overlap section. Once the images are arranged properly they are combined into one larger image.

Blending

The blending stage aims to minimize the difference between images on the overlapping section, this difference may be a product of different camera configurations such as exposure or gain.

The cuda-stitcher is not limited to any number of inputs, it has support for panoramic stitching of up to N images.

Warping & Blending