Image Stitching for NVIDIA Jetson/User Guide/Homography estimation: Difference between revisions
m (Add ssim parameter) |
(Update script parameters) |
||
Line 112: | Line 112: | ||
==== Script usage ==== | ==== Script usage ==== | ||
The script has | The script has six modes: | ||
* '''video_left_fixed_lowpass''': where the left video is fixed and the right video will be transformed by the homography, averages the homographies for each frame. | |||
* '''video_left_fixed_refinement''': where the left video is fixed and the right video will be transformed by the homography, refines the homography on each frame. | |||
* '''left_fixed''': where the left image is fixed and the right image will be transformed by the homography. | * '''left_fixed''': where the left image is fixed and the right image will be transformed by the homography. | ||
* '''video_right_fixed_lowpass''': where the right video is fixed and the left video will be transformed by the homography, averages the homographies for each frame. | |||
* '''video_right_fixed_refinement''': where the right video is fixed and the left video will be transformed by the homography, refines the homography on each frame. | |||
* '''right_fixed''': where the right image is fixed and the left image will be transformed by the homography. | * '''right_fixed''': where the right image is fixed and the left image will be transformed by the homography. | ||
All modes can be adjusted to the sigma value of the Gaussian filter and the width size of the overlap between the two images. The following are the complete options of the script: | |||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> | ||
python homography_estimation.py --help | python homography_estimation.py --help | ||
usage: homography_estimation.py [-h] {left_fixed,right_fixed} ... | usage: homography_estimation.py [-h] | ||
{video_left_fixed_lowpass,video_left_fixed_refinement,left_fixed,video_right_fixed_lowpass,video_right_fixed_refinement,right_fixed} | |||
... | |||
Tool for | Tool for estimating the homography between two images. | ||
positional arguments: | positional arguments: | ||
{left_fixed,right_fixed} | {video_left_fixed_lowpass,video_left_fixed_refinement,left_fixed,video_right_fixed_lowpass,video_right_fixed_refinement,right_fixed} | ||
video_left_fixed_lowpass | |||
Estimation of homography between two videos, with the | |||
left one fixed. | |||
video_left_fixed_refinement | |||
Estimation of homography between two videos, with the | |||
left one fixed. | |||
left_fixed Estimation of homography between two images, with the | left_fixed Estimation of homography between two images, with the | ||
left one fixed. | |||
video_right_fixed_lowpass | |||
Estimation of homography between two videos, with the | |||
left one fixed. | |||
video_right_fixed_refinement | |||
Estimation of homography between two videos, with the | |||
left one fixed. | left one fixed. | ||
right_fixed Estimation of homography between two images, with the | right_fixed Estimation of homography between two images, with the | ||
Line 143: | Line 161: | ||
python homography_estimation.py left_fixed --help | python homography_estimation.py left_fixed --help | ||
usage: homography_estimation.py left_fixed [-h] [--config CONFIG] | usage: homography_estimation.py left_fixed [-h] [--config CONFIG] | ||
[--referenceImage REFERENCEIMAGE] | |||
[--targetImage TARGETIMAGE] | [--targetImage TARGETIMAGE] | ||
[--scaleWidth SCALEWIDTH] | [--scaleWidth SCALEWIDTH] | ||
[--scaleHeight SCALEHEIGHT] | [--scaleHeight SCALEHEIGHT] | ||
Estimation of homography between two images, with the left one fixed. | Estimation of homography between two images, with the left one fixed. | ||
Line 154: | Line 171: | ||
-h, --help show this help message and exit | -h, --help show this help message and exit | ||
--config CONFIG Path of the configuration file | --config CONFIG Path of the configuration file | ||
--referenceImage REFERENCEIMAGE | |||
Path of the reference image | |||
--targetImage TARGETIMAGE | --targetImage TARGETIMAGE | ||
Path of the target image | Path of the target image | ||
--scaleWidth SCALEWIDTH | --scaleWidth SCALEWIDTH | ||
Scale width dimension of the input images | Scale width dimension of the input images | ||
--scaleHeight SCALEHEIGHT | --scaleHeight SCALEHEIGHT | ||
Scale height dimension of the input images | Scale height dimension of the input images | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Line 170: | Line 186: | ||
python homography_estimation.py right_fixed --help | python homography_estimation.py right_fixed --help | ||
usage: homography_estimation.py right_fixed [-h] [--config CONFIG] | usage: homography_estimation.py right_fixed [-h] [--config CONFIG] | ||
[--referenceImage REFERENCEIMAGE] | |||
[--targetImage TARGETIMAGE] | [--targetImage TARGETIMAGE] | ||
[--scaleWidth SCALEWIDTH] | [--scaleWidth SCALEWIDTH] | ||
[--scaleHeight SCALEHEIGHT] | [--scaleHeight SCALEHEIGHT] | ||
Line 180: | Line 196: | ||
-h, --help show this help message and exit | -h, --help show this help message and exit | ||
--config CONFIG Path of the configuration file | --config CONFIG Path of the configuration file | ||
--referenceImage REFERENCEIMAGE | |||
Path of the reference image | |||
--targetImage TARGETIMAGE | --targetImage TARGETIMAGE | ||
Path of the target image | Path of the target image | ||
--scaleWidth SCALEWIDTH | --scaleWidth SCALEWIDTH | ||
Scale width dimension of the input images | Scale width dimension of the input images | ||
Line 210: | Line 226: | ||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> | ||
// config | // config.json | ||
{ | { | ||
Line 230: | Line 246: | ||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> | ||
python homography_estimation.py left_fixed --config /path/to/config | python homography_estimation.py left_fixed --config /path/to/config.json --referenceImage /path/to/cam1.png --targetImage /path/to/cam2.png | ||
</syntaxhighlight> | </syntaxhighlight> | ||
The '''-- | The '''--referenceImage''' option corresponds to the image that is fixed (left in this case) and the '''--targetImage''' corresponds to the image that will be transformed. The output will be something like this: | ||
<syntaxhighlight lang=bash> | <syntaxhighlight lang=bash> |
Revision as of 19:52, 29 March 2021
Image Stitching for NVIDIA®Jetson™ |
---|
Before Starting |
Image Stitching Basics |
Overview |
Getting Started |
User Guide |
Resources |
Examples |
Spherical Video |
Performance |
Contact Us |
Homography
A homography in the context of image stitching describes a relation between two images, that relation transforms a target image based on a reference image. This provides a way to specify the arrangement and adjustments between images for stitching.
These adjustments and transformations are represented in a homography matrix that synthesizes the mapping between two image planes. For a two dimension homography, the matrix looks like the following, where x and y are the original coordinates while x' and y' are the new ones.
Homography decomposition
This mapping mentioned above can be decomposed into translation, rotation, shear and scale. each one represented by its own transformation matrix.
From that original homography matrix, the translation can be factored out like so:
Where and are the x and y axis translations respectively.
The reduced resulting matrix
can then be decomposed into three more matrices representing rotation, shear and scale; however that process is not as intuitive as the previous one; therefore it is not recommended to manually define the homography matrix unless the transformation is only a translation.
Even with simple homographies it is highly recommended to use the included homography estimation tool, since it is easier and yields better results.
Homography estimation tool
The following section will introduce a way to estimate an initial homography matrix that can be used in the cudastitcher element. This method consists of a Python script that estimates the homography between two images. Find the script in the scripts
directory of the rrstitcher project.
Dependencies
- Python 3.6
- OpenCV 4.0 or later.
- Numpy
Installing the dependencies using a virtual environment
The above dependencies can be installed making use of a Python virtual environment. A virtual environment is useful for installing Python packages without damaging some other environment in your machine. To create a new virtual environment, run the following command:
python3.6 -m venv ENV-NAME
A new folder will be created with the name ENV-NAME. To activate the virtual environment, run the following command:
source <ENV-NAME>/bin/activate
Source the virtual environment each time you want to use it. To install the packages in the virtual environment:
pip install numpy pip install opencv-contrib-python
Script estimation flow
The steps performed by the script are the following:
- Load the images.
- Remove the distortion of the images. (Optional)
- Perform a preprocessing to the images removing the noise using a Gaussian filter.
- Extract the keypoint and the corresponding descriptors with the SIFT algorithm.
- Find the correspondences among the key points of the two images.
- With the resulting keypoints, estimate the homography.
Lens distortion correction
In case there is need to apply distortion correction on the camera, follow the camera calibration process found in this wiki to obtain the camera matrix and distortion parameters.
Script usage
The script has six modes:
- video_left_fixed_lowpass: where the left video is fixed and the right video will be transformed by the homography, averages the homographies for each frame.
- video_left_fixed_refinement: where the left video is fixed and the right video will be transformed by the homography, refines the homography on each frame.
- left_fixed: where the left image is fixed and the right image will be transformed by the homography.
- video_right_fixed_lowpass: where the right video is fixed and the left video will be transformed by the homography, averages the homographies for each frame.
- video_right_fixed_refinement: where the right video is fixed and the left video will be transformed by the homography, refines the homography on each frame.
- right_fixed: where the right image is fixed and the left image will be transformed by the homography.
All modes can be adjusted to the sigma value of the Gaussian filter and the width size of the overlap between the two images. The following are the complete options of the script:
python homography_estimation.py --help usage: homography_estimation.py [-h] {video_left_fixed_lowpass,video_left_fixed_refinement,left_fixed,video_right_fixed_lowpass,video_right_fixed_refinement,right_fixed} ... Tool for estimating the homography between two images. positional arguments: {video_left_fixed_lowpass,video_left_fixed_refinement,left_fixed,video_right_fixed_lowpass,video_right_fixed_refinement,right_fixed} video_left_fixed_lowpass Estimation of homography between two videos, with the left one fixed. video_left_fixed_refinement Estimation of homography between two videos, with the left one fixed. left_fixed Estimation of homography between two images, with the left one fixed. video_right_fixed_lowpass Estimation of homography between two videos, with the left one fixed. video_right_fixed_refinement Estimation of homography between two videos, with the left one fixed. right_fixed Estimation of homography between two images, with the right one fixed. optional arguments: -h, --help show this help message and exit Type "homography_estimation.py <command> -h" for more information.
Options for the left_fixed mode:
python homography_estimation.py left_fixed --help usage: homography_estimation.py left_fixed [-h] [--config CONFIG] [--referenceImage REFERENCEIMAGE] [--targetImage TARGETIMAGE] [--scaleWidth SCALEWIDTH] [--scaleHeight SCALEHEIGHT] Estimation of homography between two images, with the left one fixed. optional arguments: -h, --help show this help message and exit --config CONFIG Path of the configuration file --referenceImage REFERENCEIMAGE Path of the reference image --targetImage TARGETIMAGE Path of the target image --scaleWidth SCALEWIDTH Scale width dimension of the input images --scaleHeight SCALEHEIGHT Scale height dimension of the input images
Options for the right_fixed mode:
python homography_estimation.py right_fixed --help usage: homography_estimation.py right_fixed [-h] [--config CONFIG] [--referenceImage REFERENCEIMAGE] [--targetImage TARGETIMAGE] [--scaleWidth SCALEWIDTH] [--scaleHeight SCALEHEIGHT] Estimation of homography between two images, with the right one fixed. optional arguments: -h, --help show this help message and exit --config CONFIG Path of the configuration file --referenceImage REFERENCEIMAGE Path of the reference image --targetImage TARGETIMAGE Path of the target image --scaleWidth SCALEWIDTH Scale width dimension of the input images --scaleHeight SCALEHEIGHT Scale height dimension of the input images
Configuration file
The script uses a configuration file to access the values of the different variables needed to set up the algorithm before performing the homography estimation. The configuration file contains the following parameters:
- cameraMatrix: Array with the values corresponding to the camera matrix.
- distortionParameters: Array with the values corresponding to the distortion coefficients.
- reprojError: Reprojection error for the homography calculation.
- matchRatio: Max distance ratio for a possible match of keypoints.
- sigma: Sigma value for the Gaussian filter.
- overlap: Degrees of overlap between the two images.
- crop: Degrees of the crop to apply in the sides of the image corresponding to the seam.
- fov: Field of view in degrees of the cameras.
- ssim: Structural similarity index measure threshold
- undistort: Bool value to enable the application or not of the distortion removal.
Example
The following example will estimate the homography of two images, with the left one fixed. In this case, the following configuration file is used:
// config.json { "cameraMatrix":[1, 0, 0, 0, 1, 0, 0, 0, 1], "distortionParameters":[0, 0, 0, 0, 0], "reprojError":4.5, "matchRatio":0.75, "sigma":0.5, "overlap":15, "crop":0, "fov":70, "ssim":0.5, "undistort":false }
The command to perform the estimation is:
python homography_estimation.py left_fixed --config /path/to/config.json --referenceImage /path/to/cam1.png --targetImage /path/to/cam2.png
The --referenceImage option corresponds to the image that is fixed (left in this case) and the --targetImage corresponds to the image that will be transformed. The output will be something like this:
matches: 69 RC_HOMOGRAPHY="{\"h00\":0.16665120123596464,\"h01\":-0.08897097209511769, \"h02\":1808.7657933620071, \"h10\":-0.2887392749063616, \"h11\":0.9790321622250535, \"h12\":39.68771747852058, \"h20\":-0.00038491321338392687, \"h21\":-4.358286543507097e-06, \"h22\":1.0}"
Also, the script will generate some images to evaluate the quality of the homography:
To scale the homography to fit another dimension of the input images, use the --homographyScale option.