Image Stitching for NVIDIA Jetson/Spherical Video/Equirectangular Projection: Difference between revisions

To work with the stitcher you need to define an homography configuration, this is nothing else than a description of the relationship between the images. The homography configuration is defined in pairs of images, where one image is the reference and the other one is the target, the target will be transformed to match and adjust to the reference image using a homography matrix. In the case of fisheye 360 stitching, the projector already made most of the transformations required to match the images, and at this stage only an offset definition will be required. For cameras aligned in the same plane, you only need to define the horizontal offset corresponding to reference equirectangular image size minus the overlap region that will correspond to the angular distance between the cameras. In general, for 360 stitching your homography matrix will look like the following:

One of the challenges of stitching together 360 equirectangular videos is the parallax effect, corresponding to the difference in the apparent position of an object viewed from different points of view and lines of sight. To have a perfect stitching with no parallax effect your cameras should be in the same position which is physically impossible, so there cannot be a perfect stitching in the real world. But you can adjust the projections and stitching configuration to have a nice blending at a certain depth, meaning a certain distance from the camera's array. However if you adjust to some distance probably objects closer or further from that distance will present parallax issues. Also, the parallax will be determined by the distance between the cameras, smaller cameras closer together will be closer to the ideal setup where both cameras occupy the same space so the parallax effect is small but for bigger cameras further apart the parallax effect will be more noticeable.