Image Stitching for NVIDIA Jetson/Spherical Video/Equirectangular Projection: Difference between revisions

The relationship between the fisheye image and the equirectangular projection can be determined geometrically, we are not going to get into details, but you can visualize it in the figures below. As you can see, the region projected in the equirectangular frame will change according to the aperture. For the equirectangular projection of a fisheye of 180 degrees or less half of the image is totally black doesn’t provide any relevant information, so that memory could be discarded to save resources.

You need to know that a fisheye to equirectangular projector converts the fisheye image where the real world verticals are curved to actual vertical lines in the equirectangular image and the horizon is converted to an horizontal line. If the resultant equirectangular projection doesn’t look like this you should adjust the projector configuration to accomplish these results.

To work with the stitcher you need to define an homography configuration, this is nothing else than a description of the relationship between the images. The homography configuration is defined in pairs of images, where one image is the reference and the other one is the target, the target will be transformed to match and adjust to the reference image using a homography matrix. In the case of fisheye 360 stitching, the projector already made most of the transformations required to match the images, and at this stage only an offset definition will be required. For cameras aligned in the same plane, you only need to define the horizontal offset corresponding to reference equirectangular image size minus the overlap region that will correspond to the angular distance between the cameras. In general, for 360 stitching your homography matrix will look like the following:

One of the challenges of stitching together 360 equirectangular videos is the parallax effect, corresponding to the difference in the apparent position of an object viewed from different points of view and lines of sight. To have a perfect stitching with no parallax effect your cameras should be in the same position which is physically impossible, so there cannot be a perfect stitching in the real world. But you can adjust the projections and stitching configuration to have a nice blending at a certain depth, meaning a certain distance from the camera's array. However if you adjust to some distance probably objects closer or further from that distance will present parallax issues. Also, the parallax will be determined by the distance between the cameras, smaller cameras closer together will be closer to the ideal setup where both cameras occupy the same space so the parallax effect is small but for bigger cameras further apart the parallax effect will be more noticeable.