Projector

Image Stitching for NVIDIA®Jetson™
Before Starting
Image Stitching Basics
Stitching Projections
Overview
Stitcher Projector Typical Applications Limitations and known issues
Getting Started
Evaluating the Stitcher Getting the code Building Image Stitching
User Guide
Quick Start Guide Controlling the Stitcher Calibration GStreamer Additional details
Resources
3D bases Panoramic data 360 data
Examples
Basic pipelines Using Gstd Other pipelines
Spherical Video
About Equirectangular Projection Projector Building Projector Stitcher Calibration Tool
Performance
Orin Xavier Nano
Contact Us

The projector is a GStreamer plugin intended to contain elements that perform video projections, transforming images into new ones using a mapping relationship. Currently, the plugin only contains one element: rreqrprojector which transforms a fisheye image into an equirectangular projected image.

RR Equirectangular Projector

RR Equirectangular Projector is a single input/single output GStreamer element named rreqrprojector. As its name states it will transform an input image into an equirectangular projection, specifically the input image should be a fisheye image.

The required mapping to transform the input image to equirectangular is determined by the element’s properties; you must ensure that the properties correctly describe your fisheye lens and camera orientation to have an accurate equirectangular image. See the properties description below.

This element uses CUDA to accelerate the transformation, specifically inheriting from RidgeRuns’s base class CudaBaseFilter of the project gst-cuda. Since it is using CUDA, the element has special memory requirements, it can only handle CUDA or NVMM memory, however, the element proposes an allocator to upstream elements to provide CUDA memory, so most of the elements will be able to connect easily to rreqrprojector. Also, currently, the element only supports RGBA format at its src and sink pads.

As you know for an equirectangular image, the width:height relationship is 2:1, so the element’s default caps negotiation is going to keep that relationship, maintaining the input height and calculating the output width to be 2 times the height. However, the element can handle any resolution that you define at the cost of data appearance distortion.

Properties

• center-x: An integer property that defines the center position of the fisheye circle over the input’s image X axis(horizontal). If 0 automatic center is calculated as half of the input image.

• center-y: An integer property that defines the center position of the fisheye circle over the input’s image Y axis(vertical). If 0 automatic center is calculated as half of the height image.

• crop: A boolean property that states whether the output image should be cropped to half when the lens aperture is smaller or equal to 180 degrees. The objective of this property is to reduce the required memory to store the resultant equirectangular projection. As you know if the lens aperture is 180 or less, the right half of the image will be black and will not provide useful information so cropping it to half will save unused memory and save some processing time. The previous statement is correct if you maintain the rot-z property to 0 (used for stitching), if you modify this angle you may lose information when cropping the image.

• lens: A float property that defines the fisheye lens aperture.

• radius: An integer property that defines in pixels the radius of the circle containing the fisheye image.

• rot-x: A float property that defines the camera’s tilt angle correction in degrees between -180 and 180. This is assuming a coordinate system over the camera where the X axis looks over the side of the camera, and you rotate that axis, rotating the camera up and down. This property should be used to correct the tilt angle of the camera, but you must set the actual camera rotation not the correction needed. If your camera is looking a little bit up over the horizon plane you must set a positive value indicating the angle at this camera is looking up, on the other hand, if your camera is looking down a little you must set a negative value indicating that angle to restore the center of the camera to the horizon.

• rot-y: A float property that defines the camera’s roll angle correction in degrees between -180 and 180. This is assuming a coordinate system over the camera where the camera lens looks over the Y axis, if you rotate that axis, the camera will rotate rolling over its center. This property should be used to correct the roll angle of the camera, but you must set the actual camera rotation not the correction needed. Meaning if your camera image is inclined, a little bit to the left, like a rotated clock-counter wise you must set a negative value to indicate this rotation, and the projector will set the image straight. On the other hand, if your image is inclined to the right you must rotate it clock-counter wise to contrarest but you must set the positive value so the projector knows the camera has inclined that angle clockwise.

• rot-z: A float property that defines the camera’s pan angle correction in degrees between -180 and 180. This is assuming a coordinate system over the camera where the Z axis is over the camera, if you rotate that axis, the camera will pan over its center moving around the 360 horizon. This property should not be used for stitching, since the homography will be used to adjust the image over the equirectangular longitude.

Sample Pipelines

The following pipelines read a fisheye JPEG image, decode it and project it to the equirectangular equivalent using default properties values, and save the result to a jpeg image. One is using software elements to decode/encode and send cuda memory to the projector; the other one is using NVIDIA elements and NVMM memory.

gst-launch-1.0 filesrc location=fisheye180.jpeg ! jpegdec ! videoconvert ! queue !  rreqrprojector ! videoconvert ! queue !  jpegenc ! filesink location=proj0.jpeg