RidgeRun Projector Plugin

The RidgeRun Projector is a GStreamer plugin intended to contain elements that perform video projections, transforming images into new ones using a mapping relationship. Currently, the plugin only contains one element: rreqrprojector which transforms a fisheye image into an equirectangular projected image.

RidgeRun Equirectangular Projector

RidgeRun Equirectangular Projector is a single input/single output GStreamer element named rreqrprojector. As its name states it will transform an input image into an equirectangular projection, specifically the input image should be a fisheye image.

The required mapping to transform the input image to equirectangular is determined by the element’s properties; you must ensure that the properties correctly describe your fisheye lens and camera orientation to have an accurate equirectangular image. See the properties description below.

This element uses CUDA to accelerate the transformation, specifically inheriting from RidgeRun’s base class CudaBaseFilter of the project gst-cuda. Since it is using CUDA, the element has special memory requirements, it can only handle CUDA or NVMM memory, however, the element proposes an allocator to upstream elements to provide CUDA memory, so most of the elements will be able to connect easily to rreqrprojector. Also, currently, the element only supports RGBA format at its src and sink pads.

As you know for an equirectangular image, the width:height relationship is 2:1, so the element’s default caps negotiation is going to keep that relationship, maintaining the input height and calculating the output width to be 2 times the height. However, the element can handle any resolution that you define at the cost of data appearance distortion.

Properties

center-x: An integer property that defines the center position of the fisheye circle over the input’s image X axis(horizontal). If 0 automatic center is calculated as half of the input image.

center-y: An integer property that defines the center position of the fisheye circle over the input’s image Y axis(vertical). If 0 automatic center is calculated as half of the height image.

crop: A boolean property that states whether the output image should be cropped to half when the lens aperture is smaller or equal to 180 degrees. The objective of this property is to reduce the required memory to store the resultant equirectangular projection. As you know if the lens aperture is 180 or less, the right half of the image will be black and will not provide useful information so cropping it to half will save unused memory and save some processing time. The previous statement is correct if you maintain the rot-z property to 0 (used for stitching), if you modify this angle you may lose information when cropping the image.

lens: A float property that defines the fisheye lens aperture.

radius: An integer property that defines in pixels the radius of the circle containing the fisheye image.

rot-x: A float property that defines the camera’s tilt angle correction in degrees between -180 and 180. This is assuming a coordinate system over the camera where the X axis looks over the side of the camera, and you rotate that axis, rotating the camera up and down. This property should be used to correct the tilt angle of the camera, but you must set the actual camera rotation not the correction needed. If your camera is looking a little bit up over the horizon plane you must set a positive value indicating the angle at this camera is looking up, on the other hand, if your camera is looking down a little you must set a negative value indicating that angle to restore the center of the camera to the horizon.

rot-y: A float property that defines the camera’s roll angle correction in degrees between -180 and 180. This is assuming a coordinate system over the camera where the camera lens looks over the Y axis, if you rotate that axis, the camera will rotate rolling over its center. This property should be used to correct the roll angle of the camera, but you must set the actual camera rotation, not the correction needed. Meaning if your camera image is inclined, a little bit to the left, like a rotated clock-counter wise you must set a negative value to indicate this rotation, and the projector will set the image straight. On the other hand, if your image is inclined to the right you must rotate it clock-counter wise to contrarest but you must set the positive value so the projector knows the camera has inclined that angle clockwise.

rot-z: A float property that defines the camera’s pan angle correction in degrees between -180 and 180. This is assuming a coordinate system over the camera where the Z axis is over the camera, if you rotate that axis, the camera will pan over its center moving around the 360 horizon. This property can be used for stitching to distribute the cameras along the longitude axis in conjunction with identity homographies, since the rotation will take care of the required translation. On the other hand, if you want to use the crop property is recommended to use this rotation in 0 and make the required translations with the homography.

Sample Pipeline

The following pipelines read a fisheye JPEG image, decode it and project it to the equirectangular equivalent using default properties values. The result is saved to a jpeg image.

gst-launch-1.0 filesrc location=fisheye180.jpeg ! jpegdec ! videoconvert ! queue !  rreqrprojector ! videoconvert ! queue !  jpegenc ! filesink location=proj0.jpeg

360 Stitching

The RidgeRun’s cuda stitching solution includes a GStreamer element capable of performing video stitching, the element is named cudastitcher. The cudastitcher uses CUDA acceleration to perform the stitching process, so as the projector, it has special memory requirements and uses RidgeRun’s project gst-cuda. Specifically this element inherits from CudaBaseMiso, since it can receive video from any amount of inputs and produces a single output video. The stitcher can accept RGBA and GRAY8 video streams in cuda memory and only RGBA format in NVMM memory. Find more information in the Image Stitching wiki page.

The cudastitcher element can performed 360 video stitching combined with equirectangular projections of fisheye cameras. To perform the stitch the equrectangular projection required the projection parameters and the stitcher requires the homography matrices configurations. Both requirements can be obtain using the Calibration tool.

Sample Pipeline

The following pipeline stitches 2 fisheye cameras back to back of 190 degrees. This pipeline makes use of the stitcher pads properties to cut out the borders, reduce the overlap region and some artifacts caused by the gray regions in the right side of the image.

{
    "homographies":[
        {
            "images":{
                "target":1,
                "reference":0
            },
            "matrix":{
                "h00": 1, "h01": 0, "h02": 2840,
                "h10": 0, "h11": 1, "h12": 0,
                "h20": 0, "h21": 0, "h22": 1
            }
        }
    ]
}

gst-launch-1.0 -e cudastitcher sink_0::right=2800 sink_1::right=3000 name=stitcher homography-list="`cat homographies.json | tr -d "\n" | tr -d " "`" \
filesrc location= 360_right.jpg ! nvjpegdec ! nvvidconv ! rreqrprojector radius=1447 center_x=1454 center_y=1446 lens=190 ! queue ! stitcher.sink_0 \
filesrc location= 360_left.jpg ! nvjpegdec ! nvvidconv ! rreqrprojector radius=1447 center_x=1440 center_y=1447 lens=190 rot-y=-1 ! queue ! stitcher.sink_1 \
stitcher. ! perf print-arm-load=true  ! queue ! videoconvert ! jpegenc ! filesink location=360_stitch.jpeg

Previous: User Guide/Calibration

Index

Next: Examples

❯