Architecture
🚧 Documentation is under development
The Video Stitching for Embedded Systems guide is currently under active development. Some sections may be incomplete or change without notice.
Questions? Contact RidgeRun or email to support@ridgerun.com.
Architecture
The Video Stitching for Embedded Systems is a RidgeRun software solution that involves multiple steps. It is mostly based on RidgeRun LibPanorama (a library specialized in image transformations).
Camera streams enter the projector elements, which use LibPanorama remapping to convert each view into equirectangular space, allowing better transformation and stitching of non-rectilinear images. Afterwards, each projected frame is wrapped together with a stitcher mask that describes its valid coverage, useful when blending two images over the same frame.
Then, the stitcher element receives the frames and masks, then uses LibPanorama to remap camera views according to the calibration and blend them using the masks, leading to the stitched panorama.
Design Overview
The projector converts each camera image into equirectangular space using projection parameters and LibPanorama's remapping facilities. It also generates a per-camera float mask describing valid projection coverage and, in GStreamer pipelines, attaches that mask to the projected frame through the stitcher mask metadata helper. The stitcher engine coordinates the product-specific work needed to combine these views: it loads and validates calibration, resolves camera relationships, derives per-camera transforms, output geometry and regions of interest, and processes the ordered input images. During configuration it uses the blend API to generate default stitcher masks from the camera maps and regions of interest. The stitcher elements retrieve projector masks and pass them to Stitcher::SetMask(), replacing the default masks before blending. Both rrstitcher and rrglstitcher adapt GStreamer buffers to the same Stitcher API, while standalone C++ applications can call that API directly.
The projector and stitcher engine are implemented in separate product repositories and both depend on LibPanorama for reusable image, backend, transform, and remapping capabilities; the stitcher also uses its blending operations. Projector models and mask generation, calibration-file handling, camera graph construction, multi-camera ordering, mask handoff, and GStreamer integration are product-level responsibilities. Keeping those components outside LibPanorama lets the core library remain independent of projector and stitcher policy and reusable by other products. Dependencies point from the projector and stitcher products to LibPanorama; LibPanorama does not depend on either product.
Processing Backends
Since the stitcher is fully based on LibPanorama, below is the list of supported backends to perform the remapping and blending:
- CPU: hybrid implementation of OpenCV-C++
- CUDA: hybrid implementation of NPP-CUDA, adequate for NVIDIA-based platforms
- OpenGL: based on EGL and GLES implementation, adequate for non-NVIDIA platforms like Qualcomm- and NXP-based.
Input Processing Flows
The full processing workflow starts by projecting the input image to equirectangular coordinates, such that all fisheye images are projected with minimal loss of information. The projector generates both, the projected frame and a mask with the valid data. Then the stitcher receives all projected images and masks and blends them according to the mask, which usually has weights based on the centre of the view. This preserves the same functionality as in Image Stitching for NVIDIA Jetson/Spherical Video.
Moreover, it is possible to feed the stitcher with rectilinear images, preceded by an undistortion stage, as Image Stitching for NVIDIA Jetson/User Guide/Quick Start Guide.
Overall, the Video stitching for Embedded Systems is backwards compatible with the previous RidgeRun solution for Jetson.
GStreamer Element Architecture
The GStreamer plug-in for the stitcher includes two possible inheritors: the GStreamer System elements (GstBaseTransform or GstAggregator), and the GStreamer GL inheritors (GstGLMixer and GstGLFilter), easing context handling and texture allocation, simplifying the implementation and maintenance of the elements by offloading the OpenGL responsibility.
