Spherical Video PTZ: User Guide - Quick Start Guide

Spherical Video PTZ
Getting Started
Spherical Video PTZ Projections Used Evaluating the Product Getting the code
User Guide
Building and Installation Quick Start Guide
Examples
Gst-launch GstD
Performance
Jetson AGX Xavier
Contact Us

Libpanorama

This wiki introduces a basic use of Spherical Video PTZ for converting equirectangular images to rectilinear. The core of Spherical Video PTZ is what we call Libpanorma, a C++ library designed to perform multiple projections over video or images.

Here you will find examples on how to use Libpanorama and also how to use GstPanoramaPTZ which is a Gstreamer plugin built on top of Libpanorama.

Minimal Application

After Building and Installation, follow these steps:

1. Download the sample image, if you haven't already.

cd $SAMPLES
wget "https://unsplash.com/photos/PYpkPbBCNFw/download?ixid=M3wxMjA3fDB8MXxhbGx8fHx8fHx8fHwxNzExNTE1MTQxfA&force=true" -O example_image.jpg

2. This example demonstrates the use of the Spherical Video PTZ engine to convert equirectangular images into rectilinear format. This command processes example_image.jpg, converting it from an equirectangular format to a rectilinear view. But you can use any other reference image as long as it is equirectangular. Run the example as:

cd $LIBPANORAMA_PATH/libpanorama
./builddir/examples/equirectangular_to_rectilinear $SAMPLES/example_image.jpg

3. For this example you can use the interactive controls with the Spherical Video PTZ (Pan-Tilt-Zoom) for dynamic exploration of panoramic images. Hit the specified keys when the example is running:

• Zoom In/Out: Adjust the zoom level to get a closer view or a wider perspective of the image.
  • In: i
  • Out: o
• Pan Left/Right: Rotate the view horizontally to explore the left or right sides of the panoramic image.
  • Left: 4
  • Right: 6
• Tilt Up/Down: Adjust the vertical angle of the camera to look up or down within the panoramic image.
  • Up: 8
  • Down: 2

4. press the Esc key to exit the program.

Spherical Video PTZ Engine

The Video PTZ Engine simplifies the use of PTZ controls, making it easier to integrate into your code. To get started, add the following includes and namespaces to your code:

#include <iostream>
#include <lp/allocators/cudaimage.hpp>
#include <lp/engines/equirectangular_to_rectilinear.hpp>
#include <lp/image.hpp>
#include <lp/io/opencv.hpp>
#include <lp/rgba.hpp>

using namespace lp;
using namespace lp::io;

Once this have been done, create the engine. To do so, instantiate the class as demonstrated:

lp::engines::EquirectangularToRectilinear<RGBA<uint8_t>> engine;

Next, configure the parameters to be manipulated using the properties of the Spherical Video PTZ. Instantiate them as showed in the code below. The initial parameters, represented as {{0.0f, 0.0f}, 2.0f}, control the panning, tilting, and zooming of the image in the format {{pan, tilt}, zoom}. The panning and tilting values can are valid in the range: [-π,π] while the zooming in the range: [0.1, 10]. Additionally, {dst.GetSize()} is used to define the output size in {width, height} format, while {io.GetSize()} is utilized to set the input size, also in {width, height} format.

engines::EquirectangularToRectilinearParams params{{
                                                     {0.0f, 0.0f},
                                                     2.0f,
                                                   },
                                                   {
                                                     dst.GetSize(),
                                                   },
                                                   {io.GetSize()}};

Then, set the initial parameters with the SetParameters method:

lp::engine.SetParameters(params);

Finally, use the Process method with the input image as the first parameter. The second parameter will contain the result after applying the equirectangular to rectilinear projection transformation. Please note that the Engine supports both CudaImages and Images for processing. However, if you choose to use an Image, the Engine will internally allocate a Cuda buffer and copy the Image content into it, potentially affecting the application's performance.

lp::Image<RGBA<uint8_t>> img;
lp::allocators::CudaImage<RGBA<uint8_t>> dst;

lp::engine.Process(img, dst);

Consider the following pseudo-code snippet as an example of how to use the Engine in a loop:

#include <iostream>
#include <lp/engines/equirectangular_to_rectilinear.hpp>
#include <lp/image.hpp>
#include <lp/io/opencv.hpp>
#include <lp/rgba.hpp>

using namespace lp;
using namespace lp::io;

int main(int argc, char **argv) {
  ImageSize size{500, 500};                                    /* size of the output image */
  const size_t rawsize = size.PixelCount();

  OpenCV<RGBA<uint8_t>> io.Open(image_path);
  Image<RGBA<uint8_t>> img = io.ReadImage();
  Image<RGBA<uint8_t>> dst =
      Image(size, std::shared_ptr<RGBA<uint8_t>[]>(new RGBA<uint8_t>[rawsize]));
  
  engines::EquirectangularToRectilinear<RGBA<uint8_t>> engine;
  engines::EquirectangularToRectilinearParams params{{
                                                       {0.0f, 0.0f},
                                                       2.0f,
                                                     },
                                                     {
                                                       dst.GetSize(),
                                                     },
                                                     {io.GetSize()}};
  
  engine.SetParameters(params);

  for (int i = 1; i <= 100; i++) {
    /* Process image */
    engine.Process(img, dst);

    /* Do anything you want with dst (display, stream, save, ...) */

    /* Update parameters */
    params.equirectangular.r -= 0.05;                          /* zoom out */
    params.equirectangular.viewpoint + Point2D{0.05f, 0.00f};  /* tilt right */
    params.equirectangular.viewpoint + Point2D{0.00f, 0.05f};  /* pan up */
    engine.SetParameters(params);
  }
}

GstRrPanoramaptz

After Building and Installation, follow these steps:

The GstRrPanoramaptz plugin allows for real-time PTZ adjustments on panoramic video feeds, enabling users to explore video scenes in greater detail or from different perspectives.

Overview

Features

• CUDA-accelerated PTZ transformations: Leverage the power of NVIDIA CUDA technology. This acceleration helps with a smooth and high-performance video processing.
• Support for RGBA video format.
• Dynamic parameter adjustments: Users can dynamically adjust PTZ parameters such as pan, tilt, and zoom during playback, providing a versatile and interactive video experience.

Properties

The GstRrPanoramaptz plugin introduces three primary properties for real-time video manipulation:

• Pan (Horizontal Rotation): Adjusts the video feed's horizontal orientation. Pan adjustments allow viewers to rotate the video around its vertical axis, simulating a left or right looking direction.
  • Syntax: pan=<value>.
  • Range: -360 to 360 degrees.
  • Default: 0.

• Tilt (Vertical Rotation): This property adjusts the vertical viewing angle of the video feed. It simulates a vertical rotation of the camera view.
  • Syntax: tilt=<value>.
  • Range: -360 to 360 degrees.
  • Default: 0.

• Zoom: This property adjusts the zoom level of the video feed. It simulates moving the camera closer or further away from the scene.
  • Syntax: zoom=<value>.
  • Range: 0.1 to 10.
  • Behavior: Zoom out for zoom < 1, Zoom in for zoom > 1.
  • Default: 1.

Caps and Formats

• The plugin accepts and outputs video in the video/x-raw format, utilising the RGBA color space. This support ensures compatibility with a wide range of video processing scenarios.
• Enhanced performance on NVIDIA hardware is achieved through support for both system memory and NVMM (NVIDIA Multi-Media) memory inputs. This flexibility allows users to optimise their video processing pipelines based on the available hardware resources.

Basic use example

This pipeline creates a test video, then applies a 30 degree rotation to the right, tilts it upwards by 30 degrees, and a zoom level of 2.

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1920,height=1080" ! rrpanoramaptz pan=90 tilt=90 zoom=2 ! videoconvert ! autovideosink

You should see an output as the one below:

Libpanorama example

The example uses a standard video, not a panoramic one, causing some distortion, but we'll explore distortion-free examples with equirectangular images soon.