Spherical Video PTZ/User Guide/Quick Start Guide

Spherical Video PTZ
Getting Started
Spherical Video PTZ Projections Used Evaluating the Product Getting the code
User Guide
Building and Installation Quick Start Guide
Examples
Gst-launch GstD
Performance
Jetson AGX Xavier
Contact Us

Libpanorama

This wiki introduces a basic use of Spherical Video PTZ for converting equirectangular images to rectilinear format with an engine. It includes a simple example and instructions on how to use the engine for different needs. The engine makes it easy to change panoramic images into a straight view, useful for many projects.

Minimal Application

After Building and Installation, follow these steps:

1. Download the sample images, if you haven't already.

cd $SAMPLES
./download_samples.sh

2. This example demonstrates the use of the Spherical Video PTZ engine to convert equirectangular images into rectilinear format. This command processes example_image.jpg, converting it from an equirectangular format to a rectilinear view. But you can use any other reference image as long as it is equirectangular. Run the example as:

cd $LIBPANORAMA_PATH/libpanorama
./builddir/examples/equirectangular_to_rectilinear $SAMPLES/example_image.jpg

3. For this example you can use the interactive controls with the Spherical Video PTZ (Pan-Tilt-Zoom) for dynamic exploration of panoramic images. Hit the specified keys when the example is running:

• Zoom In/Out: Adjust the zoom level to get a closer view or a wider perspective of the image.
  • In: i
  • Out: o
• Pan Left/Right: Rotate the view horizontally to explore the left or right sides of the panoramic image.
  • Left: 4
  • Right: 6
• Tilt Up/Down: Adjust the vertical angle of the camera to look up or down within the panoramic image.
  • Up: 8
  • Down: 2

4. press the Esc key to exit the program.

Spherical Video PTZ Engine

The Video PTZ Engine simplifies the use of PTZ controls, making it easier to integrate into your code. To get started, include the wrapper in your code and instantiate the class as demonstrated below:

#include "lp/engines/equirectangular_to_rectilinear.hpp"

lp::engines::EquirectangularToRectilinear<RGBA<uint8_t>> engine;

Then, set the initial parameters with the SetParameters method:

lp::engine.SetParameters(params);

Next, configure the parameters to be manipulated using the properties of the Spherical Video PTZ. Instantiate them as demonstrated in the code below. The initial parameters, represented as {{0.0f, 0.0f}, 2.0f}, control the panning, tilting, and zooming of the image in the format {{pan, tilt}, zoom}. Additionally, {dst.GetSize()} is used for data normalization in {x, y} format, while {io.GetSize()} is utilized for data denormalization, also in {x, y} format.

engines::EquirectangularToRectilinearParams params{{
                                                     {0.0f, 0.0f},
                                                     2.0f,
                                                   },
                                                   {
                                                     dst.GetSize(),
                                                   },
                                                   {io.GetSize()}};

The first parameters, represented as {{0.0f, 0.0f}, 2.0f}, are used to adjust the panning, tilting, and zoom of the image, following this format: {{pan, tilt}, zoom}. Additionally, {dst.GetSize()} is used for data normalization with {x, y} format, while {io.GetSize()} is used for data denormalization with {x, y} format.

Finally, use the Process method with the input image as the first parameter. The second parameter will contain the result after applying the equirectangular to rectilinear projection transformation.

#include "lp/allocators/cudaimage.hpp"
#include "lp/image.hpp"

lp::Image<RGBA<uint8_t>> img;
lp::allocators::CudaImage<RGBA<uint8_t>> dst;

lp::engine.Process(img, dst);

Please note that the Engine supports both CudaImages and Images for processing. However, if you choose to use an Image, the Engine will internally allocate a Cuda buffer and copy the Image content into it, potentially affecting the application performance. Take this pseudo-code snippet as an loop example of the engine's usage:

#include "lp/image.hpp"

int main(int argc, char **argv) {
  ImageSize size{500, 500};                                    /* size of the output image */
  const size_t rawsize = size.PixelCount();

  OpenCV<RGBA<uint8_t>> io.Open(image_path);
  Image<RGBA<uint8_t>> img = io.ReadImage();
  lp::Image<RGBA<uint8_t>> dst =
      Image(size, std::shared_ptr<RGBA<uint8_t>[]>(new RGBA<uint8_t>[rawsize]));
  
  engines::EquirectangularToRectilinear<RGBA<uint8_t>> engine;
  engines::EquirectangularToRectilinearParams params{{
                                                       {0.0f, 0.0f},
                                                       2.0f,
                                                     },
                                                     {
                                                       dst.GetSize(),
                                                     },
                                                     {io.GetSize()}};
  
  engine.SetParameters(params);
  engine.Process(img, dst);

  for (int i = 1; i <= 100; i++) {
    params.equirectangular.r -= 0.05;                          /* zoom out */
    params.equirectangular.viewpoint + Point2D{0.05f, 0.00f};  /* tilt right */
    params.equirectangular.viewpoint + Point2D{0.00f, 0.05f};  /* pan up */

    engine.SetParameters(params);
    engine.Process(img, dst);
  }
}

GstRrPanoramaptz

After Building and Installation, follow these steps:

The GstRrPanoramaptz plugin allows for real-time PTZ adjustments on panoramic video feeds, enabling users to explore video scenes in greater detail or from different perspectives.

Overview

Features

• CUDA-accelerated PTZ transformations: Leverage the power of NVIDIA CUDA technology. This acceleration helps with a smooth and high-performance video processing.
• Support for RGBA video format.
• Dynamic parameter adjustments: Users can dynamically adjust PTZ parameters such as pan, tilt, and zoom during playback, providing a versatile and interactive video experience.

Properties

The GstRrPanoramaptz plugin introduces three primary properties for real-time video manipulation:

• Pan (Horizontal Rotation): Adjusts the video feed's horizontal orientation. Pan adjustments allow viewers to rotate the video around its vertical axis, simulating a left or right looking direction.
  • Syntax: pan=<value>.
  • Range: -360 to 360 degrees.
  • Default: 0.

• Tilt (Vertical Rotation): This property adjusts the vertical viewing angle of the video feed. It simulates a vertical rotation of the camera view.
  • Syntax: tilt=<value>.
  • Range: -360 to 360 degrees.
  • Default: 0.

• Zoom: This property adjusts the zoom level of the video feed. It simulates moving the camera closer or further away from the scene.
  • Syntax: zoom=<value>.
  • Range: 0.1 to 10.
  • Behavior: Zoom out for zoom < 1, Zoom in for zoom > 1.
  • Default: 1.

Caps and Formats

• The plugin accepts and outputs video in the video/x-raw format, utilizing the RGBA color space. This support ensures compatibility with a wide range of video processing scenarios.
• Enhanced performance on NVIDIA hardware is achieved through support for both system memory and NVMM (NVIDIA Multi-Media) memory inputs. This flexibility allows users to optimize their video processing pipelines based on the available hardware resources.

Basic use example

This pipeline creates a test video, then applies a 0.5-degree rotation to the right, tilts it upwards by 0.5 degrees, and enhances the view with a zoom level of 2.

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1920,height=1080" ! rrpanoramaptz pan=0.5 tilt=0.5 zoom=2 ! videoconvert ! autovideosink

You should see an output as the one below:

Libpanorama example

The example uses a standard video, not a panoramic one, causing some distortion, but we'll explore distortion-free examples with equirectangular images soon.