CUDA ISP for NVIDIA Jetson API usage

General

CUDA ISP provides several algorithms to process raw image data from a camera sensor in a variety of ways.

Here we illustrate a simple example that involves all the basic algorithms. This example processes a raw BAYER10 image into a I420 image with white balancing using the following steps:

The shift algorithm converts BAYER10 into BAYER8
The debayer algorithm converts BAYER8 to RGB
The white balancer algorithm improves the color balance in the RGB image without changing the format
The color space converter converts RGB to I420

For a detailed explanation of each algorithm, please refer to CUDA_ISP_for_NVIDIA_Jetson/CUDA_ISP_for_NVIDIA_Jetson_Basics.

The following diagram illustrates the complete processing pipeline:

A simple process of this type can be used to encode camera information into a video file or a network stream.

Backend

We first create a backend object. The backend object provides factories to create algorithms and buffers. For CUDA ISP, we specify CUDA as the desired backend type. All algorithms created using this backend will be CUDA accelerated.

auto backend = rr::IBackend::Build(rr::kCUDA);

Creating the algorithms

The creation of any given algorithm requires a single function call:

auto shift                 = backend->CreateAlgorithm(rr::kShift);
auto debayer               = backend->CreateAlgorithm(rr::kDebayer);
auto white_balancer        = backend->CreateAlgorithm(rr::kWhiteBalancer);
auto color_space_converter = backend->CreateAlgorithm(rr::kColorSpaceConverter);

Creating the buffers

The image data is stored in CUDA ISP Buffers. These Buffers are used as inputs and outputs to the algorithms. Buffers can be created using the backend CreateBuffer method, similar to the CreateAlgorithms method. The CreateBuffer method requires the buffer size and a parameter object. In CUDA ISP, parameter objects are used to provide many methods with additional information. In this example, the parameter object used in buffer creation indicates the buffer CUDA memory type (host, device, unified or external). See #Parameter objects for more detail.

auto bayer10_buffer        = backend->CreateBuffer(bayer10_size, buffer_params);
auto bayer8_buffer         = backend->CreateBuffer(bayer8_size,  buffer_params);
auto unbalanced_rgb_buffer = backend->CreateBuffer(rgb_size,     buffer_params);
auto balanced_rgb_buffer   = backend->CreateBuffer(rgb_size,     buffer_params);
auto i420_buffer           = backend->CreateBuffer(i420_size,    buffer_params);

Parameter objects

Parameter objects are created directly and not through a factory method. There are several different parameter objects used in different methods. The general list of parameter objects is as follows:

ColorSpaceParams - Used to indicate whether to apply an RGB to I420 conversion or an I420 to RGB conversion during a ColorSpaceConverter application.
CudaBufferParams - Used to indicate buffer CUDA memory type while creating a Buffer using the CUDA backend. The memory type can be Host, Device, Unified or External.
DebayerParams - Used to indicate the input Bayer layout during a Debayer application. The Bayer layout can be RGGB, BGGR, BRGB, or BGRB.
ShiftParams - Used to indicate the shift value during a Shift application. The shift value refers to how many bits to right-shift each input bayer pixel before converting to BAYER8.

This example creates a CudaBufferParams object and sets its CUDA memory type to external:

auto buffer_params = std::make_shared<CudaBufferParams>();
buffer_params->type = CudaType::kExternal;

Applying the algorithms

Each algorithm has an apply method that expects an input buffer, output buffer and parameter object.

shift->Apply(bayer10_buffer, bayer8_buffer, shift_params);
debayer->Apply(bayer8_buffer, unbalanced_rgb_buffer, debayer_params);
white_balancer->Apply(unbalanced_rgb_buffer, balanced_rgb_buffer, white_balancer_params);
color_space_converter->Apply(balanced_rgb_buffer, i420_buffer, color_space_converter_params);

More information

You can find more examples in our User Manual.

Previous: User Manual

Index

Next: Examples/GStreamer usage

❯