AI Based Object Redaction/Examples/Library Examples

AI Based Object Redaction
Overview
Architecture Supported Platforms Capabilities
Getting Started
Dependencies Evaluating librrobjectredaction Building librrobjectredaction
GStreamer Plugin
rrfacedetector rrobjectredactor
Examples
Library Examples GStreamer Pipelines
Performance
Jetson Xavier NX Jetson Xavier AGX
Troubleshooting
FAQ
Contact Us

In this section will be explained an example for face redaction running on GPU. The first thing you want to do is to include the library header as follows:

#include <rd/redaction.hpp>

Backend

First, we create the backend object. The backend provides factories to create the redaction algorithm and buffers. It provides the user the ability to select the desired backend for the execution of the algorithm for object redaction. The backend could be CPU or GPU.

std::shared_ptr<rd::IBackend> backend = std::make_shared<rd::gpu::Backend>();

In case the input buffer is not already in GPU memory, we also need to create a CPU backend to allocate the buffer in CPU memory.

std::shared_ptr<rd::IBackend> cpu_backend = std::make_shared<rd::cpu::Backend>();

Get algorithm

The GetAlgorithm method is used to obtain the redaction algorithm to process the input buffer.

std::shared_ptr<rd::IRedaction> algorithm = backend->GetAlgorithm();

Get Model

The getModel method is used to obtain the AI model that will be used for the detection of the desired object. In this case: faces.

std::shared_ptr<rd::IModel> model = backend->getModel(rd::Model::FACE_DETECTION);

Buffers

Buffers are the structures used to manipulate and load the data corresponding to the video frames. A buffer consists of a resolution and a format.

Resolution

Resolution is a structure that consists of two parameters: width and height. Here we define the input resolution as 1080x720.

#define INPUT_WIDTH 1080
#define INPUT_HEIGHT 720
rd::Resolution input_resolution = rd::Resolution(INPUT_WIDTH, INPUT_HEIGHT);

Format

Format is an enumeration of values for the supported formats, which are: RGBA, RGB, GREY and YUV. Here we define the input format as YUY2.

rd::Format input_format = rd::Format::YUY2;

Allocate Buffers

With the resolution and formats defined, the buffer objects can be created.

With the cpu backend create an input and output buffers in CPU memory with the input video/image resolution and format. The input buffer must contain the image/frame data deploy in an array containing a pointer to each color component of the data.

std::shared_ptr<rd::io::IBuffer> input = backend_cpu->getBuffer(imageData, input_resolution, input_format);
std::shared_ptr<rd::io::IBuffer> output_final = backend_cpu->getBuffer(input_resolution, input_format);

With the gpu backend create an input and output buffers in GPU memory with the input video/image resolution and format.

std::shared_ptr<rd::io::IBuffer> input_gpu = backend->getBuffer(input_resolution, input_format);
std::shared_ptr<rd::io::IBuffer> output = backend->getBuffer(input_resolution, input_format);

The CPU input buffer must be moved to GPU memory when using GPU, to accomplish this, use the copyFromHost method to upload the input buffer to GPU memory.

input_gpu->copyFromHost(input);

Redaction Algorithm

The Object Redaction library comprises the following stages: detect, track (optional), and redact. These stages can be performed in a single step using the apply method or in a step-by-step process.

The Object Redaction library uses a vector of a structure Rectangle to save the detected and tracked faces coordinates in an image for the redaction algorithm to modify the output buffer. This vector must be initialized before performing the detect stage.

std::vector<rd::Rectangle> faces;

Step-by-step

• The first step is to detect the faces in the input image and save the coordinates in the vector of rectangles.

algorithm->detect(model, input_gpu, &faces);

algorithm->convert(input_gpu, input_convert);

• The second and optional step would be to perform tracking (this is a work in progress)

• The final step is to redact the detected faces in the given coordinates.

algorithm->redact(input_gpu, output, faces, rd::RedactionAlgorithm::BLURRING);

Apply method

To apply the redaction algorithm in a single step use the apply method set by the algorithm. This method

rd::IRedaction::apply(backend, model, input_gpu, output, &faces, rd::RedactionAlgorithm::BLURRING);

Download buffer to CPU memory

When using GPU the output buffer must be moved back to CPU memory. To accomplish this, use the copyToHost method to download the output buffer to CPU memory.

output->copyToHost(output_final);

The output final buffer contains the modified image where the detected faces have been redacted.

Full example

The full example script should look like:

#include <rd/redaction.hpp>

#include <unistd.h>

#include <fstream>
#include <iostream>
#include <memory>
#include <string>

#define INPUT_WIDTH 1080
#define INPUT_HEIGHT 720
#define INPUT_BPP 2

static void save_buffer(std::shared_ptr<rd::io::IBuffer> buffer,
                        std::string name) {
  /*Save the buffer*/
  std::vector<unsigned char*> data = buffer->data();
  uint size = buffer->stride()[0] * buffer->size().height;

  FILE* file = fopen(name.c_str(), "wb");
  fwrite(data[0], size, 1, file);
  fclose(file);
}

int main() {
  /* Open the image file using fstream */
  std::ifstream file(SEVEN_FACES, std::ios::binary);

  if (!file.is_open()) {
    std::cerr << "Error: Unable to open the image file." << std::endl;
    return -1;
  }
  /* Determine the file size */
  int file_size = INPUT_WIDTH * INPUT_HEIGHT * INPUT_BPP;

  /* Read the image data into a vector */
  unsigned char* data_ptr = new unsigned char[file_size];
  std::vector<unsigned char*> imageData;
  file.read(reinterpret_cast<char*>(data_ptr), file_size);
  imageData.push_back(data_ptr);

  /* Create GPU Backend */
  std::shared_ptr<rd::IBackend> backend = std::make_shared<rd::gpu::Backend>();
  /* Create CPU backend to save the final image */
  std::shared_ptr<rd::IBackend> backend_cpu =
      std::make_shared<rd::cpu::Backend>();

  /* Get Algorithm */
  std::shared_ptr<rd::IRedaction> algorithm = backend->getAlgorithm();
  std::shared_ptr<rd::IModel> model =
      backend->getModel(rd::Model::FACE_DETECTION);

  /* Buffers */
  rd::Resolution input_resolution = rd::Resolution(INPUT_WIDTH, INPUT_HEIGHT);
  rd::Format input_format = rd::Format::YUY2;

  /* Allocate Buffers */
  std::shared_ptr<rd::io::IBuffer> input =
      backend_cpu->getBuffer(imageData, input_resolution, input_format);
  std::shared_ptr<rd::io::IBuffer> output_final =
      backend_cpu->getBuffer(input_resolution, input_format);
  std::shared_ptr<rd::io::IBuffer> output =
      backend->getBuffer(input_resolution, input_format);
  std::shared_ptr<rd::io::IBuffer> input_gpu =
      backend->getBuffer(input_resolution, input_format);

  /*Copy input buffer to GPU*/
  input_gpu->copyFromHost(input);

  /* Detect Faces */
  std::vector<rd::Rectangle> faces;
  algorithm->detect(model, input_gpu, &faces);

  /* Print out detected faces */
  std::cout << faces.size() << std::endl;
  for (size_t i = 0; i < faces.size(); i++) {
    std::cout << faces[i] << std::endl;
  }

  /* Redact detected faces */
  algorithm->redact(input_gpu, output, faces, rd::RedactionAlgorithm::BLURRING);

  /* Download buffer back to CPU memory */
  output->copyToHost(output_final);

  /* Save redacted image */
  save_buffer(output_final, "output_final");
  std::cout << "Exit!!" << std::endl;

  return 0;
}