GstInference and GstDetectionMeta metadata

GstInference
Introduction
Getting started
Getting the code Building the plugin Quick Starting Guide
Supported architectures
InceptionV1 InceptionV2 InceptionV3 InceptionV4 TinyYoloV2 TinyYoloV3 YoloV2 ResNetV1 MobileNetV2 MobileNetV2 + SSD AlexNet
Supported backends
TensorFlow Tensorflow-Lite Coral from Google Caffe TensorRT ONNXRT ONNXRT ACL ONNXRT OpenVINO
Metadata and Signals
GstClassificationMeta GstEmbeddingMeta GstDetectionMeta GstInferenceMeta Signals
Overlay Elements
Inference Overlay
Utils Elements
InferenceFilter InferenceCrop InferenceDebug InferenceBin
Legacy pipelines
PC NANO TX2 Xavier IMX8
Example pipelines
Example applications
Classification Detection Hand Pose Recognition
Benchmarks
Model Zoo
Project Status
Releases Roadmap
Contact Us

This metadata consists of an array of bounding boxes, that were previously filtered on the detection element to remove low probability and duplicated boxes.

GstDetectionMeta example

Fields

All detection elements on GstInference use the same metadata standard. GstDetectionMeta consist on the following fields:

field	type	description
num_boxes	gint	The number of boxes outputted by the model. This can vary over time.
boxes	BBox *	The array of bounding boxes.

Each bounding box is an struct with:

field	type	description
label	gint	The label represented.
prob	gdouble	The probability of the box. Note that the probability of detection works differently from classification. This probability does not reflect actual probability but rather the certainty level of the neural network.
x	gdouble	X coordinate (top left corner)
y	gdouble	Y coordinate (top left corner)
width	gdouble	Box width
height	gdouble	Box height

Access metadata

If you want to access this metadata from your custom Gstreamer element instead the process is fairly easy:

1. Add a GstInference detection element such as TinyYoloV2 to your pipeline
2. Include GstInference metadata header: #include "gst/r2inference/gstinferencemeta.h"
3. Get a GstDetectionMeta object from the buffer: detect_meta = (GstDetectionMeta *) gst_buffer_get_meta (frame->buffer, GST_DETECTION_META_API_TYPE);

All GstInference detection elements also raise a signal containing GstDetectionMeta, for details on how to use this signal please check the example applications section.

In the following section of code, we add the include like in point 2 and safe the GstDetectionMeta in the detect_meta variable like in point 3.

#include "gst/r2inference/gstinferencemeta.h"
static void 
get_buffer(GstPadProbeInfo * info)
{
  GstBuffer *buffer;
  GstDetectionMeta *meta;
  buffer = gst_pad_probe_info_get_buffer (info);
  detect_meta = (GstDetectionMeta *) gst_buffer_get_meta (buffer,
                  GST_DETECTION_META_API_TYPE);
  g_print ("Box: 0 has prob %f\n", detect_meta->boxes[0].prob);
}