GstInference and GstInferenceMeta metadata

GstInference
Introduction
Getting started
Getting the code Building the plugin Quick Starting Guide
Supported architectures
InceptionV1 InceptionV2 InceptionV3 InceptionV4 TinyYoloV2 TinyYoloV3 YoloV2 ResNetV1 MobileNetV2 MobileNetV2 + SSD AlexNet
Supported backends
TensorFlow Tensorflow-Lite Coral from Google Caffe TensorRT ONNXRT ONNXRT ACL ONNXRT OpenVINO
Metadata and Signals
GstClassificationMeta GstEmbeddingMeta GstDetectionMeta GstInferenceMeta Signals
Overlay Elements
Inference Overlay
Utils Elements
InferenceFilter InferenceCrop InferenceDebug InferenceBin
Legacy pipelines
PC NANO TX2 Xavier IMX8
Example pipelines
Example applications
Classification Detection Hand Pose Recognition
Benchmarks
Model Zoo
Project Status
Releases Roadmap
Contact Us

Overview

GstInferenceMeta is a new hierarchical approach towards storing inferred data on a buffer. This meta replace the old GstDetectionMeta and GstClassificationMeta combining the data generated by detection and classification elements in a single structure that simplifies the pipeline.

GstInferenceMeta follow this structure:

• The GstInferenceMeta is initialized with a root prediction (the complete image).
• Root prediction has a list of GstInferenceClassification.
• Root prediction contains n-ary tree structure, in which each node contains a prediction, a pointer to struct of type GstInferencePrediction.
• Each prediction child has an associated bounding box (Filled by detection elements).
• Each prediction child has a list of GstInferenceClassification (Filled by classification and detection elements).
• Finally each prediction child could have child predictions of type GstInferencePrediction.

GstInferenceMeta Structure

The following section shows how this structure is used in a real scenario.

Real Example

Consider the following scene with a cat and a dog.

Graphical Representation

Detection with new metadata

Prediction Metadata Diagram

The information in the scene is summarized as in the following diagram:

Full inferencemeta diagram example

Serialized Prediction

GstInference will serialize the structure above as a recursive JSON object as the following:

{
  id : 597,
  enabled : True,
  bbox : {
    x : 0
    y : 0
    width : 416
    height : 416
  },
  classes : [
    
  ],
  predictions : [
    {
      id : 598,
      enabled : True,
      bbox : {
        x : 30
        y : 73
        width : 161
        height : 318
      },
      classes : [
        {
          Id : 648
          Class : 7
          Label : cat
          Probability : 16,072840
          Classes : 20
        }, 
      ],
      predictions : [
        
      ]
    }, {
      id : 599,
      enabled : True,
      bbox : {
        x : 211
        y : 38
        width : 182
        height : 406
      },
      classes : [
        {
          Id : 649
          Class : 11
          Label : dog
          Probability : 16,978619
          Classes : 20
        }, 
      ],
      predictions : [
        
      ]
    }, 
  ]
}

Detailed Description

The following tables provide the specific details of each structure within the prediction tree.

GstInferenceMeta

It is the GStreamer meta structure attached to the buffer. It contains the root prediction.

field	type	description
meta	GstMeta	Buffer metadata
prediction	GstInferencePrediction *	Contains all the predictions associated to this node.

GstInferencePrediction

Each prediction on each inference meta node contains the following information:

field	type	description
prediction_id	guint64	A unique id for this specific prediction.
enabled	gboolean	This flag indicates whether or not this prediction should be used for further inference.
bbox	BoundingBox	Bouding box for this specific prediction.
classifications	GList *	a linked list of GstInfereferenceClassification associated to this prediction.
predictions	GNode *	a n-ary tree of child predictions within this specific prediction. It is recommended to to access the tree directly, but to use this module's API to interact with the children.

Each bounding box is a struct defined as follows:

field	type	description
x	gint	horizontal coordinate of the upper left position of the bounding box in pixels
y	gint	vertical coordinate of the upper left position of the bounding box in pixels
width	guint	width of the bounding box in pixels
height	guint	height of the bounding box in pixels

GstInferenceClassification

Each prediction can contain a number of classifications. The GstInferenceClassification struct holds all the potentially relevant information regarding such classifications in the fields specified as follows.

field	type	description
classification_id	guint64	A unique id associated to this classification.
class_id	gint	The numerical id associated to the assigned class.
class_prob	gdouble	The resulting probability of the assigned class. Typically ranges between 0 and 1.
class_label	gchar *	The label associated to this class or NULL if not available.
num_classes	gint	The total amount of classes of the entire prediction.
probabilities	gdouble*	The entire array of probabilities of the prediction.
labels	gchar**	The entire array of labels of the prediction. NULL if not available.

Programmatic Metadata Access

If you want to access this metadata from your custom Gstreamer element instead the process is fairly easy:

1. Include GstInference metadata header: #include "gst/r2inference/gstinferencemeta.h"
2. Get a GstInferenceMeta object from the buffer: inference_meta = (GstInferenceMeta *) gst_buffer_get_meta (frame->buffer, GST_INFERENCE_META_API_TYPE);
3. Grab the prediction from the meta: pred = inference_meta->prediction;

The following snippet exemplifies how to interact with the prediction tree.

#include "gst/r2inference/gstinferencemeta.h"

static void 
get_buffer(GstPadProbeInfo * info)
{
  GstBuffer *buffer;
  GstInferenceMeta *meta;
  GstInferencePrediction *root, *child;
  GstInferenceClassification *classification;
  GSList *child_predictions;
  GList *classes;
  
  gchar *str;
  
  buffer = gst_pad_probe_info_get_buffer (info);
  meta = (GstInferenceMeta *) gst_buffer_get_meta (buffer,
      GST_INFERENCE_META_API_TYPE);
  root = meta->prediction;

  GST_INFERENCE_PREDICTION_LOCK (root);

  /* Print the entire prediction tree */
  str = gst_inference_prediction_to_string (root);
  GST_INFO ("Prediction tree: %s", str);
  g_free (str);

  /* Iterate through the immediate child predictions */
  for (child_predictions = gst_inference_prediction_get_children(root);
       child_predictions; child_predictions = g_slist_next (child_predictions)) {
    child = (GstInferencePrediction *)child_predictions->data;

    /* On each children, iterate through the different associated classes */
    for (classes = child->classifications;
         classes;
         classes = g_list_next (classes)) {
      classification = (GstInferenceClassification *)classes->data;
    }
  }
  
  GST_INFERENCE_PREDICTION_UNLOCK (root);
  
}