GstInference and GstEmbeddingMeta metadata

GstInference
Introduction
Getting started
Getting the code Building the plugin Quick Starting Guide
Supported architectures
InceptionV1 InceptionV2 InceptionV3 InceptionV4 TinyYoloV2 TinyYoloV3 YoloV2 ResNetV1 MobileNetV2 MobileNetV2 + SSD AlexNet
Supported backends
TensorFlow Tensorflow-Lite Coral from Google Caffe TensorRT ONNXRT ONNXRT ACL ONNXRT OpenVINO
Metadata and Signals
GstClassificationMeta GstEmbeddingMeta GstDetectionMeta GstInferenceMeta Signals
Overlay Elements
Inference Overlay
Utils Elements
InferenceFilter InferenceCrop InferenceDebug InferenceBin
Legacy pipelines
PC NANO TX2 Xavier IMX8
Example pipelines
Example applications
Classification Detection Hand Pose Recognition
Benchmarks
Model Zoo
Project Status
Releases Roadmap
Contact Us

This metadata consists of a variable size elements array, that is filed with the embedding produced by the net, like facenet that uses a 128 elements array size.

GstEmbeddingMeta example

Fields

The facenet element and embedding overlay uses similar metadata as the classification plugins. GstEmbeddingMeta consist on the following fields:

field	type	description
num_dimensions	gint	The number of labels outputted by the model. This can vary from model to model. Facenet uses 128.
embedding	gdouble *	The embedding produced by the network

Access metadata

If you want to access this metadata from your custom Gstreamer element instead the process is fairly easy:

1. Add the Facenet element to your pipeline
2. Include GstInference metadata header: #include "gst/r2inference/gstinferencemeta.h"
3. Get a GstClassificationMeta object from the buffer: class_meta = (GstClassificationMeta *) gst_buffer_get_meta (frame->buffer, GST_CLASSIFICATION_META_API_TYPE);

FaceNet also raises a signal containing GstClassificationMeta, for details on how to use this signal please check the example applications section.

In the following section of code, we add the include like in point 2 and safe the GstClassificationMeta in the class_meta variable like in point 3.

#include "gst/r2inference/gstinferencemeta.h"
static void 
get_buffer(GstPadProbeInfo * info)
{
  GstBuffer *buffer;
  GstDetectionMeta *meta;
  buffer = gst_pad_probe_info_get_buffer (info);
  class_meta = (GstClassificationMeta *) gst_buffer_get_meta (buffer,
                  GST_DETECTION_META_API_TYPE);

  g_print ("Dimension: 0 has embedding %f\n", class_meta->embedding[0]);
}