GstInference Embedding application

This example receives video file, images, or camera stream as input and process each frame to detect if a face is valid. For each processed frame the application captures the signal emitted by GstInference, forwarding the prediction to a placeholder for external logic which computes if the metadata belongs to a valid face. Simultaneously, the pipeline displays the captured frames with the associated label in a window in case of a valid face, otherwise, a fail label will be displayed. Note that the image currently being displayed not necessarily matches the one being handled by the signal. The display is meant for visualization and debugging purposes.

The recognition architecture being used by the example is FaceNetV1 trained using the Large Scale CelebFaces Attributes Dataset. A pre-trained model can be downloaded from the GstInference Model Zoo

These examples serve both as an example and as a starting point for a Face Recognition application.

Building the Example

The example builds along with the GstInference project. Make sure you follow the instructions in Building the Plug-In to make sure all the dependencies are correctly fulfilled.

Once the project is built the example may be built independently by running make within the example directory.

cd tests/examples/embedding
make

The example is not meant to be installed.

Running the Example

The embedding application provides a series of cmdline options to control the behavior of the example. The basic usage is:

./embedding -m MODEL -f FILE -b BACKEND -e EMBEDDINGS -l LABELS [-v]

-m|--model: Mandatory. Path to the FaceNetV1 trained model
-f|--file: Optional. Path to the video file or image to be used; If it is not set, a camera stream will be used.
-b|--backend: Mandatory. Name of the backend to be used. See Supported Backends for a list of possible options.
-e|--embeddings: Mandatory. Path to the text file with embeddings data containing valid faces
-l|--labels: Mandatory. Path to the text file with labels associated with the embeddings file
-v: Optional. Run verbosely.

You may always run --help for further details.

./embedding --help
Usage:
  embedding [OPTION...]  - GstInference Embedding Example

Help Options:
  -h, --help                        Show help options
  --help-all                        Show all help options
  --help-gst                        Show GStreamer Options

Application Options:
  -v, --verbose                     Be verbose
  -m, --model                       Model path
  -e, --embeddings                  Path to file with encoded valid faces
  -l, --labels                      Path to labels from valid faces
  -f, --file                        File path (or camera, if omitted)
  -b, --backend                     Backend used for inference, example: tensorflow

Extending the Application

The example is composed by the following main files:

gstembedding.c: Source file with GStreamer logic
customlogic.c: Placeholder for custom logic
customlogic.h: Header file for custom logic source
Makefile: Build script
embedding: Executable generated on build
images: Images directory with the valid faces used to generate demo labels and embeddings
embeddings: Directory with demo files embeddings.txt and labels.txt using the faces from images directory

The application can be extended by filling in the placeholders in customlogic.c. In the most simple cases, the information provided in handle_prediction should suffice to react to the prediction.

void
handle_prediction (unsigned char *image,
                   int width,
                   int height,
                   unsigned int size,
                   double *embeddings,
                   int num_dimensions,
                   int verbose,
                   char **embeddings_list,
                   char **labels,
                   int num_embeddings)
{
    /* FILLME: Handle image and prediction here */
}

image: Image data in RGB raster format.
width: The width of the image, in pixels
height: The height of the image, in pixels
size: The size of the total image, in bytes
embeddings: An array of encoded data
num_dimensions: The length of the embeddings array
verbose: A flag to display the encoded metadata from the current frame
embeddings_list: An array of strings containing the metadata from valid faces
labels: The list of labels used to generate the embeddings file
num_embeddings: The amount of valid faces

Troubleshooting

The first debug level is GStreamer debug. You may execute the application with debug enabled by running:

./embeddings --gst-debug=2

For advanced support, please contact [1] with the output of the following command:

./embeddings --gst-debug=2,videoinference:6,facenetv1:6

Reporting a Bug

Please feel free to report bugs using GitHub's issue tracker.

Previous: Example Applications/Detection

Index

Next: Example Applications/Smart Lock

❯