GstInference - Example pipelines - IMX8

GstInference
Introduction
Getting started
Getting the code Building the plugin Quick Starting Guide
Supported architectures
InceptionV1 InceptionV2 InceptionV3 InceptionV4 TinyYoloV2 TinyYoloV3 YoloV2 ResNetV1 MobileNetV2 MobileNetV2 + SSD AlexNet
Supported backends
TensorFlow Tensorflow-Lite Coral from Google Caffe TensorRT ONNXRT ONNXRT ACL ONNXRT OpenVINO
Metadata and Signals
GstClassificationMeta GstInferenceMeta Signals
Overlay Elements
Inference Overlay
Utils Elements
InferenceFilter InferenceCrop InferenceDebug InferenceBin
Legacy pipelines
PC NANO TX2 Xavier IMX8
Example pipelines
Example applications
Classification Detection Hand Pose Recognition
Benchmarks
Model Zoo
Project Status
Releases Roadmap
Contact Us

Tensorflow

Inceptionv2 inference on image file using Tensorflow

• Get the graph used on this example from this link
• You will need a video file from one of ImageNet classes
• Pipeline

IMAGE_FILE=cat.jpg
MODEL_LOCATION='graph_inceptionv2_tensorflow.pb'
INPUT_LAYER='input'
OUTPUT_LAYER='Softmax'
GST_DEBUG=inceptionv2:6 gst-launch-1.0 \
multifilesrc location=$IMAGE_FILE start-index=0 stop-index=0 loop=true  ! jpegparse ! jpegdec ! videoconvert ! videoscale ! videorate ! queue ! net.sink_model \
inceptionv2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER  backend::output-layer=$OUTPUT_LAYER

• Output

0:00:09.549749856 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:10.672917685 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:10.672976676 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 284 : (0,691864)
0:00:10.673064576 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:11.793890820 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:11.793951581 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 284 : (0,691864)
0:00:11.794041207 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:12.920027410 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:12.920093762 26945       0xaf9cf0 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 284 : (0,691864)

Inceptionv2 inference on video file using Tensorflow

• Get the graph used on this example from this link
• You will need a video file from one of ImageNet classes
• Pipeline

VIDEO_FILE='cat.mp4'
MODEL_LOCATION='graph_inceptionv2_tensorflow.pb'
INPUT_LAYER='input'
OUTPUT_LAYER='Softmax'
GST_DEBUG=inceptionv2:6 gst-launch-1.0 \
filesrc location=$VIDEO_FILE ! decodebin ! videoconvert ! videoscale ! queue ! net.sink_model \
inceptionv2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER  backend::output-layer=$OUTPUT_LAYER

• Output

0:00:11.878158663 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:13.006776924 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:13.006847113 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 282 : (0,594995)
0:00:13.006946305 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:14.170203673 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:14.170277808 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 282 : (0,595920)
0:00:14.170384768 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:15.285901546 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:15.285964794 27048      0x1d49800 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 282 : (0,593185)

Inceptionv2 inference on camera stream using Tensorflow

• Get the graph used on this example from this link
• You will need a v4l2 compatible camera
• Pipeline

CAMERA='/dev/video0'
MODEL_LOCATION='graph_inceptionv2_tensorflow.pb'
INPUT_LAYER='input'
OUTPUT_LAYER='Softmax'
GST_DEBUG=inceptionv2:6 gst-launch-1.0 \
v4l2src device=$CAMERA ! videoconvert ! videoscale ! queue ! net.sink_model \
inceptionv2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER  backend::output-layer=$OUTPUT_LAYER

• Output

0:00:14.614862363 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:15.737842669 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:15.737912053 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 838 : (0,105199)
0:00:15.738007534 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:16.855603761 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:16.855673578 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 838 : (0,093981)
0:00:16.855768558 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:199:gst_inceptionv2_preprocess:<net> Preprocess
0:00:17.980784789 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:231:gst_inceptionv2_postprocess:<net> Postprocess
0:00:17.980849612 27227      0x19cd4a0 LOG              inceptionv2 gstinceptionv2.c:252:gst_inceptionv2_postprocess:<net> Highest probability is label 838 : (0,077824)

Inceptionv2 visualization with classification overlay Tensorflow

• Get the graph used on this example from this link
• You will need a v4l2 compatible camera
• Pipeline

CAMERA='/dev/video0'
MODEL_LOCATION='graph_inceptionv2_tensorflow.pb'
INPUT_LAYER='input'
OUTPUT_LAYER='Softmax'
LABELS='imagenet_labels.txt'
gst-launch-1.0 \
v4l2src device=$CAMERA ! "video/x-raw, width=1280, height=720" ! tee name=t \
t. ! videoconvert ! videoscale ! queue ! net.sink_model \
t. ! queue ! net.sink_bypass \
inceptionv2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER  backend::output-layer=$OUTPUT_LAYER \
net.src_bypass ! videoconvert ! classificationoverlay labels="$(cat $LABELS)" font-scale=4 thickness=4 ! videoconvert ! xvimagesink sync=false

• Output

Classification overlay example output

TinyYolov2 inference on image file using Tensorflow

• Get the graph used on this example from this link
• You will need an image file from one of TinyYOLO classes
• Pipeline

IMAGE_FILE='cat.jpg'
MODEL_LOCATION='graph_tinyyolov2_tensorflow.pb'
INPUT_LAYER='input/Placeholder'
OUTPUT_LAYER='add_8'
GST_DEBUG=tinyyolov2:6 gst-launch-1.0 \
multifilesrc location=$IMAGE_FILE start-index=0 stop-index=0 loop=true  ! jpegparse ! jpegdec ! videoconvert ! videoscale ! videorate ! queue ! net.sink_model \
tinyyolov2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER  backend::output-layer=$OUTPUT_LAYER

• Output

0:00:06.401015400 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:06.817243785 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:06.817315935 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:7, x:-55,170727, y:25,507316, width:396,182867, height:423,241143, prob:14,526398]
0:00:06.817426814 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:07.236310555 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:07.236379100 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:7, x:-55,170727, y:25,507316, width:396,182867, height:423,241143, prob:14,526398]
0:00:07.236486242 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:07.659870194 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:07.659942388 12340      0x1317cf0 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:7, x:-55,170727, y:25,507316, width:396,182867, height:423,241143, prob:14,526398]

TinyYolov2 inference on video file using Tensorflow

• Get the graph used on this example from this link
• You will need a video file from one of TinyYOLO classes
• Pipeline

VIDEO_FILE='cat.mp4'
MODEL_LOCATION='graph_tinyyolov2_tensorflow.pb'
INPUT_LAYER='input/Placeholder'
OUTPUT_LAYER='add_8'
GST_DEBUG=tinyyolov2:6 gst-launch-1.0 \
filesrc location=$VIDEO_FILE ! decodebin ! videoconvert ! videoscale ! queue ! net.sink_model \
tinyyolov2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER  backend::output-layer=$OUTPUT_LAYER

• Output

0:00:08.545063684 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:08.955522899 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:08.955600820 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:7, x:-36,012765, y:-37,118160, width:426,351621, height:480,353663, prob:14,378592]
0:00:08.955824676 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:09.364908234 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:09.364970901 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:7, x:-36,490694, y:-38,108817, width:427,474399, height:482,318385, prob:14,257683]
0:00:09.365090340 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:09.775848590 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:09.775932404 12504       0xce4400 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:7, x:-35,991940, y:-37,482425, width:426,533537, height:480,917142, prob:14,313076]

TinyYolov2 inference on camera stream using Tensorflow

• Get the graph used on this example from this link
• You will need a v4l2 compatible camera
• Pipeline

CAMERA='/dev/video0'
MODEL_LOCATION='graph_tinyyolov2_tensorflow.pb'
INPUT_LAYER='input/Placeholder'
OUTPUT_LAYER='add_8'
GST_DEBUG=tinyyolov2:6 gst-launch-1.0 \
v4l2src device=$CAMERA ! videoconvert ! videoscale ! queue ! net.sink_model \
tinyyolov2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER  backend::output-layer=$OUTPUT_LAYER

• Output

0:00:06.823064776 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:07.242114002 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:07.242183276 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:14, x:116,796387, y:-31,424289, width:240,876587, height:536,305261, prob:11,859128]
0:00:07.242293677 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:07.660324555 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:07.660388215 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:14, x:113,453324, y:-27,681194, width:248,010337, height:528,964842, prob:11,603928]
0:00:07.660503502 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:479:gst_tinyyolov2_preprocess:<net> Preprocess
0:00:08.079154860 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:501:gst_tinyyolov2_postprocess:<net> Postprocess
0:00:08.079230404 12678       0xec24a0 LOG               tinyyolov2 gsttinyyolov2.c:384:print_top_predictions:<net> Box: [class:14, x:113,736444, y:-33,747251, width:246,987389, height:541,188374, prob:11,888664]

TinyYolov2 visualization with detection overlay Tensorflow

• Get the graph used on this example from this link
• You will need a v4l2 compatible camera
• Pipeline

CAMERA='/dev/video0'
MODEL_LOCATION='graph_tinyyolov2_tensorflow.pb'
INPUT_LAYER='input/Placeholder'
OUTPUT_LAYER='add_8'
LABELS='labels.txt'
gst-launch-1.0 \
v4l2src device=$CAMERA ! "video/x-raw, width=1280, height=720" ! tee name=t \
t. ! videoconvert ! videoscale ! queue ! net.sink_model \
t. ! queue ! net.sink_bypass \
tinyyolov2 name=net model-location=$MODEL_LOCATION backend=tensorflow backend::input-layer=$INPUT_LAYER backend::output-layer=$OUTPUT_LAYER \
net.src_bypass ! videoconvert ! detectionoverlay labels="$(cat $LABELS)" font-scale=1 thickness=2 ! videoconvert ! xvimagesink sync=false

• Output

Classification overlay example output