DeepStream 6.3 pipelines

Introduction

On this page, you are going to find a set of DeepStream pipelines used on Jetson Xavier NX DevKit. This wiki page tries to describe some of the DeepStream features for the NVIDIA platforms and other multimedia features. The goal is to provide you some example pipelines.

Some deepstream elements:

• nvinfer: The Gst-nvinfer element performs inference on input data using NVIDIA TensorRT. It accepts batched NV12/RGBA buffers, converts and scales the data according to network requirements, and passes it to the TensorRT engine for inference. This plugin can operate in primary, secondary, or preprocessed tensor input modes and supports various object detection and classification networks.
• nvtracker: The Gst-nvtracker element allows the DeepStream pipeline to use a tracking library to track detected objects with unique IDs over time. It converts input buffers as needed and processes data in batch mode for efficient GPU execution. The plugin updates metadata with tracked object coordinates, tracker confidence, and object IDs.
• nvvideotestsrc (Deepstream 6.4): The Gst-nvvideotestsrc element generates test video data using CUDA, creating output buffers on the GPU in various formats and patterns. It can introduce buffer latencies to simulate network conditions and also supports raw video input to generate GPU buffers. It is currently supported only on X86.
• nvvideoconvert: The Gst-nvvideoconvert element performs video color format conversion. It accepts NVMM and RAW memory and provides NVMM or RAW memory at the output. It supports various input and output formats and allows for scaling and cropping of images.
• nvmsgbroker : The Gst-nvmsgbroker element works like a messenger that sends data to a server using a specific protocol, such as Kafka or MQTT. It takes data that already has a special tag called NvDsPayload and sends it to the server. To work, it needs a special library that tells it how to use the chosen communication protocol.
• nvvideo4linux2(decoders/encoders): Gst-nvvideo4linux2 is used to decode and encode videos using special hardware on Jetson and dGPU devices. It decodes compressed videos and encodes uncompressed videos.
• nvdsmetautils (nvdsmetainsert/nvdsmetaextract) (Deepstream 6.4) Gst-nvdsmetautils contain nvdsmetainsert and nvdsmetaextract plugins. nvdsmetainsert adds metadata to video or audio buffers, allowing extra information to be attached to the data. nvdsmetaextract extracts this metadata from the buffers for use by other system components. Together, these elements enable the insertion and subsequent reading of additional information in multimedia streams, making it easier to handle metadata at different stages of data processing.
• Among others elements.

Deepstream Reference Apps

The DeepStream SDK includes several precompiled reference applications designed to help users understand and leverage the capabilities of DeepStream. Some of these applications can be found at: /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/. Now we will continue with the example of the deepstream-test1 app:

• Change to the directory

cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1

• Install dependencies

sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgstrtspserver-1.0-dev libx11-dev libjson-glib-dev libyaml-cpp-dev

• Export CUDA_VER

export CUDA_VER=11.4

• Compile

sudo -E make

• Run

./deepstream-test1-app dstest1_config.yml

Output:

You can explore the other examples by following the instructions in their respective README files, in each application directory located at /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps.

Pipelines - Jetson Xavier NX Developer Kit - Deepstream 6.3

Common deepstream pipeline stages:

1. Capture
2. Decode (nvv4l2decoder) This plugin's input streams are generally encoded camera streams (H264, H265, JPEG, MJPEG). The output is a stream of NV12 buffers batched together by Nvstreammux and provided as input to the inference engine, Nvinfer. This plugin is based on the GStreamer v4l2videodec plugin, but allows you to use hardware accelerated codecs on the Jetson devices.
3. Image-Processing
4. Batching (nvstreammux). The main objective of the nvstreammux plugin is to create a batch to feed into the pipeline’s inference engine, the nvinfer plugin.
5. Inference (nvinfer)
6. Object Tracking (nvtracker)
7. Overlay (nvdsosd)
8. Output

Test nvvideoconvert (DeepStream element) to crop

As a sanity check to validate the functioning of deepstream, you can run this pipeline that uses nvvideoconvert (a deepstream element) and explore its capabilities. For example, in this case it can be used to crop the image.

gst-launch-1.0 videotestsrc ! "video/x-raw, width=1280, height=720" ! nvvideoconvert src-crop="0:0:640:480" ! nv3dsink

Inference pipelines

The following DeepStream pipelines processes a video stream by reading it from a file, decoding it, batching the frames, running inference to detect objects, tracking those objects, overlaying the results, and encoding the processed video into a new file. It captures the video, decodes it using hardware acceleration, batches the frames, detects objects, tracks them, overlays the results and sends it to the sink.

The model configured in this file is a network based on ResNet10 trained for object detection. The .caffemodel and .prototxt files define the architecture and weights of the trained network. The .engine file is a TensorRT engine optimized for fast inference on the GPU. You can find this out by reading the config-file-path used.

The SDK includes some pre-trained models that you can use with the nvinfer element. You can find the configuration files at this path:

/opt/nvidia/deepstream/deepstream-6.3/samples/models

Primary_Detector + Overlay

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/overlay.mp4

You can play the video using:

gst-launch-1.0 filesrc location=~/make.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! nv3dsink

• Streaming:

For convenience, the following pipeline is provided, which is the same as the previous one but performs streaming.

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.yml ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nv3dsink -e

• Using RtspSink:

For convenience, the following pipeline is provided in case you want to check the compatibility of deepstream with rtspsink.

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.yml ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc insert-sps-pps=true idrinterval=30 ! video/x-h264, mapping=/stream1  ! perf ! rtspsink service=49152

On the client, export the server's IP address and the service port.

export IP_ADD=192.168.0.10

export PORT=49152

You can play the streaming:

gst-launch-1.0 rtspsrc location=rtsp://$IP_ADD:$PORT/stream1 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker)

In the past, DeepStream 5.1 included three low-level tracker libraries to configure the tracking mode in the nvtracker element:

• libnvds_mot_iou.so
• libnvds_mot_klt.so
• libnvds_nvdcf.so

All of these three low-level tracker libraries are deprecated in DeepStream 6.X, and a unified low-level tracker library is introduced, which is libnvds_nvmultiobjecttracker.so.

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/tracker.mp4

Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_CarColor

You could use more than one nvinfer element for inference and thus combine different objectives:

 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 unique-id=1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 unique-id=2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/color.mp4

Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_CarMake

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 unique-id=1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 unique-id=2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_carmake.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/make.mp4

Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_VehicleTypes

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 unique-id=1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 unique-id=2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/types.mp4

Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_VehicleTypes + Secondary_CarColor with GstD

Alternatively, you can use three types of inference in a single pipeline. This time, the solution is presented using GstD so you can see its use with Deepstream (GstD is a more flexible and powerful version of gst-launch; you can read more about it here: gstd). As you can see in the resulting video, the pipeline identifies three things:

• It is a car.
• It is a truck.
• It is a red car.

gstd-client pipeline_create p0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt" ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer3 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/gsdt.mp4

gstd-client element_set p0 nvinfer1 unique-id 1

gstd-client element_set p0 nvinfer2 unique-id 2

gstd-client element_set p0 nvinfer3 unique-id 3

gstd-client pipeline_play p0

Output:

RidgeRun Resources

Quick Start Client Engagement Process RidgeRun Blog Homepage Technical and Sales Support RidgeRun Online Store RidgeRun Videos Contact Us RidgeRun.ai: Artificial Intelligence | Generative AI | Machine Learning

Contact Us Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering information is available at RidgeRun Engineering Services, RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page.