View source for DeepStream 6.3 pipelines

<seo title="DeepStream pipelines | DeepStream | RidgeRun Developer" titlemode="replace" metakeywords="GStreamer, Linux SDK, Linux BSP,  Embedded Linux, Device Drivers, Nvidia, Xilinx, TI, NXP, Freescale, Embedded Linux driver development, Linux Software development, Embedded Linux SDK, Embedded Linux Application development, GStreamer Multimedia Framework, Jetson, Jetson AGX Xavier, Jeston Xavier NX, Xavier NX, Jetson Xavier, Xavier, Jetson TX1, Jetson TX2, AI, Deep Learning, NVIDIA Jetson Orin, Jetson Orin, Orin, NVIDIA Orin, NVIDIA Jetson AGX Orin, Jetson AGX Orin, DeepStream pipelines, DeepStream" metadescription="This wiki has DeepStream pipelines used on Jetson Xavier and Nano, describe some of the DeepStream features for the NVIDIA platforms and other multimedia features"></seo>

{{NVIDIA Pref Partner logo and RR Contact}}
<br>
<td>
{{GStreamer debug}}
</td>

<!----
<div style="float:right; clear:both; margin-left:0.5em;">__TOC__</div>
---->

==Introduction==
On this page, you are going to find a set of DeepStream pipelines used on Jetson Xavier NX DevKit. This wiki page tries to describe some of the DeepStream features for the NVIDIA platforms and other multimedia features. The goal is to provide you some example pipelines. A general NVIDIA Deepstream overview can be found at: [https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Overview.html NVIDIA Deepstream overview].

'''Various Deepstream elements found in this wiki:'''
* '''[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html nvinfer]''': The Gst-nvinfer element performs inference on input data using NVIDIA TensorRT. It accepts batched NV12/RGBA buffers, converts and scales the data according to network requirements, and passes it to the TensorRT engine for inference. This plugin can operate in primary, secondary, or preprocessed tensor input modes and supports various object detection and classification networks.
* '''[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html nvtracker]''': The Gst-nvtracker element allows the DeepStream pipeline to use a tracking library to track detected objects with unique IDs over time. It converts input buffers as needed and processes data in batch mode for efficient GPU execution. The plugin updates metadata with tracked object coordinates, tracker confidence, and object IDs.
* '''[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvvideotestsrc.html nvvideotestsrc]''' (Deepstream 6.4): The Gst-nvvideotestsrc element generates test video data using CUDA, creating output buffers on the GPU in various formats and patterns. It can introduce buffer latencies to simulate network conditions and also supports raw video input to generate GPU buffers. It is currently supported only on X86.
* '''[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvvideoconvert.html nvvideoconvert]''': The Gst-nvvideoconvert element performs video color format conversion. It accepts NVMM and RAW memory and provides NVMM or RAW memory at the output. It supports various input and output formats and allows for scaling and cropping of images.
* '''[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmsgbroker.html nvmsgbroker]''' : The Gst-nvmsgbroker element works like a messenger that sends data to a server using a specific protocol, such as Kafka or MQTT. It takes data that already has a special tag called NvDsPayload and sends it to the server. To work, it needs a special library that tells it how to use the chosen communication protocol.
* '''[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvvideo4linux2.html nvvideo4linux2]'''(decoders/encoders): Gst-nvvideo4linux2 is used to decode and encode videos using special hardware on Jetson and dGPU devices. It decodes compressed videos and encodes uncompressed videos.
* '''[https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsmetautils.html nvdsmetautils]''' (nvdsmetainsert/nvdsmetaextract) (Deepstream 6.4) Gst-nvdsmetautils contain nvdsmetainsert and nvdsmetaextract plugins. nvdsmetainsert adds metadata to video or audio buffers, allowing extra information to be attached to the data. nvdsmetaextract extracts this metadata from the buffers for use by other system components. Together, these elements enable the insertion and subsequent reading of additional information in multimedia streams, making it easier to handle metadata at different stages of data processing.
* Among others elements.

==Deepstream Reference Apps==
The DeepStream SDK includes several precompiled reference applications designed to help users understand and leverage the capabilities of DeepStream. Some of these applications can be found at: <code>/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/</code>. Now we will continue with the example of the <code>deepstream-test1</code> app:

* Change to the directory
 cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1
* Install dependencies
<pre>
sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgstrtspserver-1.0-dev libx11-dev libjson-glib-dev libyaml-cpp-dev
</pre>
* Export <code>CUDA_VER</code>
<pre>
export CUDA_VER=11.4
</pre>
* Compile
<pre>
sudo -E make
</pre>
* Run
<pre>
./deepstream-test1-app dstest1_config.yml
</pre>

Output:
[[File:output_deepstream-test1-dp6.png|thumbnail|center|840px]]

You can explore the other examples by following the instructions in their respective README files, in each application directory located at <code>/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps</code>.

==Pipelines - Jetson Xavier NX Developer Kit - [https://docs.nvidia.com/metropolis/deepstream/6.3/dev-guide/index.html Deepstream 6.3]==

Common deepstream pipeline stages:
# '''Capture'''
# '''Decode''' (nvv4l2decoder) This plugin's input streams are generally encoded camera streams (H264, H265, JPEG, MJPEG). The output is a stream of NV12 buffers batched together by Nvstreammux and provided as input to the inference engine, Nvinfer. This plugin is based on the GStreamer v4l2videodec plugin, but allows you to use hardware accelerated codecs on the Jetson devices.
# '''Image-Processing'''
# '''Batching''' (nvstreammux). The main objective of the nvstreammux plugin is to create a batch to feed into the pipeline’s inference engine, the nvinfer plugin. 
# '''Inference''' (nvinfer)
# '''Object Tracking''' (nvtracker)
# '''Overlay''' (nvdsosd)
# '''Output'''

===Test nvvideoconvert (DeepStream element) to crop===
As a sanity check to validate the functioning of deepstream, you can run this pipeline that uses nvvideoconvert (a deepstream element) and explore its capabilities. For example, in this case it can be used to crop the image.

<pre>
gst-launch-1.0 videotestsrc ! "video/x-raw, width=1280, height=720" ! nvvideoconvert src-crop="0:0:640:480" ! nv3dsink
</pre>

===Inference pipelines===
The following DeepStream pipelines processes a video stream by reading it from a file, decoding it, batching the frames, running inference to detect objects, tracking those objects, overlaying the results, and encoding the processed video into a new file. It captures the video, decodes it using hardware acceleration, batches the frames, detects objects, tracks them, overlays the results and sends it to the sink.

The model configured in this file is a network based on ResNet10 trained for object detection. The .caffemodel and .prototxt files define the architecture and weights of the trained network. The .engine file is a TensorRT engine optimized for fast inference on the GPU. You can find this out by reading the <code>config-file-path</code> used.

The SDK includes some pre-trained models that you can use with the <code>nvinfer</code> element. You can find the configuration files at this path:
 /opt/nvidia/deepstream/deepstream-6.3/samples/models

====Primary_Detector + Overlay====
<pre>
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/overlay.mp4
</pre>
You can play the video using:
<pre>
gst-launch-1.0 filesrc location=~/make.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! nv3dsink
</pre>

* '''Streaming''': 

For convenience, the following pipeline is provided, which is the same as the previous one but performs streaming.

 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.yml ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nv3dsink -e

* '''Using RtspSink''':
For convenience, the following pipeline is provided in case you want to check the compatibility of deepstream with [[GstRtspSink | rtspsink]].
 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.yml ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc insert-sps-pps=true idrinterval=30 ! video/x-h264, mapping=/stream1  ! perf ! rtspsink service=49152

On the client, export the server's IP address and the service port.
 export IP_ADD=192.168.0.10

 export PORT=49152

You can play the streaming:
 gst-launch-1.0 rtspsrc location=rtsp://$IP_ADD:$PORT/stream1 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

====Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker)====
In the past, DeepStream 5.1 included three low-level tracker libraries to configure the tracking mode in the <code>nvtracker</code> element:
* libnvds_mot_iou.so
* libnvds_mot_klt.so
* libnvds_nvdcf.so

All of these three low-level tracker libraries are deprecated in DeepStream 6.X, and a unified low-level tracker library is introduced, which is <code>libnvds_nvmultiobjecttracker.so</code>.
 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/tracker.mp4

====Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_CarColor====
You could use more than one <code>nvinfer</code> element for inference and thus combine different objectives:
  gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 unique-id=1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 unique-id=2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/color.mp4

====Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_CarMake====
 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 unique-id=1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 unique-id=2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_carmake.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/make.mp4

====Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_VehicleTypes====
 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 unique-id=1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 unique-id=2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/types.mp4

====Primary_Detector + Overlay + tracker (libnvds_nvmultiobjecttracker) + Secondary_VehicleTypes + Secondary_CarColor with GstD====

Alternatively, you can use three types of inference in a single pipeline. This time, the solution is presented using GstD so you can see its use with Deepstream (GstD is a more flexible and powerful version of gst-launch; you can read more about it here: [[GStreamer%20Daemon | gstd]]). As you can see in the resulting video, the pipeline identifies three things:
* It is a car.
* It is a truck.
* It is a red car.

 gstd-client pipeline_create p0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer name=nvinfer1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer2 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt" ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ! nvinfer name=nvinfer3 process-mode=secondary infer-on-gie-id=1 infer-on-class-ids="0:" batch-size=16 config-file-path="/opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt" ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=~/gsdt.mp4

 gstd-client element_set p0 nvinfer1 unique-id 1

 gstd-client element_set p0 nvinfer2 unique-id 2

 gstd-client element_set p0 nvinfer3 unique-id 3

 gstd-client pipeline_play p0

Output:
[[File:3_inferences_deepstream63.png|thumbnail|center|840px]]

{{ContactUs}}

[[Category:AI]] [[Category:DeepStream]][[Category:Jetson]]