View source for GStreamer AI Inference -RTSP - WebRTC demo

{{#seo:
|title=GStreamer AI inference RTSP WebRTC demo
|title_mode=replace
|description={{{description|This wiki has the details about setting up the RidgeRun demo on GStreamer AI inference with RTSP, WebRTC and other RidgeRun GStreamer plugins}}}
}}
{{NVIDIA Pref Partner logo & RR Contact & GstDebug}}

==Introduction to RidgeRun GStreamer AI inference demo==

This demo demonstrates the capabilities of several of Ridgerun's GStreamer products while leveraging the NVIDIA Jetson TX2 hardware components for speedups in the video encoding and decoding.

The system consists of 4 different pipelines:
* A camera connected to an interpipesink.
* A second video source connected through a rstpsink-src to an interpipesink.
* An interpipesrc feeding into an AI object detector.
* An interpipesrc connected to a jpeg encoder.

The different interpipesrc and sinks connections are changed dynamically using Gstd, which allows passing either of the video sources through the GstInference element.
The system also takes snapshots whenever a person is detected on the current video stream.

<br>
[[File:Inference_demo_diagram-portrait.png|thumb|center|2000px|'''Demo system diagram'''.]]
<br>

==Getting the Code==
This demo is an open source project, the project is hosted at Github:
<pre>
https://github.com/RidgeRun/inference-demo
</pre>

To clone the latest version you may run:
<pre>
git clone https://github.com/RidgeRun/inference-demo
</pre>

==Installation==

===Hardware and system requirements===
* Jetson TX2 EVM
* Camera included in Jetson TX2 EVM
* Jetpack 4.2
* Google Chrome version up to 77.0 (newer version might work but haven't been tested)
* Internet connection

This guide considers you already installed Jetpack 4.2 including all the AI and Computer Vision software>
* CUDA 10
* OpenCV 3.3.1
You can use [[Installing_JetPack_4.2_-_Nvidia_SDK_Manager|this page]] as a reference to install a clean Jetpack 4.2 with all the needed dependencies.

====Required GStreamer based Products====

*'''TensorFlow'''

Check our TensorFlow building and installation guide by accessing this [[R2Inference/Supported_backends/TensorFlow|this page]].

*'''R2Inference'''

Check our R2Inference building and installation guide by accessing this [[R2Inference/Getting_started/Building_the_library|this page]].

*'''GstInference'''
We added a new element to GstInference for this demo that takes the detection and classification metadata and generates a signal if it matches the configured label-index. To use this element you need to install a specific branch: '''feature/inference-alert-element'''
Follow these instructions to build GstInference with the '''inferencealert''' element:

<syntaxhighlight lang="bash">

sudo apt-get install -y autoconf automake pkg-config libtool gtk-doc-tools libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev  gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-libav libopencv-dev

git clone https://github.com/RidgeRun/gst-inference.git

cd gst-inference

git checkout feature/inference-alert-element

./autogen.sh --prefix /usr/ --libdir /usr/lib/aarch64-linux-gnu/

make

sudo make install

cd ..

</syntaxhighlight>

*'''GstInterpipe'''

[[GstInterpipe_-_Building_and_Installation_Guide|GstInterpipe Build and Installation Guide]].

*'''GstD'''

[[GStreamer_Daemon_-_Building_GStreamer_Daemon#Development_Version|GstD Build and Installation Guide]].

It's important to make sure you are installing '''develop''' branch, using the Development Version instructions

*'''GstWebRTC'''

[[GstWebRTC_-_Evaluating_GstWebRTC|GstWebRTC evaluation binary installation]]. If you don't have an evaluation binary, please [https://www.ridgerun.com/contact contact us] with providing following info : * '''Platform''' (i.e.: iMX6, TX2, Ultrascale+, etc...), * '''gst-launch-1.0 --gst-version''' and * '''uname -a'''.

*'''GstRtspSink'''

[[GstRtspSink_-_Evaluating_GstRtspSink|GstRtspSink evaluation binary installation]]. If you don't have an evaluation binary, please [https://www.ridgerun.com/contact contact us] with providing following info : * '''Platform''' (i.e.: iMX6, TX2, Ultrascale+, etc...), * '''gst-launch-1.0 --gst-version''' and * '''uname -a'''.

====Check dependencies====

* You can check the installed GStreamer elements by running the following commands. Each of them should output all the element information properly:

<syntaxhighlight lang="bash">
gst-inspect-1.0 inference
gst-inspect-1.0 rrwebrtcbin
gst-inspect-1.0 interpipesink
gst-inspect-1.0 interpipesrc
gst-inspect-1.0 rtspsink
gst-inspect-1.0 detectionoverlay
gst-inspect-1.0 classificationoverlay
gst-inspect-1.0 inferencealert
</syntaxhighlight>

* Check the GstD installation:

<syntaxhighlight lang="bash">
gstd-client --version

#Expected Output
gstd 0.7.0
Copyright (c) 2015 RidgeRun Engineering
</syntaxhighlight>

====Required External Python Libraries====

*'''pip3'''
Install pip3 as follow:
<syntaxhighlight lang="bash">
sudo apt install python3-pip
</syntaxhighlight>

*'''psutil:'''
This is a GstD dependency. You can install it with the following command:

<syntaxhighlight lang="bash">
sudo -H pip3 install psutil
</syntaxhighlight>

==Usage==

1. Enter the TX2, you can do it natively or via SSH.

Enter the gst-inference-demo folder. The folder has the following structure.
<pre>
.
├── README.md
└── src
    ├── graph_tinyyolov2_tensorflow.pb
    ├── gst
    │   ├── gstc.py
    │   ├── pygstd.py
    │   └── pygst.py
    ├── main.py
    ├── pipe_config.json
    └── tinyyolov2_labels.txt
</pre>
<br>
2. Open a terminal on another PC/board and run the following pipeline:
<syntaxhighlight lang="bash">
PORT=5002
gst-launch-1.0 v4l2src device=/dev/video0 ! autovideoconvert ! vp8enc ! queue ! rtspsink service=$PORT
</syntaxhighlight>

Get that PCs ip address by running ifconfig, and coping the found value to the <code>rtsp_ip_address</code> in the <code>pipe_config.json</code> file.

The port can be defined to be any value that isn't already reserved by the system (usual ports larger than 1024 should be safe), with the exception of the <code>5001</code> port which is used by the demo for internal communication. This value should match the <code>rtsp_port</code> value in <code>pipe_config.json</code> file.

<br>
3. Open this link: https://webrtc.ridgerun.com:8443 in a Chrome tab. This can be done outside of the TX2.
* Set your desired '''Session id''', this can be any string but make sure to also change the session_id in the <code>pipe_config.json</code> file to match.
* Select video and press join.
<br>
4. Go back to the original terminal (located in the demo folder) and start the demo:

<syntaxhighlight lang="bash">
cd src
python3 main.py
</syntaxhighlight>

After a few seconds ou should get an output similar to the following:

<pre>
Starting GstInference Application...
Process 7154 died: No such process; trying to remove PID file. (/usr/local/var/run/gstd//gstd.pid)

    ** Menu **
 1) Camera source
 2) RTSP source
 3) Take snapshot
 4) Exit

</pre>

By pressing a number and Enter the system will change to the desired mode, where:

1 Camera source: EVM camera.

2 RTSP source: data obtained from the pipeline ran in step 2.

3 Take snapshot: the system will take a jpeg snapshot and save it to <code/tmp/</code> under the <code>output#.jpeg</code> naming scheme.

4 Exit: will exit the demo.

==Contact Us==
{{ContactUs}}

[[Category:Jetson]][[Category:GstWebRTC‏‎]][[Category:JetsonNano]][[Category:JetsonTX2]][[Category:NVIDIA Xavier]]