GStreamer pipelines for Jetson TX2: Difference between revisions

Recall that this board is a quad core A57 with some hardware units dedicated to encode/decode. Plugins accelerated by Hardware available:

gst-inspect-1.0 | grep omx

omx:  nvoverlaysink: OpenMax Video Sink

omx:  omxh264dec: OpenMAX H.264 Video Decoder

omx:  omxmpeg4videodec: OpenMAX MPEG4 Video Decoder

Nvidia plugins:

gst-inspect-1.0 | grep nv

omx:  nvoverlaysink: OpenMax Video Sink

nvvidconv:  nvvidconv: NvVidConv Plugin

nvivafilter:  nvivafilter: NvIVAFilter Plugin

== Measure ARM load ==

In order to test some of the videos or incoming streams produced by the pipelines below you might need VLC installed in your Jetson module, there is a known issue with VLC version 2.2 and you need to download and install VLC 2.4 or newer, in order to do it please follow these steps:

sudo apt-get build-dep vlc && sudo apt-get install libtool build-essential

wget http://download.videolan.org/pub/videolan/vlc/2.2.6/vlc-2.2.6.tar.xz   

make

#you can also try: make install  

It takes around 10min to build. Once you have built the new vlc, you can run this to open a plain UDP stream for instance

This plugin was created by nvidia and it has access to the ISP that helps converting from bayer to yuv suitable for the video encoders. However, it seems that its usage is limited to the ov5693 sensor until NVIDIA releases its source code or until it adds support to v4l2 to use the ISP.

gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! fakesink silent=false -v

When the pipeline is launched a daemon appears on top nvcamera-daemon, one question about the daemon was posted on the [https://devtalk.nvidia.com/default/topic/926578/jetson-tx1/nvcamera-daemon/ Nvidia Embedded Computing Forum]

ARM consumption: 45.7% of one core (4 cores available 400% - this means something like 11.25% of total)

RAM 1119/7853MB (lfb 1258x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267]

RAM 1119/7853MB (lfb 1258x4MB) cpu [28%@345,off,off,7%@345,3%@345,16%@345]

RAM 1119/7853MB (lfb 1257x4MB) cpu [14%@345,off,off,10%@345,13%@345,17%@345]

RAM 1119/7853MB (lfb 1257x4MB) cpu [10%@345,off,off,12%@345,14%@345,13%@345]

  1574 root      20  0  9.917g 116260  37488 S  45.7  1.4  1:24.81 nvcamera-daemon                                                                                                                                                     

11085 nvidia    20  0  331488  9492  6004 S  3.3  0.1  0:06.98 gst-launch-1.0     

The nvvidconv is required normally to convert the memory type, however, on this case it is used to flip the image. Removing the flip from the pipeline saves like 5% of ARM load.

nvcamerasrc exposes several properties to control the ISP

  wbmode              : White balance affects the color temperature of the photo

                         flags: readable, writable

                         Integer. Range: 0 - 360 Default: -1  

=== v4l2src ===

Starting on L4T R23.2 there is a /dev/video0 node to capture, however, this node will give you frames in bayer which are NOT suitable to encode because it grabs frames directly from the [https://www.ovt.com/sensors/OV5693 ov5693 camera] without using the ISP.

https://devtalk.nvidia.com/default/topic/923984/ov5693-through-v4l2-interface-not-working-in-23-2/

It works also with the UVC driver.

=== nvcamerasrc ===

gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e

ARM consumption: 46.1% of one core (4 cores available 400% - this means something like 14% of total). However, only the capture is 11.25% so the display consumes '''2.75%'''. For other camera you can find in our wiki latency numbers for [https://developer.ridgerun.com/wiki/index.php?title=Jetson_glass_to_glass_latency#Introduction  glass to glass measurements]

RAM 1184/7853MB (lfb 1246x4MB) cpu [2%@345,off,off,0%@345,0%@345,0%@345]

RAM 1194/7853MB (lfb 1246x4MB) cpu [0%@345,off,off,0%@345,2%@345,0%@345]

RAM 1194/7853MB (lfb 1246x4MB) cpu [0%@345,off,off,0%@345,1%@345,0%@345]

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+ COMMAND                                         

  1574 root      20  0  9.979g 208932  44656 S  46.2  2.6  5:18.85 nvcamera-daemon                                                                                                                                                     

11507 nvidia    20  0  423932  11508  8796 S  3.6  0.1  0:00.66 gst-launch-1.0     

== H264 encoding ==

ARM load: 157% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.

gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=test.h264 -e

VLC can play the file correctly at 30fps and reports as codec: H264-MPEG-4 AVC (part 10) (h264). The fan on top of the SoC didn't turn on.

=== nvcamerasrc ===

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxh264enc bitrate=8000000 \

! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=test.h264 -e

Codecvisa reports bitrate on 7.96Mbps when variable mode is selected on the encoder.

ARM consumption 45.4% of one core (4 cores available 400% - this means something like 16.75% of total). However, only the capture is 11.25% so the h264 consumes '''5.5%'''  

RAM 1279/7853MB (lfb 1237x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267]

RAM 1280/7853MB (lfb 1237x4MB) cpu [8%@345,off,off,19%@345,10%@345,12%@345]

RAM 1234/7853MB (lfb 1235x4MB) cpu [0%@345,off,off,1%@345,0%@345,0%@345]

RAM 1234/7853MB (lfb 1235x4MB) cpu [0%@345,off,off,3%@345,0%@345,0%@345]

  1574 root      20  0  9.979g 214836  44684 S  45.4  2.7  7:10.27 nvcamera-daemon                                                           

11614 nvidia    20  0  700192  12800  9236 S  8.6  0.2  0:00.88 gst-launch-1.0           

The encoder exposes the following parameters

   control-rate        : Bitrate control method

                         flags: readable, writable, changeable only in NULL or READY state

== Dual H264 encoding ==

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \

streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testA.h264 \

streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testB.h264

ARM consumption ~44.1% of one core (4 cores available 400% - this means something like 16.5% of total). However, only the capture is 11.25% so the dual h264 consumes '''5.25%'''

RAM 1320/7853MB (lfb 1221x4MB) cpu [0%@1420,off,off,0%@1420,0%@1420,0%@1420]

RAM 1320/7853MB (lfb 1221x4MB) cpu [19%@345,off,off,16%@345,10%@345,9%@345]

RAM 1275/7853MB (lfb 1216x4MB) cpu [0%@345,off,off,2%@345,0%@345,0%@345]

RAM 1275/7853MB (lfb 1216x4MB) cpu [0%@345,off,off,1%@345,0%@345,0%@345]

== H264 UDP Streaming ==

server:

CLIENT_IP=<IP_ADDRESS>

gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \

omxh264enc control-rate=2 bitrate=4000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! \

h264parse ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

PC client:

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264 ! xvimagesink sync=false async=false -e

   

Using VLC  

*Server:  

CLIENT_IP=<IP_ADDRESS>

gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \

omxh264enc control-rate=2 bitrate=4000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! \

h264parse ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

*Client:

vlc -vv --demux h264 --h264-fps 60  udp://@:5000  

=== h264 parse missing===

In case gstreamer complains when running the pipeline for missing the h264parse plugin, run the following command to install it:  

sudo apt-get install gstreamer1.0-plugins-bad

=== Latency ===

ARM load: 156% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.

gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxvp8enc ! qtmux ! filesink location=test.mp4 -e

VLC can play the file correctly at 30fps and reports as codec: Google/On2's VP8 Video (VP80). The fan on top of the SoC didn't turn on.

=== nvcamerasrc ===

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxvp8enc bitrate=8000000 ! qtmux ! filesink location=test.mp4 -e

VLC reports bitrate on ~8Mbps.

ARM consumption ~50% of one core (4 cores available 400% - this means something like 15.5% of total). However, only the capture is 11.25% so the VP8 consumes '''4.25%'''

RAM 1442/7853MB (lfb 1134x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267]

RAM 1442/7853MB (lfb 1134x4MB) cpu [16%@345,off,off,12%@345,15%@345,14%@345]

RAM 1443/7853MB (lfb 1134x4MB) cpu [14%@345,off,off,20%@345,8%@345,12%@345]

RAM 1443/7853MB (lfb 1134x4MB) cpu [15%@345,off,off,21%@345,6%@345,10%@345]

  1574 root      20  0  9.979g 264008  44716 S  45.7  3.3  14:43.58 nvcamera-daemon                                                                                                                                                     

13065 nvidia    20  0  719944  14140  9996 S  9.2  0.2  0:09.34 gst-launch-1.0           

The encoder exposes the following parameters

bitrate            : Target bitrate

                         flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state

                           (3): SlowPreset      - SlowPreset

== H265 encoding ==

In order to support h265 you need VLC 2.2 in your computer to play the videos

sudo add-apt-repository ppa:mc3man/trusty-media

sudo apt-get update

sudo apt-get install vlc vlc-plugin-*

=== videotestsrc ===

ARM load: 156% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.

gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh265enc ! filesink location=test.h265 -e

</pre>

=== nvcamerasrc ===

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxh265enc ! filesink location=test.h265 -e

Codecvisa reports bitrate on 7.96Mbps when variable mode is selected on the encoder.

ARM consumption ~55% of one core (4 cores available 400% - this means something like 13% of total). However, only the capture is 11.25% so the h264 consumes '''1.75%'''  

RAM 1487/7853MB (lfb 993x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267]

RAM 1487/7853MB (lfb 993x4MB) cpu [12%@345,off,off,12%@345,10%@345,14%@345]

RAM 1488/7853MB (lfb 991x4MB) cpu [8%@345,off,off,13%@345,20%@345,10%@345]

RAM 1488/7853MB (lfb 991x4MB) cpu [11%@345,off,off,12%@345,11%@345,21%@345]

  1574 root      20  0  9.979g 266012  44612 S  47.7  3.3  17:54.85 nvcamera-daemon                                                           

13273 nvidia    20  0  694396  12920  9156 S  8.6  0.2  0:01.32 gst-launch-1.0                         

The encoder exposes the following parameters

   control-rate        : Bitrate control method

                         flags: readable, writable, changeable only in NULL or READY state

                         Unsigned Long. Range: 0 - 18446744073709551615 Default: 0  

== Dual H265 encoding ==

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \

streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testA.h264 \

streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testB.h264

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \

nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \

streams. ! omxh265enc bitrate=8000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! filesink location=testA.h265 \

streams. ! omxh265enc bitrate=8000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! filesink location=testB.h265

ARM consumption ~48.7% of one core (4 cores available 400% - this means something like 13.5% of total). However, only the capture is 11.25% so the dual h265 consumes '''2.25%'''

RAM 1503/7853MB (lfb 965x4MB) cpu [0%@1420,off,off,0%@1420,0%@1420,0%@1420]

RAM 1503/7853MB (lfb 965x4MB) cpu [19%@345,off,off,10%@345,13%@345,16%@345]

RAM 1457/7853MB (lfb 957x4MB) cpu [14%@345,off,off,9%@345,10%@345,15%@345]

RAM 1456/7853MB (lfb 957x4MB) cpu [0%@345,off,off,1%@345,0%@345,0%@345]

== H265 UDP Streaming ==

Currently it is not possible to receive the stream using the PC because there is not a h265 decoder available on gstreamer. In order to evaluate the performance let's receive the stream on the Jetson module as well.

export PKG_CONFIG_PATH=/home/$USER/gst_$VERSION/out/lib/pkgconfig

VERSION=1.8.0

export LD_LIBRARY_PATH=/home/$USER/gst_$VERSION/out/lib/

server:

CLIENT_IP=<IP_ADDRESS>

gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \

omxh265enc low-latency=1 control-rate=2 bitrate=4000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! \

h265parse ! rtph265pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

client:

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H265,payload=96 ! rtph265depay \

! h265parse ! queue ! omxh265dec ! nvoverlaysink sync=false async=false -e

=== Latency ===

This is an application based on gstreamer and omx to capture, encode and save video to the filesystem. It has a lot of options. The source code is not released by NVIDIA yet but they are planning to release in future versions of Linux 4 Tegra (L4T), you can read about it at [https://devtalk.nvidia.com/default/topic/898129/jetson-tx1/enabling-camera-on-jetson-tx1-board/ Nvidia Embedded Computing Forum]

nvidia@tegra-ubuntu:~$ nvgstcapture-1.0 --help

Encoder null, cannot set bitrate!

       Enter 'f'  

== Capture snapshots ==

Sometimes when the board has been just flashed you need to run reboot in order to run nvgstcapture-1.0 successfully. To take snapshots:

nvgstcapture-1.0

#then press j to take an snapshot

q  

Resolutions available to create snapshot:

           (2) : 640x480

           (3) : 1280x720

           (12): 5632x4224

  nvgstcapture-1.0 -m 1 --image-res=10 #set 1 to snapshot mode

In general the application is consuming 15% of each of four cores. It is always displaying video this could affect.

== Capture and record video ==

nvgstcapture-1.0 -m 2 # set the mode in video

1 # start video recording

2 # take a snapshot

0 # stop video recording

In case of video it should support:

           (2) : 640x480

           (3) : 1280x720

           (8) : 3840x2160

           (9) : 3896x2192

However, it is not possible to set other value than 2,3,4,5 up to 9.

nvgstcapture-1.0 -m 2 --video-res=5

'''top'''

top - 22:15:09 up  2:30,  4 users,  load average: 0.47, 0.45, 0.30

27513 nvidia    20  0  815040  13932  10272 S  4.3  0.2  0:01.48 nvgstcapture-1.

There is a daemon and a pipeline. Interesting...