NVIDIA Jetson Xavier - Capture and Display GStreamer pipelines

NVIDIA®Jetson Xavier™
Introduction
SoM Overview Carrier Boards NVIDIA Xavier Developer Kit Connect Tech Rogue Connect Tech Mimic Software Support NVIDIA JetPack
JetPack 5.X
Getting Started Installing JetPack Flashing the board with GUI installer JetPack Components Compiling Source Code Flashing the Board from Cmdline Performance Tuning Evaluating Performance Set Values Manually Tuning Power Maximizing Performance
Development in the Board
Getting into the Board Using Serial Console Using SSH Using Ubuntu Login Installing Packages
Video Capture and Display
Description Supported Cameras Imx219 Driver Software Support Using GStreamer Using Libargus
Processors
HD Audio and Video Subystem Video Encoder Video Decoder JPEG Encoder and Decoder Video Image Compositor Audio Processing Engine Deep Learning Accelerator Volta GPU Description Using CUDA Using OpenGL
Interfaces
I2C SD/eMMC Controller USB Controller
GStreamer Pipelines
Capture and Display H264 H265 VP9 MPEG4 Simultaneous Streams Compositor
Deep Learning
TensorRT Using TensorRT integrated with Tensorflow Parsing a Tensorflow model for TensorRT Parsing Caffe model for TensorRT Building TensorRT API examples Deep Learning Accelerator Software Support Deep Learning Tutorials Jetson Inference Jetson Reinforcement
RidgeRun Products
GstPTZR GstCUDA GstColorTransfer GstWebRTC GstRtspSink GstInterpipe GstInference GStreamer Daemon GstShark
Reference Documentation
Contact Us

Capture

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' \
! fakesink silent=false -v

Performance:

Operation Mode	CPU%	GPU%	FPS
0 (max performance)	0.57	4.48	30.02
1 (min power)	0.0	23.27	30.03
2 (default)	0.9	8.91	30.02

Capture and Display

Using nvarguscamerasrc (with ov5693 camera sensor) This sensor has 3 operation modes:

GST_ARGUS: 2592 x 1944 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000;
GST_ARGUS: 2592 x 1458 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000;
GST_ARGUS: 1280 x 720 FR = 120.000005 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 22000, max 358733000;

These modes can capture natively. Other modes are artificial and consume more resources. We are experiencing that the 120 fps mode is only capturing at 60 fps.

1920x1080@30 (non-native):

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' \
! nvoverlaysink

Performance:

Operation Mode	CPU%	GPU%	FPS	latency(ms)
0 (max performance)	0.0	4.92	30.01	114
1 (min power)	0.0	49.37	30.02	114
2 (default)	1.07	10.91	30.02	115

2592x1944@30 (native):

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)NV12, framerate=(fraction)30/1' \
! nvoverlaysink

Performance:

Operation Mode	CPU%	GPU%	FPS	latency(ms)
0 (max performance)	0.1	4.66	30.01	116
1 (min power)	0.0	24.2	30.06	115
2 (default)	0.5	12.0	30.02	117

1920x1080@30 using v4l2src:

Sensor ov5693 can capture BG10 Bayer, this can be opened using visualization tools like vooya. Lamentably, Gstreamer made changes to v4l2src on version 1.12 and format BG10 is not supported. Because of this, we tested performance with a webcam (Logitech, Inc. HD Pro Webcam C920) instead.

gst-launch-1.0 v4l2src device=/dev/video1 ! "image/jpeg,width=1920,height=1080" ! jpegdec ! queue ! nvoverlaysink

Performance:

Operation Mode	CPU%	GPU%	FPS
0 (max performance)	0.0	3.75	29.77
1 (min power)	0.0	93.66	20.27
2 (default)	0.0	17.46	29.99

2592x1944@30 using xvimagesink:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)NV12, framerate=(fraction)30/1' \
! nvvidconv ! queue ! xvimagesink

Performance:

Operation Mode	CPU%	GPU%	FPS	latency(ms)
0 (max performance)	5.47	7.23	30.01	174
1 (min power)	40.17	62.02	30.08	173
2 (default)	49.8	41.93	30.03	172

Latency

We thought that 115 ms of glass to glass latency is a bit too much for simple capture and display pipeline. So, we made other measurements to find out where is this latency coming from. We used a camera sensor ov5693 for all the tests.

• The first measurement was the pipeline latency. We used GstShark to measure latency from the `nvarguscamerasrc` source pad to the `nvoverlaysink` source pad, and it was around 15 ms. The exact command used can be found below:

GST_TRACERS="interlatency" GST_DEBUG=GST_TRACER:7 gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)NV12, framerate=(fraction)30/1' ! nvoverlaysink

• We measured latency using the system clock instead of a timer to dismiss the timer precision and display lag. With this method, we measured 130ms latency instead of 115ms but we needed to add an nvvidconv that converts from NVMM memory to normal memory. We measured this conversion with GstShark and it took at most 20ms for a 2592x1944 buffer. So, even with a more precise result, the latency was over 100ms for a simple capture pipeline. The method used for this measurement is listed below:

1. Patch the element `multifilesink` so that the files are saved with the system clock in milliseconds instead of the usual name plus a numeral

git clone https://github.com/GStreamer/gst-plugins-good.git
cd gst-plugins-good/
git checkout 1.14.1

Apply this patch:

diff --git a/gst/multifile/gstmultifilesink.c b/gst/multifile/gstmultifilesink.c
index d2a55ef96..15f9b07ba 100644
--- a/gst/multifile/gstmultifilesink.c
+++ b/gst/multifile/gstmultifilesink.c
@@ -121,6 +121,8 @@
 #include <gst/video/video.h>
 #include <glib/gstdio.h>
 #include "gstmultifilesink.h"
+#include <sys/time.h>
+#include <math.h>
 
 static GstStaticPadTemplate sinktemplate = GST_STATIC_PAD_TEMPLATE ("sink",
     GST_PAD_SINK,
@@ -614,6 +616,7 @@ gst_multi_file_sink_write_buffer (GstMultiFileSink * multifilesink,
   gboolean ret;
   GError *error = NULL;
   gboolean first_file = TRUE;
+  struct timeval tod;
 
   gst_buffer_map (buffer, &map, GST_MAP_READ);
 
@@ -621,8 +624,8 @@ gst_multi_file_sink_write_buffer (GstMultiFileSink * multifilesink,
     case GST_MULTI_FILE_SINK_NEXT_BUFFER:
       gst_multi_file_sink_ensure_max_files (multifilesink);
 
-      filename = g_strdup_printf (multifilesink->filename,
-          multifilesink->index);
+      gettimeofday(&tod, NULL);
+      filename = g_strdup_printf ("%lf", round((double)tod.tv_sec * 1.0e3 + ((double)tod.tv_usec * 1.0e-3)));
       ret = g_file_set_contents (filename, (char *) map.data, map.size, &error);
       if (!ret)
         goto write_error;

./autogen.sh --disable-gtk-doc --disable-tests --disable-nls 
make -j8

2. Connect to the Xavier using the serial console:

sudo minicom -D /dev/ttyACM0 -8 -b 115200

3. With the camera pointing at the serial console run:

{ for i in {1..1000}; do   echo $(($(date +%s%N)/1000000)); done; }&
GST_PLUGIN_PATH=/home/nvidia/gst-plugins-good/gst/multifile/.libs/ gst-launch-1.0 nvarguscamerasrc num-buffers=10 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw' ! multifilesink

4. Open the resulting raw image files with raw pixels and compare the filename with the displayed image.