NVIDIA Jetson Xavier - Capture and Display GStreamer pipelines
|
Capture
gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' \ ! fakesink silent=false -v
Performance:
| Operation Mode | CPU% | GPU% | FPS |
|---|---|---|---|
| 0 (max performance) | 0.57 | 4.48 | 30.02 |
| 1 (min power) | 0.0 | 23.27 | 30.03 |
| 2 (default) | 0.9 | 8.91 | 30.02 |
Capture and Display
Using nvarguscamerasrc (with ov5693 camera sensor) This sensor has 3 operation modes:
GST_ARGUS: 2592 x 1944 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000; GST_ARGUS: 2592 x 1458 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 34000, max 550385000; GST_ARGUS: 1280 x 720 FR = 120.000005 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 22000, max 358733000;
These modes can capture natively. Other modes are artificial and consume more resources. We are experiencing that the 120 fps mode is only capturing at 60 fps.
1920x1080@30 (non-native):
gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' \ ! nvoverlaysink
Performance:
| Operation Mode | CPU% | GPU% | FPS | latency(ms) |
|---|---|---|---|---|
| 0 (max performance) | 0.0 | 4.92 | 30.01 | 114 |
| 1 (min power) | 0.0 | 49.37 | 30.02 | 114 |
| 2 (default) | 1.07 | 10.91 | 30.02 | 115 |
2592x1944@30 (native):
gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)NV12, framerate=(fraction)30/1' \ ! nvoverlaysink
Performance:
| Operation Mode | CPU% | GPU% | FPS | latency(ms) |
|---|---|---|---|---|
| 0 (max performance) | 0.1 | 4.66 | 30.01 | 116 |
| 1 (min power) | 0.0 | 24.2 | 30.06 | 115 |
| 2 (default) | 0.5 | 12.0 | 30.02 | 117 |
1920x1080@30 using v4l2src:
Sensor ov5693 can capture BG10 Bayer, this can be opened using visualization tools like vooya. Lamentably, Gstreamer made changes to v4l2src on version 1.12 and format BG10 is not supported. Because of this, we tested performance with a webcam (Logitech, Inc. HD Pro Webcam C920) instead.
gst-launch-1.0 v4l2src device=/dev/video1 ! "image/jpeg,width=1920,height=1080" ! jpegdec ! queue ! nvoverlaysink
Performance:
| Operation Mode | CPU% | GPU% | FPS |
|---|---|---|---|
| 0 (max performance) | 0.0 | 3.75 | 29.77 |
| 1 (min power) | 0.0 | 93.66 | 20.27 |
| 2 (default) | 0.0 | 17.46 | 29.99 |
2592x1944@30 using xvimagesink:
gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)NV12, framerate=(fraction)30/1' \ ! nvvidconv ! queue ! xvimagesink
Performance:
| Operation Mode | CPU% | GPU% | FPS | latency(ms) |
|---|---|---|---|---|
| 0 (max performance) | 5.47 | 7.23 | 30.01 | 174 |
| 1 (min power) | 40.17 | 62.02 | 30.08 | 173 |
| 2 (default) | 49.8 | 41.93 | 30.03 | 172 |
Latency
We thought that 115 ms of glass to glass latency is a bit too much for simple capture and display pipeline. So, we made other measurements to find out where is this latency coming from. We used a camera sensor ov5693 for all the tests.
- The first measurement was the pipeline latency. We used GstShark to measure latency from the `nvarguscamerasrc` source pad to the `nvoverlaysink` source pad, and it was around 15 ms. The exact command used can be found below:
GST_TRACERS="interlatency" GST_DEBUG=GST_TRACER:7 gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)2592, height=(int)1944, format=(string)NV12, framerate=(fraction)30/1' ! nvoverlaysink
- We measured latency using the system clock instead of a timer to dismiss the timer precision and display lag. With this method, we measured 130ms latency instead of 115ms but we needed to add an nvvidconv that converts from NVMM memory to normal memory. We measured this conversion with GstShark and it took at most 20ms for a 2592x1944 buffer. So, even with a more precise result, the latency was over 100ms for a simple capture pipeline. The method used for this measurement is listed below:
- Patch the element `multifilesink` so that the files are saved with the system clock in milliseconds instead of the usual name plus a numeral
- Connect to the Xavier using the serial console:
- With the camera pointing at the serial console run:
- Open the resulting raw image files with raw pixels and compare the filename with the displayed image.
git clone https://github.com/GStreamer/gst-plugins-good.git cd gst-plugins-good/ git checkout 1.14.1
Apply this patch:
diff --git a/gst/multifile/gstmultifilesink.c b/gst/multifile/gstmultifilesink.c
index d2a55ef96..15f9b07ba 100644
--- a/gst/multifile/gstmultifilesink.c
+++ b/gst/multifile/gstmultifilesink.c
@@ -121,6 +121,8 @@
#include <gst/video/video.h>
#include <glib/gstdio.h>
#include "gstmultifilesink.h"
+#include <sys/time.h>
+#include <math.h>
static GstStaticPadTemplate sinktemplate = GST_STATIC_PAD_TEMPLATE ("sink",
GST_PAD_SINK,
@@ -614,6 +616,7 @@ gst_multi_file_sink_write_buffer (GstMultiFileSink * multifilesink,
gboolean ret;
GError *error = NULL;
gboolean first_file = TRUE;
+ struct timeval tod;
gst_buffer_map (buffer, &map, GST_MAP_READ);
@@ -621,8 +624,8 @@ gst_multi_file_sink_write_buffer (GstMultiFileSink * multifilesink,
case GST_MULTI_FILE_SINK_NEXT_BUFFER:
gst_multi_file_sink_ensure_max_files (multifilesink);
- filename = g_strdup_printf (multifilesink->filename,
- multifilesink->index);
+ gettimeofday(&tod, NULL);
+ filename = g_strdup_printf ("%lf", round((double)tod.tv_sec * 1.0e3 + ((double)tod.tv_usec * 1.0e-3)));
ret = g_file_set_contents (filename, (char *) map.data, map.size, &error);
if (!ret)
goto write_error;
./autogen.sh --disable-gtk-doc --disable-tests --disable-nls make -j8
sudo minicom -D /dev/ttyACM0 -8 -b 115200
{ for i in {1..1000}; do echo $(($(date +%s%N)/1000000)); done; }&
GST_PLUGIN_PATH=/home/nvidia/gst-plugins-good/gst/multifile/.libs/ gst-launch-1.0 nvarguscamerasrc num-buffers=10 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw' ! multifilesink
