GStreamer pipelines for Jetson TX2
|
Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help. |
Introduction to NVIDIA®Jetson™ TX2 GStreamer pipelines
On this page, you are going to find a set of pipelines used on Jetson TX2, specifically used with the Jetson board. The wiki page tries to describe some of the multimedia features of the platform like the NVIDIA model to handle the ISP through its custom (and close) plugin called nvcamerasrc. It also includes a brief description of the sample application called nvgstcapture-1.0 and ways to measure the ARM load. The goal is to provide you with some example pipelines and introduction to the board. It is a work in progress.
The camera sensor used for tests is the OmniVision OV5693. These steps were run on Jetson TX2.
Jetson TX2 GStreamer pipelines
Recall that this board is a quad-core A57 with some hardware units dedicated to encode/decode. Plugins accelerated by Hardware available:
gst-inspect-1.0 | grep omx omx: nvoverlaysink: OpenMax Video Sink omx: nvhdmioverlaysink: OpenMax HDMI Video Sink omx: omxvp9enc: OpenMAX VP9 Video Encoder omx: omxvp8enc: OpenMAX VP8 Video Encoder omx: omxh265enc: OpenMAX H.265 Video Encoder omx: omxh264enc: OpenMAX H.264 Video Encoder omx: omxwmvdec: OpenMAX WMV Video Decoder omx: omxmpeg2videodec: OpenMAX MPEG2 Video Decoder omx: omxvp9dec: OpenMAX VP9 Video Decoder omx: omxvp8dec: OpenMAX VP8 Video Decoder omx: omxh265dec: OpenMAX H.265 Video Decoder omx: omxh264dec: OpenMAX H.264 Video Decoder omx: omxmpeg4videodec: OpenMAX MPEG4 Video Decoder
NVIDIA plugins:
gst-inspect-1.0 | grep nv omx: nvoverlaysink: OpenMax Video Sink omx: nvhdmioverlaysink: OpenMax HDMI Video Sink eglglessink: nveglglessink: EGL/GLES vout Sink nvjpeg: nvjpegdec: JPEG image decoder nvjpeg: nvjpegenc: JPEG image encoder nvegltransform: nvegltransform: NvEGLTransform nvcamera: nvtee: NvTee nvcamera: nvcamerasrc: Nvidia Camera Source nvvideosink: nvvideosink: nVidia Video Sink nveglstreamsrc: nveglstreamsrc: nVidia EGL Stream nvvidconv: nvvidconv: NvVidConv Plugin nvivafilter: nvivafilter: NvIVAFilter Plugin
Measure ARM load
There is a tool to measure the ARM load created by NVIDIA for the Jetson TX2. This tool seems to give better statistics of the ARM cores than top. In the remaining sections of this wiki the ARM load reported by top is included for reference to the reader but tegrastats should contain the real load.
Run it with sudo to get more information.
ubuntu@tegra-ubuntu:~$ ./tegrastats RAM 1787/7853MB (lfb 173x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267] RAM 1787/7853MB (lfb 173x4MB) cpu [0%@345,off,off,0%@345,1%@345,1%@345] RAM 1787/7853MB (lfb 173x4MB) cpu [0%@499,off,off,0%@499,1%@499,2%@499] RAM 1787/7853MB (lfb 173x4MB) cpu [0%@345,off,off,4%@345,1%@345,1%@345] EMC – memory controller AVP – audio/video processor VDE – video decoder engine GR3D – GPU
Jetson TX2 ARM load
This table tries to summarize the ARM load of different user cases of the Jetson TX2 using gstreamer and the capture plugin. The second column has the ARM load of the pipeline measured in one instant, using that value and the known value for the capture process the third column estimates the load of the encoding or display process (subtract the capture load). The load represent a global load of the system (4 ARMs) just to give an idea to the reader of the system capabilities.
Feature | Pipeline load (%) | Feature load (%) |
---|---|---|
Capture | 11.25 | 11.25 |
Capture-Display | 14 | 2.75 |
H264 Encoding | 16.75 | 5.5 |
VP8 Encoding | 15.5 | 4.25 |
H265 Encoding | 13 | 1.75 |
VLC on Jetson TX1 and Jetson TX2
In order to test some of the videos or incoming streams produced by the pipelines below you might need VLC installed in your Jetson module, there is a known issue with VLC version 2.2 and you need to download and install VLC 2.4 or newer, in order to do it please follow these steps:
sudo apt-get build-dep vlc && sudo apt-get install libtool build-essential wget http://download.videolan.org/pub/videolan/vlc/2.2.6/vlc-2.2.6.tar.xz tar -xf vlc-2.2.6.tar.xz cd vlc-2.2.6 ./configure --disable-qt make #you can also try: make install
It takes around 10min to build. Once you have built the new vlc, you can run this to open a plain UDP stream for instance
./vlc -vv --demux h264 --h264-fps 60 udp://@:5000
this issue has been reported on NVIDIA's forum
Capture
In order to capture there are two ways: Using v4l2 or using nvcamerasrc.
nvcamerasrc
This plugin was created by NVIDIA and it has access to the ISP that helps converting from bayer to yuv suitable for the video encoders. However, it seems that its usage is limited to the ov5693 sensor until NVIDIA releases its source code or until it adds support to v4l2 to use the ISP.
gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! fakesink silent=false -v
When the pipeline is launched a daemon appears on top nvcamera-daemon, one question about the daemon was posted on the NVIDIA Embedded Computing Forum
ARM consumption: 45.7% of one core (4 cores available 400% - this means something like 11.25% of total)
RAM 1119/7853MB (lfb 1258x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267] RAM 1119/7853MB (lfb 1258x4MB) cpu [28%@345,off,off,7%@345,3%@345,16%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [23%@345,off,off,12%@345,6%@345,12%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [21%@345,off,off,11%@345,9%@345,13%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [23%@345,off,off,7%@345,6%@345,10%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [16%@345,off,off,16%@345,9%@345,7%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [23%@345,off,off,9%@345,7%@345,7%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [16%@345,off,off,14%@345,6%@345,14%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [13%@345,off,off,15%@345,10%@345,14%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [18%@345,off,off,17%@345,4%@345,13%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [13%@345,off,off,10%@345,7%@345,15%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [16%@345,off,off,13%@345,8%@345,11%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [14%@345,off,off,15%@345,12%@345,14%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [18%@345,off,off,13%@345,14%@345,15%@345] RAM 1119/7853MB (lfb 1258x4MB) cpu [19%@345,off,off,11%@345,19%@345,11%@345] RAM 1119/7853MB (lfb 1257x4MB) cpu [14%@345,off,off,18%@345,13%@345,13%@345] RAM 1119/7853MB (lfb 1257x4MB) cpu [14%@345,off,off,10%@345,13%@345,17%@345] RAM 1119/7853MB (lfb 1257x4MB) cpu [10%@345,off,off,12%@345,14%@345,13%@345]
1574 root 20 0 9.917g 116260 37488 S 45.7 1.4 1:24.81 nvcamera-daemon 11085 nvidia 20 0 331488 9492 6004 S 3.3 0.1 0:06.98 gst-launch-1.0
The nvvidconv is required normally to convert the memory type, however, on this case it is used to flip the image. Removing the flip from the pipeline saves like 5% of ARM load.
nvcamerasrc exposes several properties to control the ISP
wbmode : White balance affects the color temperature of the photo flags: readable, writable Enum "GstNvCamWBMode" Default: 1, "auto" (0): off - GST_NVCAM_WB_MODE_OFF (1): auto - GST_NVCAM_WB_MODE_AUTO (2): incandescent - GST_NVCAM_WB_MODE_INCANDESCENT (3): fluorescent - GST_NVCAM_WB_MODE_FLUORESCENT (4): warm-fluorescent - GST_NVCAM_WB_MODE_WARM_FLUORESCENT (5): daylight - GST_NVCAM_WB_MODE_DAYLIGHT (6): cloudy-daylight - GST_NVCAM_WB_MODE_CLOUDY_DAYLIGHT (7): twilight - GST_NVCAM_WB_MODE_TWILIGHT (8): shade - GST_NVCAM_WB_MODE_SHADE (9): manual - GST_NVCAM_WB_MODE_MANUAL scene-mode : property to set scene mode flags: readable, writable Enum "GstNvCamSceneMode" Default: 0, "face-priority" (0): face-priority - GST_NVCAM_SCENE_MODE_FACE_PRIORITY (1): action - GST_NVCAM_SCENE_MODE_ACTION (2): portrait - GST_NVCAM_SCENE_MODE_PORTRAIT (3): landscape - GST_NVCAM_SCENE_MODE_LANDSCAPE (4): night - GST_NVCAM_SCENE_MODE_NIGHT (5): night-portrait - GST_NVCAM_SCENE_MODE_NIGHT_PORTRAIT (6): theatre - GST_NVCAM_SCENE_MODE_THEATRE (7): beach - GST_NVCAM_SCENE_MODE_BEACH (8): snow - GST_NVCAM_SCENE_MODE_SNOW (9): sunset - GST_NVCAM_SCENE_MODE_SUNSET (10): steady-photo - GST_NVCAM_SCENE_MODE_STEADYSHOT (11): fireworks - GST_NVCAM_SCENE_MODE_FIREWORKS (12): sports - GST_NVCAM_SCENE_MODE_SPORTS (13): party - GST_NVCAM_SCENE_MODE_PARTY (14): candle-light - GST_NVCAM_SCENE_MODE_CANDLE_LIGHT (15): barcode - GST_NVCAM_SCENE_MODE_BARCODE color-effect : property to set color effect flags: readable, writable Enum "GstNvCamColorEffectMode" Default: 1, "off" (1): off - GST_NVCAM_COLOR_EFFECT_OFF (2): mono - GST_NVCAM_COLOR_EFFECT_MONO (3): negative - GST_NVCAM_COLOR_EFFECT_NEGATIVE (4): solarize - GST_NVCAM_COLOR_EFFECT_SOLARIZE (5): sepia - GST_NVCAM_COLOR_EFFECT_SEPIA (6): posterize - GST_NVCAM_COLOR_EFFECT_POSTERIZE (7): aqua - GST_NVCAM_COLOR_EFFECT_AQUA auto-exposure : property to set auto exposure mode flags: readable, writable Enum "GstNvCamAutoExposureMode" Default: 2, "on" (1): off - GST_NVCAM_AUTO_EXPOSURE_OFF (2): on - GST_NVCAM_AUTO_EXPOSURE_ON (3): OnAutoFlash - GST_NVCAM_AUTO_EXPOSURE_AUTOFLASH (4): OnAlwaysFlash - GST_NVCAM_AUTO_EXPOSURE_ALWAYSFLASH (5): OnFlashRedEye - GST_NVCAM_AUTO_EXPOSURE_AUTO_RED_EYE flash : property to set flash mode flags: readable, writable Enum "GstNvCamFlashMode" Default: 0, "off" (0): off - GST_NVCAM_FLASH_OFF (1): on - GST_NVCAM_FLASH_ON (2): torch - GST_NVCAM_FLASH_TORCH (3): auto - GST_NVCAM_FLASH_AUTO flicker : property to select flicker detection and avoidance mode flags: readable, writable Enum "GstNvCamFlickerMode" Default: 3, "auto" (0): off - GST_NVCAM_FLICKER_OFF (1): 50Hz - GST_NVCAM_FLICKER_50HZ (2): 60Hz - GST_NVCAM_FLICKER_60HZ (3): auto - GST_NVCAM_FLICKER_AUTO contrast : property to adjust contrast value flags: readable, writable Float. Range: 0 - 1 Default: 0 saturation : property to adjust saturation value flags: readable, writable Float. Range: 0 - 2 Default: 1 tnr-strength : property to adjust temporal noise reduction strength flags: readable, writable Float. Range: 0 - 1 Default: 0 tnr-mode : property to select temporal noise reduction mode flags: readable, writable Enum "GstNvCamTNRMode" Default: 0, "NoiseReduction_Off" (0): NoiseReduction_Off - GST_NVCAM_NR_OFF (1): NoiseReduction_Fast - GST_NVCAM_NR_FAST (2): NoiseReduction_HighQuality - GST_NVCAM_NR_HIGHQUALITY edge-enhancement : property to adjust edge enhnacement value flags: readable, writable Float. Range: 0 - 1 Default: -1 trigger : property to fire pre capture trigger flags: readable, writable Enum "GstNvCamCaptureTrigger" Default: 0, "" (1): idle - GST_NVCAM_TRIGGER_IDLE (2): start - GST_NVCAM_TRIGGER_START intent : property to select capture intent flags: readable, writable Enum "GstNvCamCaptureIntent" Default: 1, "preview" (1): preview - GST_NVCAM_INTENT_PREVIEW (2): still - GST_NVCAM_INTENT_STILL (3): video - GST_NVCAM_INTENT_VIDEO (4): video-snapshot - GST_NVCAM_INTENT_VIDEO_SNAPSHOT (5): zsl - GST_NVCAM_INTENT_ZSL sensor-id : Set the id of camera sensor to use. Default 0.) flags: readable, writable Integer. Range: 0 - 255 Default: 0 enable-exif : Return Exif data with captured frame flags: readable, writable Boolean. Default: false enable-meta : Return Sensor meta data with captured frame flags: readable, writable Boolean. Default: false aeRegion : Property to set region of interest for auto exposure. Use GArray, with values of ROI coordinates (top,left,bottom,right) and weight in that order, to set the property. flags: readable, writable Boxed pointer of type "GArray" wbRegion : Property to set region of interest for white balance. Use GArray, with values of ROI coordinates (top,left,bottom,right) and weight in that order, to set the property. flags: readable, writable Boxed pointer of type "GArray" fpsRange : Property to set FPS range. Use string with values of FPS Range (low, high) in that order, to set the property. flags: readable, writable String. Default: null exposure-time : Property to adjust exposure time. This property will have effect only when Exposure mode is Off. flags: readable, writable Float. Range: 0 - 3.402823e+38 Default: 0.033 wbManualMode : Property to set manual mode of White balance. This property will have effect only when wbMode is manual. flags: readable, writable Enum "GstNvCamWBManualMode" Default: 0, "off" (0): off - GST_NVCAM_WB_MANUAL_OFF (1): reset - GST_NVCAM_WB_MANUAL_RESET (2): restore - GST_NVCAM_WB_MANUAL_RESTORE (3): external-gains - GST_NVCAM_WB_MANUAL_EXTGAINS wbGains : Property to set external white balance Gains. Use GArray, with values of WB gains (R, GR, GB, B) in that order, to set the property. This will have effect only when wbMode is manual. flags: readable, writable Boxed pointer of type "GArray" aeLock : Property to lock auto exposure. flags: readable, writable Boolean. Default: false callback : Property to set user callback. It is required if user wants to change capture settings per frame. Provide address of GstNvCamSrcCallbacks type struct as value. flags: writable Pointer. Write only dump-bayer : Dump raw bayer data in still capture mode. flags: readable, writable Boolean. Default: false num-sensors : Property to get number of sensors. Returns GArray containing number of sensors {usb,csi} flags: readable Boxed pointer of type "GArray" enable-surround-cameras: Enable Surround Camera (Res: 3456x2304) flags: readable, writable Boolean. Default: false stitch-in-w : Set the stitch in W. flags: readable, writable Integer. Range: 0 - 4336 Default: 4336 stitch-in-h : Set the stitch in H. flags: readable, writable Integer. Range: 0 - 2440 Default: 2440 surface-layout : Set the surface layout. Default 0. (0=Blocklinear, 1=Pitch) flags: readable, writable Integer. Range: 0 - 1 Default: 0 orientation : Sensor orientation value. This must be accessed after PAUSED state. flags: readable Integer. Range: 0 - 360 Default: -1
v4l2src
Starting on L4T R23.2 there is a /dev/video0 node to capture, however, this node will give you frames in bayer which are NOT suitable to encode because it grabs frames directly from the ov5693 camera without using the ISP.
https://devtalk.nvidia.com/default/topic/923984/ov5693-through-v4l2-interface-not-working-in-23-2/
It works also with the UVC driver.
Capture and Display
nvcamerasrc
gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
ARM consumption: 46.1% of one core (4 cores available 400% - this means something like 14% of total). However, only the capture is 11.25% so the display consumes 2.75%. For other camera you can find in our wiki latency numbers for glass to glass measurements
RAM 1184/7853MB (lfb 1246x4MB) cpu [2%@345,off,off,0%@345,0%@345,0%@345] RAM 1186/7853MB (lfb 1246x4MB) cpu [2%@1881,off,off,0%@1881,6%@1881,0%@1881] RAM 1239/7853MB (lfb 1246x4MB) cpu [21%@345,off,off,26%@345,18%@345,16%@345] ->Start capture RAM 1239/7853MB (lfb 1246x4MB) cpu [18%@345,off,off,9%@345,9%@345,10%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [21%@345,off,off,12%@345,8%@345,6%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [9%@345,off,off,12%@345,12%@345,15%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [11%@345,off,off,14%@345,17%@345,8%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [13%@345,off,off,15%@345,17%@345,8%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [10%@345,off,off,14%@345,14%@345,10%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [14%@345,off,off,22%@345,9%@345,12%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [12%@345,off,off,14%@345,8%@345,20%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [9%@345,off,off,9%@345,17%@345,19%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [7%@345,off,off,17%@345,20%@345,12%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [9%@345,off,off,17%@345,13%@345,12%@345] RAM 1239/7853MB (lfb 1246x4MB) cpu [12%@345,off,off,20%@345,7%@345,14%@345] RAM 1194/7853MB (lfb 1246x4MB) cpu [9%@345,off,off,9%@345,15%@345,8%@345] ->Stop capture RAM 1194/7853MB (lfb 1246x4MB) cpu [0%@345,off,off,0%@345,2%@345,0%@345] RAM 1194/7853MB (lfb 1246x4MB) cpu [0%@345,off,off,0%@345,1%@345,0%@345]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1574 root 20 0 9.979g 208932 44656 S 46.2 2.6 5:18.85 nvcamera-daemon 11507 nvidia 20 0 423932 11508 8796 S 3.6 0.1 0:00.66 gst-launch-1.0
H264 encoding
videotestsrc
ARM load: 157% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.
gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=test.h264 -e
VLC can play the file correctly at 30fps and reports as codec: H264-MPEG-4 AVC (part 10) (h264). The fan on top of the SoC didn't turn on.
nvcamerasrc
gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxh264enc bitrate=8000000 \ ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=test.h264 -e
Codecvisa reports bitrate on 7.96Mbps when variable mode is selected on the encoder.
ARM consumption 45.4% of one core (4 cores available 400% - this means something like 16.75% of total). However, only the capture is 11.25% so the h264 consumes 5.5%
RAM 1279/7853MB (lfb 1237x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267] RAM 1280/7853MB (lfb 1237x4MB) cpu [8%@345,off,off,19%@345,10%@345,12%@345] RAM 1279/7853MB (lfb 1237x4MB) cpu [16%@345,off,off,16%@345,7%@345,13%@345] RAM 1279/7853MB (lfb 1237x4MB) cpu [18%@345,off,off,11%@345,10%@345,16%@345] RAM 1279/7853MB (lfb 1236x4MB) cpu [24%@345,off,off,12%@345,11%@345,13%@345] RAM 1280/7853MB (lfb 1236x4MB) cpu [15%@345,off,off,14%@345,10%@345,14%@345] RAM 1280/7853MB (lfb 1236x4MB) cpu [18%@345,off,off,14%@345,16%@345,11%@345] RAM 1280/7853MB (lfb 1236x4MB) cpu [15%@345,off,off,19%@345,13%@345,16%@345] RAM 1280/7853MB (lfb 1236x4MB) cpu [14%@345,off,off,18%@345,15%@345,11%@345] RAM 1280/7853MB (lfb 1236x4MB) cpu [20%@345,off,off,13%@345,13%@345,14%@345] RAM 1280/7853MB (lfb 1235x4MB) cpu [15%@345,off,off,14%@345,18%@345,12%@345] RAM 1280/7853MB (lfb 1235x4MB) cpu [15%@345,off,off,17%@345,10%@345,13%@345] RAM 1279/7853MB (lfb 1235x4MB) cpu [18%@345,off,off,19%@345,6%@345,13%@345] RAM 1280/7853MB (lfb 1235x4MB) cpu [15%@345,off,off,19%@345,13%@345,10%@345] RAM 1280/7853MB (lfb 1235x4MB) cpu [17%@345,off,off,18%@345,10%@345,14%@345] RAM 1235/7853MB (lfb 1235x4MB) cpu [11%@345,off,off,7%@345,7%@345,8%@345] RAM 1234/7853MB (lfb 1235x4MB) cpu [0%@345,off,off,1%@345,0%@345,0%@345] RAM 1234/7853MB (lfb 1235x4MB) cpu [0%@345,off,off,3%@345,0%@345,0%@345]
1574 root 20 0 9.979g 214836 44684 S 45.4 2.7 7:10.27 nvcamera-daemon 11614 nvidia 20 0 700192 12800 9236 S 8.6 0.2 0:00.88 gst-launch-1.0
The encoder exposes the following parameters
control-rate : Bitrate control method flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncControlRate" Default: 1, "variable" (0): disable - Disable (1): variable - Variable (2): constant - Constant (3): variable-skip-frames - Variable Skip Frames (4): constant-skip-frames - Constant Skip Frames bitrate : Target bitrate flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 4294967295 Default: 4000000 quant-i-frames : Quantization parameter for I-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-p-frames : Quantization parameter for P-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-b-frames : Quantization parameter for B-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 iframeinterval : Encoding Intra Frame occurance frequency flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false SliceIntraRefreshInterval: Set SliceIntraRefreshInterval flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 60 bit-packetization : Whether or not Packet size is based upon Number Of bits flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false vbv-size : virtual buffer size = vbv-size * (bitrate/fps) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 30 Default: 10 temporal-tradeoff : Temporal Tradeoff value for encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOmxVideoEncTemporalTradeoffType" Default: 0, "Do not drop frames" (0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES (1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES (2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES (3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES (4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES EnableMVBufferMeta : Enable Motion Vector Meta data for encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false qp-range : Qunatization range for P and I frame, Use string with values of Qunatization Range in MinQpP-MaxQpP:MinQpI-MaxQpP:MinQpB-MaxQpB order, to set the property. flags: readable, writable String. Default: "-1,-1:-1,-1:-1,-1" MeasureEncoderLatency: Enable Measure Encoder latency Per Frame flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false EnableTwopassCBR : Enable two pass CBR while encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false preset-level : HW preset level for encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncHwPreset" Default: 0, "UltraFastPreset" (0): UltraFastPreset - UltraFastPreset for high perf (1): FastPreset - FastPreset (2): MediumPreset - MediumPreset (3): SlowPreset - SlowPreset insert-sps-pps : Insert H.264 SPS, PPS at every IDR frame flags: readable, writable Boolean. Default: false num-B-Frames : Number of B Frames between two reference frames (not recommended) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 2 Default: 0 slice-header-spacing: Slice Header Spacing number of macroblocks/bits in one packet flags: readable, writable, changeable only in NULL or READY state Unsigned Long. Range: 0 - 18446744073709551615 Default: 0 profile : Set profile for encode flags: readable, writable, changeable only in NULL or READY state Enum "GstOmxVideoEncProfileType" Default: 1, "baseline" (1): baseline - GST_OMX_VIDENC_BASELINE_PROFILE (2): main - GST_OMX_VIDENC_MAIN_PROFILE (8): high - GST_OMX_VIDENC_HIGH_PROFILE
Dual H264 encoding
gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \ streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testA.h264 \ streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testB.h264
ARM consumption ~44.1% of one core (4 cores available 400% - this means something like 16.5% of total). However, only the capture is 11.25% so the dual h264 consumes 5.25%
RAM 1320/7853MB (lfb 1221x4MB) cpu [0%@1420,off,off,0%@1420,0%@1420,0%@1420] RAM 1320/7853MB (lfb 1221x4MB) cpu [19%@345,off,off,16%@345,10%@345,9%@345] RAM 1320/7853MB (lfb 1221x4MB) cpu [18%@345,off,off,18%@345,16%@345,13%@345] RAM 1321/7853MB (lfb 1221x4MB) cpu [20%@345,off,off,16%@345,19%@345,13%@345] RAM 1321/7853MB (lfb 1221x4MB) cpu [19%@345,off,off,14%@345,12%@345,15%@345] RAM 1321/7853MB (lfb 1220x4MB) cpu [18%@345,off,off,17%@345,14%@345,13%@345] RAM 1321/7853MB (lfb 1219x4MB) cpu [19%@345,off,off,12%@345,18%@345,15%@345] RAM 1321/7853MB (lfb 1219x4MB) cpu [14%@345,off,off,13%@345,14%@345,20%@345] RAM 1321/7853MB (lfb 1218x4MB) cpu [14%@345,off,off,15%@345,21%@345,18%@345] RAM 1321/7853MB (lfb 1218x4MB) cpu [15%@345,off,off,20%@345,15%@345,16%@345] RAM 1321/7853MB (lfb 1217x4MB) cpu [14%@345,off,off,17%@345,11%@345,21%@345] RAM 1321/7853MB (lfb 1217x4MB) cpu [18%@345,off,off,22%@345,13%@345,18%@345] RAM 1321/7853MB (lfb 1217x4MB) cpu [15%@345,off,off,19%@345,20%@345,12%@345] RAM 1321/7853MB (lfb 1216x4MB) cpu [15%@345,off,off,21%@345,16%@345,13%@345] RAM 1321/7853MB (lfb 1216x4MB) cpu [16%@345,off,off,22%@345,18%@345,12%@345] RAM 1275/7853MB (lfb 1216x4MB) cpu [15%@345,off,off,11%@345,11%@345,13%@345] RAM 1275/7853MB (lfb 1216x4MB) cpu [1%@345,off,off,2%@345,0%@345,0%@345] RAM 1275/7853MB (lfb 1216x4MB) cpu [0%@345,off,off,2%@345,0%@345,0%@345] RAM 1275/7853MB (lfb 1216x4MB) cpu [0%@345,off,off,2%@345,0%@345,0%@345] RAM 1275/7853MB (lfb 1216x4MB) cpu [0%@345,off,off,1%@345,0%@345,0%@345]
H264 UDP Streaming
server:
CLIENT_IP=<IP_ADDRESS> gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \ ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ omxh264enc control-rate=2 bitrate=4000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! \ h264parse ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false
PC client:
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264 ! xvimagesink sync=false async=false -e
Using VLC
- Server:
CLIENT_IP=<IP_ADDRESS> gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \ ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ omxh264enc control-rate=2 bitrate=4000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! \ h264parse ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false
- Client:
vlc -vv --demux h264 --h264-fps 60 udp://@:5000
h264 parse missing
In case gstreamer complains when running the pipeline for missing the h264parse plugin, run the following command to install it:
sudo apt-get install gstreamer1.0-plugins-bad
Latency
- 1080p30
- H264
- Bitrate 4Mbps
- Latency=138ms
VP8 encoding
videotestsrc
ARM load: 156% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.
gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxvp8enc ! qtmux ! filesink location=test.mp4 -e
VLC can play the file correctly at 30fps and reports as codec: Google/On2's VP8 Video (VP80). The fan on top of the SoC didn't turn on.
nvcamerasrc
gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxvp8enc bitrate=8000000 ! qtmux ! filesink location=test.mp4 -e
VLC reports bitrate on ~8Mbps.
ARM consumption ~50% of one core (4 cores available 400% - this means something like 15.5% of total). However, only the capture is 11.25% so the VP8 consumes 4.25%
RAM 1442/7853MB (lfb 1134x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267] RAM 1442/7853MB (lfb 1134x4MB) cpu [16%@345,off,off,12%@345,15%@345,14%@345] RAM 1442/7853MB (lfb 1134x4MB) cpu [11%@345,off,off,17%@345,14%@345,17%@345] RAM 1442/7853MB (lfb 1134x4MB) cpu [17%@345,off,off,18%@345,13%@345,10%@345] RAM 1442/7853MB (lfb 1134x4MB) cpu [19%@345,off,off,18%@345,14%@345,9%@345] RAM 1442/7853MB (lfb 1134x4MB) cpu [13%@499,off,off,14%@499,15%@499,17%@499] RAM 1442/7853MB (lfb 1134x4MB) cpu [16%@345,off,off,15%@345,8%@345,11%@345] RAM 1443/7853MB (lfb 1134x4MB) cpu [14%@345,off,off,13%@345,9%@345,15%@345] RAM 1442/7853MB (lfb 1134x4MB) cpu [9%@345,off,off,16%@345,13%@345,9%@345] RAM 1443/7853MB (lfb 1134x4MB) cpu [10%@345,off,off,15%@345,8%@345,14%@345] RAM 1442/7853MB (lfb 1134x4MB) cpu [11%@345,off,off,19%@345,10%@345,15%@345] RAM 1443/7853MB (lfb 1134x4MB) cpu [14%@345,off,off,23%@345,10%@345,13%@345] RAM 1443/7853MB (lfb 1134x4MB) cpu [14%@345,off,off,20%@345,8%@345,12%@345] RAM 1443/7853MB (lfb 1134x4MB) cpu [15%@345,off,off,21%@345,6%@345,10%@345]
1574 root 20 0 9.979g 264008 44716 S 45.7 3.3 14:43.58 nvcamera-daemon 13065 nvidia 20 0 719944 14140 9996 S 9.2 0.2 0:09.34 gst-launch-1.0
The encoder exposes the following parameters
bitrate : Target bitrate flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 4294967295 Default: 4000000 quant-i-frames : Quantization parameter for I-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-p-frames : Quantization parameter for P-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-b-frames : Quantization parameter for B-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 iframeinterval : Encoding Intra Frame occurance frequency flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false SliceIntraRefreshInterval: Set SliceIntraRefreshInterval flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 60 bit-packetization : Whether or not Packet size is based upon Number Of bits flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false vbv-size : virtual buffer size = vbv-size * (bitrate/fps) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 30 Default: 10 temporal-tradeoff : Temporal Tradeoff value for encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOmxVideoEncTemporalTradeoffType" Default: 0, "Do not drop frames" (0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES (1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES (2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES (3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES (4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES EnableMVBufferMeta : Enable Motion Vector Meta data for encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false qp-range : Qunatization range for P and I frame, Use string with values of Qunatization Range in MinQpP-MaxQpP:MinQpI-MaxQpP:MinQpB-MaxQpB order, to set the property. flags: readable, writable String. Default: "-1,-1:-1,-1:-1,-1" MeasureEncoderLatency: Enable Measure Encoder latency Per Frame flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false EnableTwopassCBR : Enable two pass CBR while encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false preset-level : HW preset level for encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncHwPreset" Default: 0, "UltraFastPreset" (0): UltraFastPreset - UltraFastPreset for high perf (1): FastPreset - FastPreset (2): MediumPreset - MediumPreset (3): SlowPreset - SlowPreset
H265 encoding
VLC with H265 support
In order to support h265 you need VLC 2.2 in your computer to play the videos
sudo add-apt-repository ppa:mc3man/trusty-media sudo apt-get update sudo apt-get install vlc vlc-plugin-*
videotestsrc
ARM load: 156% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.
gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh265enc ! filesink location=test.h265 -e </pre> VLC can play the file correctly at 30fps and reports as codec: MPEG-H Part2/HEVC (H.265)(hevc). The fan on top of the SoC didn't turn on. ==== nvcamerasrc ==== <syntaxhighlight lang='bash'> gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxh265enc ! filesink location=test.h265 -e
Codecvisa reports bitrate on 7.96Mbps when variable mode is selected on the encoder.
ARM consumption ~55% of one core (4 cores available 400% - this means something like 13% of total). However, only the capture is 11.25% so the h264 consumes 1.75%
RAM 1487/7853MB (lfb 993x4MB) cpu [0%@1267,off,off,0%@1267,0%@1267,0%@1267] RAM 1487/7853MB (lfb 993x4MB) cpu [12%@345,off,off,12%@345,10%@345,14%@345] RAM 1487/7853MB (lfb 993x4MB) cpu [9%@345,off,off,19%@345,14%@345,9%@345] RAM 1487/7853MB (lfb 993x4MB) cpu [16%@345,off,off,14%@345,12%@345,16%@345] RAM 1487/7853MB (lfb 993x4MB) cpu [16%@345,off,off,11%@345,18%@345,20%@345] RAM 1487/7853MB (lfb 992x4MB) cpu [14%@345,off,off,14%@345,12%@345,22%@345] RAM 1488/7853MB (lfb 992x4MB) cpu [16%@345,off,off,18%@345,15%@345,15%@345] RAM 1487/7853MB (lfb 992x4MB) cpu [22%@345,off,off,16%@345,8%@345,17%@345] RAM 1487/7853MB (lfb 992x4MB) cpu [24%@345,off,off,17%@345,7%@345,14%@345] RAM 1487/7853MB (lfb 992x4MB) cpu [13%@345,off,off,20%@345,20%@345,7%@345] RAM 1487/7853MB (lfb 992x4MB) cpu [17%@345,off,off,19%@345,14%@345,15%@345] RAM 1488/7853MB (lfb 992x4MB) cpu [15%@345,off,off,22%@345,12%@345,14%@345] RAM 1488/7853MB (lfb 992x4MB) cpu [16%@345,off,off,14%@345,21%@345,13%@345] RAM 1488/7853MB (lfb 992x4MB) cpu [15%@345,off,off,19%@345,13%@345,16%@345] RAM 1488/7853MB (lfb 992x4MB) cpu [11%@345,off,off,11%@345,16%@345,16%@345] RAM 1488/7853MB (lfb 991x4MB) cpu [22%@345,off,off,11%@345,15%@345,15%@345] RAM 1488/7853MB (lfb 991x4MB) cpu [18%@345,off,off,14%@345,12%@345,14%@345] RAM 1488/7853MB (lfb 991x4MB) cpu [8%@345,off,off,13%@345,20%@345,10%@345] RAM 1488/7853MB (lfb 991x4MB) cpu [11%@345,off,off,12%@345,11%@345,21%@345]
1574 root 20 0 9.979g 266012 44612 S 47.7 3.3 17:54.85 nvcamera-daemon 13273 nvidia 20 0 694396 12920 9156 S 8.6 0.2 0:01.32 gst-launch-1.0
The encoder exposes the following parameters
control-rate : Bitrate control method flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncControlRate" Default: 1, "variable" (0): disable - Disable (1): variable - Variable (2): constant - Constant (3): variable-skip-frames - Variable Skip Frames (4): constant-skip-frames - Constant Skip Frames bitrate : Target bitrate flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 4294967295 Default: 4000000 quant-i-frames : Quantization parameter for I-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-p-frames : Quantization parameter for P-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-b-frames : Quantization parameter for B-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 iframeinterval : Encoding Intra Frame occurance frequency flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false SliceIntraRefreshInterval: Set SliceIntraRefreshInterval flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 60 bit-packetization : Whether or not Packet size is based upon Number Of bits flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false vbv-size : virtual buffer size = vbv-size * (bitrate/fps) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 30 Default: 10 temporal-tradeoff : Temporal Tradeoff value for encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOmxVideoEncTemporalTradeoffType" Default: 0, "Do not drop frames" (0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES (1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES (2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES (3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES (4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES EnableMVBufferMeta : Enable Motion Vector Meta data for encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false qp-range : Qunatization range for P and I frame, Use string with values of Qunatization Range in MinQpP-MaxQpP:MinQpI-MaxQpP:MinQpB-MaxQpB order, to set the property. flags: readable, writable String. Default: "-1,-1:-1,-1:-1,-1" MeasureEncoderLatency: Enable Measure Encoder latency Per Frame flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false EnableTwopassCBR : Enable two pass CBR while encoding flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false preset-level : HW preset level for encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncHwPreset" Default: 0, "UltraFastPreset" (0): UltraFastPreset - UltraFastPreset for high perf (1): FastPreset - FastPreset (2): MediumPreset - MediumPreset (3): SlowPreset - SlowPreset slice-header-spacing: Slice Header Spacing number of macroblocks/bits in one packet flags: readable, writable, changeable only in NULL or READY state Unsigned Long. Range: 0 - 18446744073709551615 Default: 0
Dual H265 encoding
gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \ streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testA.h264 \ streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testB.h264
gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \ streams. ! omxh265enc bitrate=8000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! filesink location=testA.h265 \ streams. ! omxh265enc bitrate=8000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! filesink location=testB.h265
ARM consumption ~48.7% of one core (4 cores available 400% - this means something like 13.5% of total). However, only the capture is 11.25% so the dual h265 consumes 2.25%
RAM 1503/7853MB (lfb 965x4MB) cpu [0%@1420,off,off,0%@1420,0%@1420,0%@1420] RAM 1503/7853MB (lfb 965x4MB) cpu [19%@345,off,off,10%@345,13%@345,16%@345] RAM 1503/7853MB (lfb 965x4MB) cpu [17%@345,off,off,15%@345,9%@345,12%@345] RAM 1503/7853MB (lfb 964x4MB) cpu [13%@345,off,off,13%@345,14%@345,19%@345] RAM 1503/7853MB (lfb 964x4MB) cpu [18%@345,off,off,13%@345,10%@345,13%@345] RAM 1503/7853MB (lfb 963x4MB) cpu [12%@345,off,off,10%@345,14%@345,18%@345] RAM 1503/7853MB (lfb 962x4MB) cpu [19%@345,off,off,12%@345,19%@345,10%@345] RAM 1504/7853MB (lfb 962x4MB) cpu [10%@345,off,off,14%@345,16%@345,18%@345] RAM 1504/7853MB (lfb 961x4MB) cpu [24%@345,off,off,10%@345,13%@345,15%@345] RAM 1504/7853MB (lfb 961x4MB) cpu [14%@345,off,off,13%@345,12%@345,18%@345] RAM 1504/7853MB (lfb 961x4MB) cpu [16%@345,off,off,15%@345,9%@345,14%@345] RAM 1504/7853MB (lfb 957x4MB) cpu [17%@345,off,off,20%@345,18%@345,15%@345] RAM 1457/7853MB (lfb 957x4MB) cpu [14%@345,off,off,9%@345,10%@345,15%@345] RAM 1456/7853MB (lfb 957x4MB) cpu [0%@345,off,off,1%@345,0%@345,0%@345]
H265 UDP Streaming
In order to start streaming H265 you need the h265 payloader, to do this you need to compile the latest version of gstreamer in your Jetson TX2 module. Please follow the instructions to do it. Same instructions can be run in your PC if you want to update the gstreamer version in your host. (in case of the host I had to recompile the good plugins to get the rtph265pay).
Currently it is not possible to receive the stream using the PC because there is not a h265 decoder available on gstreamer. In order to evaluate the performance let's receive the stream on the Jetson module as well.
export PKG_CONFIG_PATH=/home/$USER/gst_$VERSION/out/lib/pkgconfig VERSION=1.8.0 export LD_LIBRARY_PATH=/home/$USER/gst_$VERSION/out/lib/
server:
CLIENT_IP=<IP_ADDRESS> gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \ ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \ omxh265enc low-latency=1 control-rate=2 bitrate=4000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! \ h265parse ! rtph265pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false
client:
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H265,payload=96 ! rtph265depay \ ! h265parse ! queue ! omxh265dec ! nvoverlaysink sync=false async=false -e
Latency
- 1080p30
- H265
- Bitrate 4Mbps
- Latency=251ms
nvgstcapture-1.0
This is an application based on gstreamer and omx to capture, encode and save video to the filesystem. It has a lot of options. The source code is not released by NVIDIA yet but they are planning to release in future versions of Linux 4 Tegra (L4T), you can read about it at Nvidia Embedded Computing Forum
nvidia@tegra-ubuntu:~$ nvgstcapture-1.0 --help Encoder null, cannot set bitrate! Encoder Profile = Baseline Usage: nvgstcapture-1.0 [OPTION...] Nvidia GStreamer Camera Model Test Help Options: -h, --help Show help options --help-all Show all help options --help-gst Show GStreamer Options Application Options: --prev-res Preview width & height. USB Range: 0 to 12 (5632x4224) and CSI Range: 2 to 12 (5632x4224) e.g., --prev-res=3 --cus-prev-res Custom Preview width & height [for CSI only] e.g., --cus-prev-res=1920x1080 --image-res Image width & height. Range: 0 to 12 (5632x4224) e.g., --image-res=3 --video-res Video width & height. Range: 0 to 9 (3896x2192) e.g., --video-res=3 -m, --mode Capture mode value (1=still 2=video) -v, --video-enc Video encoder type (0=h264[HW] 1=vp8[HW] 2=h265[HW] 3=vp9[HW]) -b, --enc-bitrate Video encoding Bit-rate(in bytes) e.g., --enc-bitrate=4000000 --enc-profile Video encoder profile (Only for H.264) (0=Baseline, 1=Main, 2=High) -J, --image-enc Image encoder type (0=jpeg_SW[jpegenc] 1=jpeg_HW[nvjpegenc]) -k, --file-type Container file type (0=mp4 1=3gp 2=mkv) --cap-dev-node Video capture device node (0=/dev/video0[default], 1=/dev/video1, 2=/dev/video2) e.g., --cap-dev-node=0 --svs [For USB] (=) chain for video Preview. [For CSI only] use "nvoverlaysink" --file-name Captured file name. nvcamtest is used by default --camsrc Camera Source to use (0=v4l2, 1=csi[default], 2=videotest, 3=eglstream) --orientation [For CSI only] Camera sensor orientation value -w, --whitebalance [For CSI only] Capture whitebalance value -s, --scene-mode [For CSI only] Camera Scene-Mode value -c, --color-effect [For CSI only] Camera Color Effect value --auto-exposure [For CSI only] Camera Auto-Exposure value --flash [For CSI only] Camera Flash value --flicker [For CSI only] Camera Flicker Detection and Avoidance Mode value --contrast [For CSI only] Camera Contrast value --saturation [For CSI only] Camera Saturation value --edge-enhancement [For CSI only] Camera Edge Enhancement value --tnr-strength [For CSI only] Camera TNR-Strength value --tnr-mode [For CSI only] Camera TNR Mode value --sensor-id [For CSI only] Camera Sensor ID value --eglstream-id [For CSI EGLStream Consumer] Select EGLStream Producer ID value. Default value 0 --display-id [For nvoverlaysink only] Display ID value --aeRegion [For CSI Only] ROI for AE coordinates(top,left,bottom,right) and weight in that order.e.g., --aeRegion="30 40 200 200 1.2" --wbRegion [For CSI Only] ROI for AWB coordinates(top,left,bottom,right) and weight in that order.e.g., --wbRegion="30 40 200 200 1.2" --fpsRange [For CSI Only] FPS range values (low, high) e.g., --fpsRange="15 30" --exposure-time [For CSI only] Capture exposure time value. e.g., --exposure-time=0.033 --wbGains [For CSI Only] WB gains values (R, GR, GB, B) in that order. e.g., --wbGains="1.2 1.4 0.8 1.6" --overlayConfig Overlay Configuration Options index and coordinates in (index, x_pos, y_pos, width, height) order e.g. --overlayConfig="0, 0, 0, 1280, 720" --enable-exif Enable Exif data --dump-bayer Dump bayer data in addition to image capture. --eglConfig EGL window Coordinates (x_pos y_pos) in that order e.g., --eglConfig="50 100" --aeLock [For CSI only] Enable auto exposure lock e.g., --aeLock --enable-meta Enable Sensor MetaData reporting --nvvideosink-create-eglstream [For nvvideosink EGLStream Producer] Enable nvvideosink EGLStream Producer --callback Enable User callback to set capture settings per frame --app-profile Enable KPI profiling --kpi-numbers Enable KPI measurement -A, --automate Run application in automation mode -S, --start-time Start capture after specified time in seconds. Default = 5 sec (use with --automate or -A only) -Q, --quit-after Quit application once automation is done after specified time in seconds. Default = 0 sec (use with --automate or -A only) -C, --count Number of iterations of automation testcase. Default = 1 (use with --automate or -A only) --capture-gap Number of milliseconds between successive image/video capture. Default = 250 msec (use with --automate and --capture-auto only) --capture-time Capture video for specified time in seconds. Default = 10 sec (use with --automate and --capture-auto only) --toggle-mode Toggle between still and video capture modes for count number of times (use with --automate or -A only) --toggle-sensor Toggle between sensor-id 0 and 1 (use with --automate or -A only) --capture-auto Do image/video capture in automation mode for count number of times(use with --automate or -A only) --enum-wb Enumerate all white-balance modes for count number of times (use with --automate or -A only) --enum-scm Enumerate all scene modes for count number of times (use with --automate or -A only) --enum-ce Enumerate all color-effect modes for count number of times (use with --automate or -A only) --enum-ae Enumerate all auto-exposure modes for count number of times (use with --automate or -A only) --enum-f Enumerate all flash modes for count number of times (use with --automate or -A only) --enum-fl Enumerate all flicker detection and avoidance modes for count number of times (use with --automate or -A only) --enum-ct Enumerate contrast value through 0 to 1 by a step of 0.1 for count number of times (use with --automate or -A only) --enum-st Enumerate saturation value through 0 to 2 by a step of 0.1 for count number of times (use with --automate or -A only) --enum-ee Enumerate edge-enhancement value through 0 to 1 by a step of 0.1 for count number of times (use with --automate or -A only) --enum-ts Enumerate TNR strength value through 0 to 1 by a step of 0.1 for count number of times (use with --automate or -A only) --enum-tnr Enumerate all TNR modes for count number of times (use with --automate or -A only) --streaming-mode [For CSI Only] Enable streaming mode (1=stream only, 2=stream+record) --streaming-file File to stream instead of camera input Supported resolutions in case of CSI Camera (2) : 640x480 (3) : 1280x720 (4) : 1920x1080 (5) : 2104x1560 (6) : 2592x1944 (7) : 2616x1472 (8) : 3840x2160 (9) : 3896x2192 (10): 4208x3120 (11): 5632x3168 (12): 5632x4224 Runtime CSI Camera Commands: Help : 'h' Quit : 'q' Set Capture Mode: mo:<val> (1): image (2): video Get Capture Mode: gmo Set Sensor Id (0 to 10): sid:<val> e.g., sid:2 Get Sensor Id: gsid Set sensor orientation: so:<val> (0): none (1): Rotate counter-clockwise 90 degrees (2): Rotate 180 degrees (3): Rotate clockwise 90 degrees Get sensor orientation: gso Set Whitebalance Mode: wb:<val> (0): off (1): auto (2): incandescent (3): fluorescent (4): warm-fluorescent (5): daylight (6): cloudy-daylight (7): twilight (8): shade (9): manual Get Whitebalance Mode: gwb Set Scene-Mode: scm:<val> (0): face-priority (1): action (2): portrait (3): landscape (4): night (5): night-portrait (6): theatre (7): beach (8): snow (9): sunset (10): steady-photo (11): fireworks (12): sports (13): party (14): candle-light (15): barcode Get Scene-Mode: gscm Set Color Effect Mode: ce:<val> (1): off (2): mono (3): negative (4): solarize (5): sepia (6): posterize (7): aqua Get Color Effect Mode: gce Set Auto-Exposure Mode: ae:<val> (1): off (2): on (3): OnAutoFlash (4): OnAlwaysFlash (5): OnFlashRedEye Get Auto-Exposure Mode: gae Set Flash Mode: f:<val> (0): off (1): on (2): torch (3): auto Get Flash Mode: gf Set Flicker Detection and Avoidance Mode: fl:<val> (0): off (1): 50Hz (2): 60Hz (3): auto Get Flicker Detection and Avoidance Mode: gfl Set Contrast (0 to 1): ct:<val> e.g., ct:0.75 Get Contrast: gct Set Saturation (0 to 2): st:<val> e.g., st:1.25 Get Saturation: gst Set Exposure Time in seconds: ext:<val> e.g., ext:0.033 Get Exposure Time: gext Set Auto Exposure Lock(0/1): ael:<val> e.g., ael:1 Get Auto Exposure Lock: gael Set Edge Enhancement (0 to 1): ee:<val> e.g., ee:0.75 Get Edge Enhancement: gee Set ROI for AE: It needs five values, ROI coordinates(top,left,bottom,right) and weight in that order aer:<val> e.g., aer:20 20 400 400 1.2 Get ROI for AE: gaer Set ROI for AWB: It needs five values, ROI coordinates(top,left,bottom,right) and weight in that order wbr:<val> e.g., wbr:20 20 400 400 1.2 Get ROI for AWB: gwbr Set FPS range: It needs two values, FPS Range (low, high) in that order fpsr:<val> e.g., fpsr:15 30 Get FPS range: gfpsr Set WB Gains: It needs four values (R, GR, GB, B) in that order wbg:<val> e.g., wbg:1.2 2.2 0.8 1.6 Get WB Gains: gwbg Set TNR Strength (0 to 1): ts:<val> e.g., ts:0.75 Get TNR Strength: gts Set TNR Mode: tnr:<val> (0): NoiseReduction_Off (1): NoiseReduction_Fast (2): NoiseReduction_HighQuality Get TNR Mode: gtnr Capture: enter 'j' OR followed by a timer (e.g., jx5000, capture after 5 seconds) OR followed by multishot count (e.g., j:6, capture 6 images) timer/multihot values are optional, capture defaults to single shot with timer=0s Start Recording : enter '1' Stop Recording : enter '0' Video snapshot : enter '2' (While recording video) Set Preview Resolution: pcr:<val> e.g., pcr:3 (2) : 640x480 (3) : 1280x720 (4) : 1920x1080 (5) : 2104x1560 (6) : 2592x1944 (7) : 2616x1472 (8) : 3840x2160 (9) : 3896x2192 (10): 4208x3120 (11): 5632x3168 (12): 5632x4224 Note - For preview resolutions 4208x3120 and more use option --svs=nveglglessink Get Preview Resolution: gpcr Set Image Resolution: icr:<val> e.g., icr:3 (2) : 640x480 (3) : 1280x720 (4) : 1920x1080 (5) : 2104x1560 (6) : 2592x1944 (7) : 2616x1472 (8) : 3840x2160 (9) : 3896x2192 (10): 4208x3120 (11): 5632x3168 (12): 5632x4224 Get Image Capture Resolution: gicr Set Video Resolution: vcr:<val> e.g., vcr:3 (2) : 640x480 (3) : 1280x720 (4) : 1920x1080 (5) : 2104x1560 (6) : 2592x1944 (7) : 2616x1472 (8) : 3840x2160 (9) : 3896x2192 Get Video Capture Resolution: gvcr Runtime USB Camera Commands: Help : 'h' Quit : 'q' Set Capture Mode: mo:<val> (1): image (2): video Get Capture Mode: gmo Capture: enter 'j' OR followed by a timer (e.g., jx5000, capture after 5 seconds) OR followed by multishot count (e.g., j:6, capture 6 images) timer/multihot values are optional, capture defaults to single shot with timer=0s Start Recording : enter '1' Stop Recording : enter '0' Set Preview Resolution: pcr:<val> e.g., pcr:2 (0) : 176x144 (1) : 320x240 (2) : 640x480 (3) : 1280x720 (4) : 1920x1080 NOTE: Preview/Encode resolution will be same as Capture resolution for USB-Camera Get Preview Resolution: gpcr Get Image Capture Resolution: gicr Get Video Capture Resolution: gvcr Set Capture Device Node: cdn:<val> e.g., cdn:0 (0): /dev/video0 (1): /dev/video1 (2): /dev/video2 Get Capture Device Node: gcdn Runtime encoder configuration options: Set Encoding Bit-rate(in bytes): br:<val> e.g., br:4000000 Get Encoding Bit-rate(in bytes): gbr Set Encoding Profile(only for H.264): ep:<val> e.g., ep:1 (0): Baseline (1): Main (2): High Get Encoding Profile(only for H.264): gep Force IDR Frame on video Encoder(only for H.264): Enter 'f'
Capture snapshots
Sometimes when the board has been just flashed you need to run reboot in order to run nvgstcapture-1.0 successfully. To take snapshots:
nvgstcapture-1.0 #then press j to take an snapshot q
Resolutions available to create snapshot:
(2) : 640x480 (3) : 1280x720 (4) : 1920x1080 (5) : 2104x1560 (6) : 2592x1944 (7) : 2616x1472 (8) : 3840x2160 (9) : 3896x2192 (10): 4208x3120 (11): 5632x3168 (12): 5632x4224
nvgstcapture-1.0 -m 1 --image-res=10 #set 1 to snapshot mode
In general the application is consuming 15% of each of four cores. It is always displaying video this could affect.
Capture and record video
nvgstcapture-1.0 -m 2 # set the mode in video 1 # start video recording 2 # take a snapshot 0 # stop video recording
In case of video it should support:
(2) : 640x480 (3) : 1280x720 (4) : 1920x1080 (5) : 2104x1560 (6) : 2592x1944 (7) : 2616x1472 (8) : 3840x2160 (9) : 3896x2192
However, it is not possible to set other value than 2,3,4,5 up to 9.
nvgstcapture-1.0 -m 2 --video-res=5
top
top - 22:15:09 up 2:30, 4 users, load average: 0.47, 0.45, 0.30 Tasks: 267 total, 1 running, 266 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.0 us, 6.0 sy, 0.0 ni, 83.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 8041332 total, 754840 free, 1611400 used, 5675092 buff/cache KiB Swap: 0 total, 0 free, 0 used. 6303032 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1639 root 20 0 9.979g 264612 45284 S 52.3 3.3 6:14.66 nvcamera-daemon 27513 nvidia 20 0 815040 13932 10272 S 4.3 0.2 0:01.48 nvgstcapture-1.
There is a daemon and a pipeline. Interesting...
While recording the pipeline (nvgstcapture) increases the ARM load to 10% and then it goes down again, even with the highest resolution. It will be interesting to understand what is the daemon doing.
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.