Gstreamer pipelines for Tegra X1

From RidgeRun Developer Connection
Jump to: navigation, search

Introduction

On this page you are going to find a set of pipelines used on Tegra X1, specifically used with the Jetson board. The wiki page tries to describe some of the multimedia features of the platform like the NVIDIA model to handle the ISP through its custom (and close) plugin called nvcamerasrc. It also includes a brief description of the sample application called nvgstcapture-1.0 and ways to measure the ARM load. The goal is to provide you some example pipelines and introduction to the board. It is a work in progress.

The camera sensor used for tests is the ov5693. These steps were run on Jetson TX1.

nvgstcapture-1.0

This is an application based on gstreamer and omx to capture, encode and save video to the filesystem. It has a lot of options. The source code is not released by NVIDIA yet but they are planning to release in future versions of Linx 4 Tegra (L4T), you can read about it at Nvidia Embedded Computing Forum

ubuntu@tegra-ubuntu:~$ nvgstcapture-0.10 --help
Usage:
  nvgstcapture-0.10 [OPTION...] Nvidia GStreamer Model Test

Help Options:
  -h, --help                        Show help options
  --help-all                        Show all help options
  --help-gst                        Show GStreamer Options

Application Options:
  -W, --prev_res                    Preview width & height eg. -W 3
  -V, --vcap_res                    Video Capture width & height eg. -V 3
  -I, --icap_res                    Image Capture width & height eg. -I 3
  -m, --mode                        Capture mode value (2=video 1=still)
  -S, --starttime                   Start capture after specified time in seconds(use with --automate or -A only)
  -T, --recordfor                   Capture video for specified time in seconds(use with --automate or -A only)
  -Q, --quitafter                   Quit application once capture done after specified time in seconds(use with --automate or -A only)
  -J, --image_enc                   Image encoder type (0=jpeg_HW[nv_omx_jpegenc] 1=jpeg_SW[jpegenc] 2=jpeg_HW[nvjpegenc])
  -v, --video_enc                   Video encoder type (0=h264[nv_omx_h264enc] 1=vp8[nv_omx_vp8enc] 2=mpeg4_SW[ffenc_mpeg4]) 3=H263_SW[ffenc_h263][Set Default Resolution:704x576(4CIF)]
  -k, --file_type                   Container file type (0=mp4 1=3gp 2=avi)
  --frate                           Capture Frame Rate (min = 5, max = 30)
  --use-camerabin                   Use Camerabin2
  --svs                             (=) chain for video Preview
  -A, --automate                    Run application in automation mode
  -P, --perf                        Enable Camera Performance index measurement
  -R, --raw                         Enable Camera raw-yuv both Preview-Capture
  -E, --raw-preview                 Enable Camera only raw-yuv Preview
  -U, --raw-capture                 Enable Camera only raw-yuv Capture
  -C, --capture_auto                Do image/video capture in automation mode
  --vip                             Use Video input
  --format_type                     Capture format type, 0=NTSC, 1=PAL
  --frame_type                      Capture frame type, 0=Progressive, 1=Interlaced
  --usbcam                          Use USB Camera input
  -H, --usbcam_type                 USB Camera type, 0=YUV, 1=MJPEG
  --testsrc                         Use video test source input
  -M, --cap_format                  Capture Format, 0=I420, 1=YV12, 2=NV12, 3=YUY2, 4=UYVY, 5=YVYU, 6=Y444(Use --usbcam for cap format 0,1,3 & --testsrc for cap format 2,4,5,6) 
  --count                           Number of Iterations/Counts of stress testcase
  --capture-gap                     Number of Seconds between successive image/video capture (only with '--automate') 
  --toggle-autofocus                Toggle AutoFocus for Count number of times (default = 1) (only with '--automate')
  --toggle-sensor                   Toggle Camera Sensor between Primary (Back-Left) and Secondary (Front) for Count number of times (default = 0) (only with '--automate')
  --toggle-mode                     Toggle Between Still and Video Capture Modes for Count number of times (default = 1) (only with '--automate')
  --all-resolution-images           Capture Images of All Resolutions (do not use 'count' in this case) (only with '--automate')
  --all-resolution-videos           Record Videos of All Resolutions (do not use 'count' in this case) (only with '--automate')
  --file-name                       File name to be used for images or videos. nvgsttest is used by default

Runtime Commands:

   HELP: '-h'
  Quit : 'q'
  Set Capture Mode:
      mo:<val>
          (2): video
          (1): image
  Get Capture Mode:
      gmo
  Capture: enter 'j' OR
           followed by a timer (eg. jx5000, capture after 5 seconds) OR
           followed by multishot count (eg. j:6, capture 6 images)
           timer/multihot values are optional, capture defaults to single shot with timer=0s
  Start Recording: enter '1'
  Stop Recording: enter '0'
  Force IDR frame on video encoder: enter 'f' 
  Set Container file type (0=mp4 1=3gp 2=avi):
      k:<val> eg k:0
  Get Container file type:
      gk
  Set Image Capture Resolution(only in image mode):
      icr:<val> eg icr:3
          (0) : 176 x 144
          (1) : 320 x 240
          (2) : 480 x 480
          (3) : 640 x 480
          (4) : 800 x 600
          (5) : 720 x 480
          (6) : 1280 x 720
          (7) : 1280 x 960
          (8) : 1600 x 1200
          (9) : 2048 x 1536
          (10) : 2240 x 1680
          (11) : 2560 x 1920
          (12) : 4096 x 3072 (only for t11x)
  Get Image Capture Resolution:
      gicr
  Set Video Capture Resolution(only in video mode):
      vcr:<val> eg vcr:3
      * H.264/MPEG-4/VP8 (supports "MP4/3GP/AVI" container)
          <0> : 176 x 144
          <1> : 320 x 240
          <3> : 640 x 480
          <5> : 720 x 480
          <6> : 1280 x 720
          <7> : 1920 x 1080
      * H.263 (supports "MP4/3GP/AVI" container)
          <0> : 176 x 144 (QCIF)
          <2> : 352 x 288 (CIF)
          <4> : 704 x 576 (4CIF)
      NOTE: USB camera capture only supports 176x144 resolution for H263 encode
  Get Video Capture Resolution:
      gvcr
  Set Capture Device Node:
      cdn:<val> Val = 0,1,2 (Capture device node)  
  Get Capture Device Node:
      gcdn
  Set Video Encoder:
      ve:<val>, val= 0(h264), 1(vp8), 2(mpeg4 software), 3(h263 software)
 Set Preview Sink Type: 
  --svs =nv_omx_videosink  for overlay, --svs =nv_omx_hdmi_videosink  for hdmi overlay, --svs =xvimagesink, --svs=ximagesink for ximagesink 
  Get Video Encoder:
      gve

Capture snapshots

Sometimes when the board has been just flashed you need to run reboot in order to run nvgstcapture-1.0 successfully. To take snapshots:

nvgstcapture-1.0
#then press j to take an snapshot
q 

Resolutions available to create snapshot:

  (2) : 640x480
  (3) : 1280x720
  (4) : 1920x1080
  (5) : 2104x1560
  (6) : 2592x1944
  (7) : 2616x1472
  (8) : 3840x2160
  (9) : 3896x2192
  (10): 4208x3120
 nvgstcapture-1.0 -m 1 --image-res=10

In general the application is consuming 70% of one core. It is always displaying video this could affect.

Capture and record video

nvgstcapture-1.0 -m 2 # set the mode in video
1 # start video recording
2 # take a snapshot
0 # stop video recording

In case of video it should support:

          (2) : 640x480
          (3) : 1280x720
          (4) : 1920x1080
          (5) : 2104x1560
          (6) : 2592x1944
          (7) : 2616x1472
          (8) : 3840x2160
          (9) : 3896x2192

However, it is not possible to set other value than 2,3,4,5 up to 9.

nvgstcapture-1.0 -m 2 --video-res=5

top

top - 21:50:33 up  1:56,  6 users,  load average: 0.99, 0.65, 0.54
Tasks: 219 total,   1 running, 218 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.7 us, 12.9 sy,  0.0 ni, 76.2 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem:   3946312 total,  2341908 used,  1604404 free,    20932 buffers
KiB Swap:        0 total,        0 used,        0 free.   378484 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                       
  757 root      20   0  830496  84904  21504 S  61.0  2.2   8:13.76 nvcamera-daemon                                                                               
 4380 ubuntu    20   0  191340  10256   5004 S  12.6  0.3   0:02.83 nvgstcapture-1. 

There is a daemon and a pipeline. Interesting...

While recording the pipeline (nvgstcapture) increases the ARM load to 10% and then it goes down again, even with the highest resolution. It will be interesting to understand what is the daemon doing.

Gstreamer pipelines

Recall that this board is a quad core A57 with some hardware units dedicated to encode/decode. Plugins accelerated by Hardware available:

 gst-inspect-1.0 | grep omx
omx:  omxmpeg4videodec: OpenMAX MPEG4 Video Decoder
omx:  omxh264dec: OpenMAX H.264 Video Decoder
omx:  omxh265dec: OpenMAX H.265 Video Decoder
omx:  omxvp8dec: OpenMAX VP8 Video Decoder
omx:  omxvp9dec: OpenMAX VP9 Video Decoder
omx:  omxmpeg2videodec: OpenMAX MPEG2 Video Decoder
omx:  omxwmvdec: OpenMAX WMV Video Decoder
omx:  omxaacdec: OpenMax AAC decoder
omx:  omxmpegaudiodec: OpenMax MP3/MP2 decoder
omx:  omxh264enc: OpenMAX H.264 Video Encoder
omx:  omxh265enc: OpenMAX H.265 Video Encoder
omx:  omxvp8enc: OpenMAX VP8 Video Encoder
omx:  nvhdmioverlaysink: OpenMax HDMI Video Sink
omx:  nvoverlaysink: OpenMax Video Sink

Nvidia plugins:

gst-inspect-1.0 | grep nv
nvcamera:  nvcamerasrc: Nvidia Camera Source
nvcamera:  nvtee: NvTee
omx:  nvhdmioverlaysink: OpenMax HDMI Video Sink
omx:  nvoverlaysink: OpenMax Video Sink
nvjpeg:  nvjpegenc: JPEG image encoder
nvjpeg:  nvjpegdec: JPEG image decoder
nvvideosink:  nvvideosink: nVidia Video Sink
nvvidconv:  nvvidconv: NvVidConv Plugin
eglglessink:  nveglglessink: EGL/GLES vout Sink
nveglstreamsrc:  nveglstreamsrc: nVidia EGL Stream

Measure ARM load

There is a tool to measure the ARM load created by NVIDIA for the tegra X1. This tool seems to give better statistics of the ARM cores than top. In the remaining sections of this wiki the ARM load reported by top is included for reference to the reader but tegrastats should contain the real load.

Run it with sudo to get more information.

ubuntu@tegra-ubuntu:~$ ./tegrastats 
RAM 162/3854MB (lfb 845x4MB) SWAP 0/0MB (cached 0MB) cpu [0%,0%,0%,0%]@-1 VDE 0 EDP limit 0
RAM 162/3854MB (lfb 845x4MB) SWAP 0/0MB (cached 0MB) cpu [0%,0%,0%,0%]@-1 VDE 0 EDP limit 0
RAM 162/3854MB (lfb 845x4MB) SWAP 0/0MB (cached 0MB) cpu [0%,1%,0%,0%]@-1 VDE 0 EDP limit 0
RAM 162/3854MB (lfb 845x4MB) SWAP 0/0MB (cached 0MB) cpu [4%,2%,1%,0%]@-1 VDE 0 EDP limit 0
RAM 162/3854MB (lfb 845x4MB) SWAP 0/0MB (cached 0MB) cpu [1%,1%,1%,0%]@-1 VDE 0 EDP limit 0

EMC – memory controller 
AVP – audio/video processor
VDE – video decoder engine
GR3D – GPU

Tegra X1 ARM load

This table tries to summarize the ARM load of different uses cases of the tegra X1 using gstreamer and the capture plugin. The second column has the ARM load of the pipeline measured in one instant, using that value and the known value for the capture process the third column estimates the load of the encoding or display process (subtract the capture load). The load represent a global load of the system (4 ARMs) just to give an idea to the reader of the system capabilities.

ARM load measured with tegrastats for Capture and encoding capabilities
Feature Pipeline load (%) Feature load (%)
Capture 11.25 11.25
Capture-Display 18 6.75
H264 Encoding 20 8.75
VP8 Encoding 19.25 8
H265 Encoding 13.75 2.75
<...>

NVMM memory

In order to get the best performance from the gstreamer elements that use the HW accelerators they have to receive something called NVMM memory, which we could imagine is contiguous and aligned memory ('nvbuffer', 'nv_buffer', 'nvmm'). At this point it is not clear what that memory is.

https://devtalk.nvidia.com/default/topic/903438/jetson-tx1/hw-accelerated-jpeg-encoding-/?offset=13#4839080
https://devtalk.nvidia.com/default/topic/901337/post/4747186/#4747186

There is a plugin to convert video/x-raw to video/x-raw(memory:NVMM)

https://devtalk.nvidia.com/default/topic/911535/jetson-tx1/gstreamer-nvvidconv-plugin-not-working/

Capture

In order to capture there are two ways: Using v4l2 or using nvcamerasrc.

nvcamerasrc

This plugin was created by nvidia and it has access to the ISP that helps converting from bayer to yuv suitable for the video encoders. However, it seems that its usage is limited to the ov5693 sensor until NVIDIA releases its source code or until it adds support to v4l2 to use the ISP.

gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! fakesink silent=false -v

When the pipeline is launched a daemon appears on top nvcamera-daemon, one question about the daemon was posted on the Nvidia Embedded Computing Forum

ARM consumption: 45% of one core (4 cores available 400% - this means something like 11.25% of total)

RAM 1295/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [0%,0%,0%,0%]@-1 VDE 0 EDP limit 0
RAM 1296/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [2%,10%,5%,4%]@-1 VDE 0 EDP limit 0
RAM 1296/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [6%,7%,7%,5%]@-1 VDE 0 EDP limit 0
RAM 1311/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,9%,41%,6%]@-1 VDE 0 EDP limit 0 ->Start
RAM 1311/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,11%,12%,7%]@-1 VDE 0 EDP limit 0
RAM 1311/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,16%,9%,14%]@-1 VDE 0 EDP limit 0
RAM 1311/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,12%,15%,12%]@-1 VDE 0 EDP limit 0
RAM 1311/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [11%,16%,12%,9%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,12%,11%,16%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [11%,11%,11%,10%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,11%,8%,16%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,14%,11%,12%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,12%,7%,11%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [11%,11%,12%,8%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,9%,14%,18%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [16%,14%,14%,10%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,11%,12%,10%]@-1 VDE 0 EDP limit 0
RAM 1312/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,8%,13%,14%]@-1 VDE 0 EDP limit 0
RAM 1302/3854MB (lfb 426x4MB) SWAP 0/0MB (cached 0MB) cpu [8%,6%,17%,12%]@-1 VDE 0 EDP limit 0
 589 root      20   0 1007120 104416  24048 S  47.2  2.6   4:02.80 nvcamera-daemon                                                             
 5557 ubuntu    20   0  157448   4228   2688 S   2.9  0.1   0:00.45 gst-launch-1.0    

The nvvidconv is required normally to convert the memory type, however, on this case it is used to flip the image. Removing the flip from the pipeline saves like 5% of ARM load.

nvcamerasrc exposes several properties to control the ISP

  wbmode              : White balance affects the color temperature of the photo
                        flags: readable, writable
                        Enum "GstNvCamWBMode" Default: 1, "auto"
                           (0): off              - GST_NVCAM_WB_MODE_OFF
                           (1): auto             - GST_NVCAM_WB_MODE_AUTO
                           (2): incandescent     - GST_NVCAM_WB_MODE_INCANDESCENT
                           (3): fluorescent      - GST_NVCAM_WB_MODE_FLUORESCENT
                           (4): warm-fluorescent - GST_NVCAM_WB_MODE_WARM_FLUORESCENT
                           (5): daylight         - GST_NVCAM_WB_MODE_DAYLIGHT
                           (6): cloudy-daylight  - GST_NVCAM_WB_MODE_CLOUDY_DAYLIGHT
                           (7): twilight         - GST_NVCAM_WB_MODE_TWILIGHT
                           (8): shade            - GST_NVCAM_WB_MODE_SHADE
                           (9): manual           - GST_NVCAM_WB_MODE_MANUAL
  scene-mode          : property to set scene mode
                        flags: readable, writable
                        Enum "GstNvCamSceneMode" Default: 0, "face-priority"
                           (0): face-priority    - GST_NVCAM_SCENE_MODE_FACE_PRIORITY
                           (1): action           - GST_NVCAM_SCENE_MODE_ACTION
                           (2): portrait         - GST_NVCAM_SCENE_MODE_PORTRAIT
                           (3): landscape        - GST_NVCAM_SCENE_MODE_LANDSCAPE
                           (4): night            - GST_NVCAM_SCENE_MODE_NIGHT
                           (5): night-portrait   - GST_NVCAM_SCENE_MODE_NIGHT_PORTRAIT
                           (6): theatre          - GST_NVCAM_SCENE_MODE_THEATRE
                           (7): beach            - GST_NVCAM_SCENE_MODE_BEACH
                           (8): snow             - GST_NVCAM_SCENE_MODE_SNOW
                           (9): sunset           - GST_NVCAM_SCENE_MODE_SUNSET
                           (10): steady-photo     - GST_NVCAM_SCENE_MODE_STEADYSHOT
                           (11): fireworks        - GST_NVCAM_SCENE_MODE_FIREWORKS
                           (12): sports           - GST_NVCAM_SCENE_MODE_SPORTS
                           (13): party            - GST_NVCAM_SCENE_MODE_PARTY
                           (14): candle-light     - GST_NVCAM_SCENE_MODE_CANDLE_LIGHT
                           (15): barcode          - GST_NVCAM_SCENE_MODE_BARCODE
  color-effect        : property to set color effect
                        flags: readable, writable
                        Enum "GstNvCamColorEffectMode" Default: 1, "off"
                           (1): off              - GST_NVCAM_COLOR_EFFECT_OFF
                           (2): mono             - GST_NVCAM_COLOR_EFFECT_MONO
                           (3): negative         - GST_NVCAM_COLOR_EFFECT_NEGATIVE
                           (4): solarize         - GST_NVCAM_COLOR_EFFECT_SOLARIZE
                           (5): sepia            - GST_NVCAM_COLOR_EFFECT_SEPIA
                           (6): posterize        - GST_NVCAM_COLOR_EFFECT_POSTERIZE
                           (7): aqua             - GST_NVCAM_COLOR_EFFECT_AQUA
  auto-exposure       : property to set auto exposure mode
                        flags: readable, writable
                        Enum "GstNvCamAutoExposureMode" Default: 2, "on"
                           (1): off              - GST_NVCAM_AUTO_EXPOSURE_OFF
                           (2): on               - GST_NVCAM_AUTO_EXPOSURE_ON
                           (3): OnAutoFlash      - GST_NVCAM_AUTO_EXPOSURE_AUTOFLASH
                           (4): OnAlwaysFlash    - GST_NVCAM_AUTO_EXPOSURE_ALWAYSFLASH
                           (5): OnFlashRedEye    - GST_NVCAM_AUTO_EXPOSURE_AUTO_RED_EYE
  flash               : property to set flash mode
                        flags: readable, writable
                        Enum "GstNvCamFlashMode" Default: 0, "off"
                           (0): off              - GST_NVCAM_FLASH_OFF
                           (1): on               - GST_NVCAM_FLASH_ON
                           (2): torch            - GST_NVCAM_FLASH_TORCH
                           (3): auto             - GST_NVCAM_FLASH_AUTO
  flicker             : property to select flicker detection and avoidance mode
                        flags: readable, writable
                        Enum "GstNvCamFlickerMode" Default: 3, "auto"
                           (0): off              - GST_NVCAM_FLICKER_OFF
                           (1): 50Hz             - GST_NVCAM_FLICKER_50HZ
                           (2): 60Hz             - GST_NVCAM_FLICKER_60HZ
                           (3): auto             - GST_NVCAM_FLICKER_AUTO
  contrast            : property to adjust contrast value
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  saturation          : property to adjust saturation value
                        flags: readable, writable
                        Float. Range:               0 -               2 Default:               1 
  tnr-strength        : property to adjust temporal noise reduction strength
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  tnr-mode            : property to select temporal noise reduction mode
                        flags: readable, writable
                        Enum "GstNvCamTNRMode" Default: 0, "NoiseReduction_Off"
                           (0): NoiseReduction_Off - GST_NVCAM_NR_OFF
                           (1): NoiseReduction_Fast - GST_NVCAM_NR_FAST
                           (2): NoiseReduction_HighQuality - GST_NVCAM_NR_HIGHQUALITY
  edge-enhancement    : property to adjust edge enhnacement value
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:              -1 
  trigger             : property to fire pre capture trigger
                        flags: readable, writable
                        Enum "GstNvCamCaptureTrigger" Default: 0, ""
                           (1): idle             - GST_NVCAM_TRIGGER_IDLE
                           (2): start            - GST_NVCAM_TRIGGER_START
  intent              : property to select capture intent
                        flags: readable, writable
                        Enum "GstNvCamCaptureIntent" Default: 1, "preview"
                           (1): preview          - GST_NVCAM_INTENT_PREVIEW
                           (2): still            - GST_NVCAM_INTENT_STILL
                           (3): video            - GST_NVCAM_INTENT_VIDEO
                           (4): video-snapshot   - GST_NVCAM_INTENT_VIDEO_SNAPSHOT
                           (5): zsl              - GST_NVCAM_INTENT_ZSL
  sensor-id           : Set the id of camera sensor to use. Default 0.)
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 0 
  enable-exif         : Return Exif data with captured frame
                        flags: readable, writable
                        Boolean. Default: false
  enable-meta         : Return Sensor meta data with captured frame
                        flags: readable, writable
                        Boolean. Default: false
  aeRegion            : Property to set region of interest for auto exposure.
			 Use GArray, with values of ROI coordinates (top,left,bottom,right)
			 and weight in that order, to set the property.
                        flags: readable, writable
                        Boxed pointer of type "GArray"
  wbRegion            : Property to set region of interest for white balance.
			 Use GArray, with values of ROI coordinates (top,left,bottom,right)
			 and weight in that order, to set the property.
                        flags: readable, writable
                        Boxed pointer of type "GArray"
  fpsRange            : Property to set FPS range.
			 Use string with values of FPS Range (low, high)
			 in that order, to set the property.
                        flags: readable, writable
                        String. Default: null
  exposure-time       : Property to adjust exposure time.
			 This property will have effect only when Exposure mode is Off.
                        flags: readable, writable
                        Float. Range:               0 -    3.402823e+38 Default:           0.033 
  wbManualMode        : Property to set manual mode of White balance.
			 This property will have effect only when wbMode is manual.
                        flags: readable, writable
                        Enum "GstNvCamWBManualMode" Default: 0, "off"
                           (0): off              - GST_NVCAM_WB_MANUAL_OFF
                           (1): reset            - GST_NVCAM_WB_MANUAL_RESET
                           (2): restore          - GST_NVCAM_WB_MANUAL_RESTORE
                           (3): external-gains   - GST_NVCAM_WB_MANUAL_EXTGAINS
  wbGains             : Property to set external white balance Gains.
			 Use GArray, with values of WB gains (R, GR, GB, B)
			 in that order, to set the property.
			 This will have effect only when wbMode is manual.
                        flags: readable, writable
                        Boxed pointer of type "GArray"
  stitchConfig        : Property to set stitching windows.
			 Use GArray, with values of sensor id and
			 window coordinates (sensor-id,top,left,bottom,right)

                        flags: readable, writable
                        Boxed pointer of type "GArray"
  callback            : Property to set user callback. It is required if user wants 
			 to change capture settings per frame. Provide address 
			 of GstNvCamSrcCallbacks type struct as value. 

                        flags: writable
                        Pointer. Write only
  dump-bayer          : Dump raw bayer data in still capture mode.
                        flags: readable, writable
                        Boolean. Default: false
  num-sensors         : Property to get number of sensors.
			 Returns GArray containing number of sensors {usb,csi}

                        flags: readable
                        Boxed pointer of type "GArray"
  orientation         : Sensor orientation value. This must be accessed in READY state.
                        flags: readable
                        Integer. Range: 0 - 360 Default: -1 


References

https://devtalk.nvidia.com/default/topic/920253/jetson-tx1/how-to-add-another-csi-camera-/
https://devtalk.nvidia.com/default/topic/918966/jetson-tx1/nvcamerasrc-source-code-request-to-support-a-sensor-other-than-the-ov5693-/

v4l2src

Starting on L4T R23.2 there is a /dev/video0 node to capture, however, this node will give you frames in bayer which are NOT suitable to encode because it grabs frames directly from the ov5693 camera without using the ISP.

https://devtalk.nvidia.com/default/topic/923984/ov5693-through-v4l2-interface-not-working-in-23-2/
https://devtalk.nvidia.com/default/topic/914042/jetson-tx1/camera-name-in-dev-folder/
https://devtalk.nvidia.com/default/topic/904169/jetson-tx1/live-camera-h264enc-does-not-work/
https://devtalk.nvidia.com/default/topic/927971/jetson-tx1/isp-apis-on-tx1/
https://devtalk.nvidia.com/default/topic/934354/jetson-tx1/typical-approaches-to-test-camera-be-only-applicable-for-l4t-r23-2-jetson-tx1-release-/

It works also with the UVC driver. David suggested the creation of a nvisp element to convert bayer to yuv

Capture and Display

nvcamerasrc

gst-launch-1.0 nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e

ARM consumption: 72% of one core (4 cores available 400% - this means something like 18% of total). However, only the capture is 11.25% so the display consumes 6.75%

RAM 1146/3854MB (lfb 470x4MB) SWAP 0/0MB (cached 0MB) cpu [2%,4%,5%,0%]@-1 VDE 0 EDP limit 0
RAM 1146/3854MB (lfb 470x4MB) SWAP 0/0MB (cached 0MB) cpu [3%,3%,5%,0%]@-1 VDE 0 EDP limit 0
RAM 1169/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [25%,32%,21%,25%]@-1 VDE 0 EDP limit 0 ->Start capture
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [17%,15%,14%,12%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [20%,16%,13%,18%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,15%,18%,16%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,18%,21%,16%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,20%,20%,11%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [23%,20%,20%,16%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [12%,10%,16%,11%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,20%,25%,15%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [20%,14%,18%,20%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [16%,15%,19%,12%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,21%,21%,19%]@-1 VDE 0 EDP limit 0
RAM 1170/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,14%,14%,15%]@-1 VDE 0 EDP limit 0
RAM 1150/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [6%,2%,4%,7%]@-1 VDE 0 EDP limit 0 ->Stop capture
RAM 1150/3854MB (lfb 469x4MB) SWAP 0/0MB (cached 0MB) cpu [0%,0%,6%,0%]@-1 VDE 0 EDP limit 0
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                        
  589 root      20   0  797332  93308  22672 S  52.7  2.4   1:35.02 nvcamera-daemon                                
 4520 ubuntu    20   0  184444   5812   3712 S   2.9  0.1   0:00.46 gst-launch-1.0  

H264 encoding

videotestsrc

ARM load: 175% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.

gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=test.h264 -e

VLC can play the file correctly at 30fps and reports as codec: H264-MPEG-4 AVC (part 10) (h264). The fan on top of the SoC didn't turn on.

nvcamerasrc

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxh264enc bitrate=8000000 \
! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=test.h264 -e

Codecvisa reports bitrate on 7.96Mbps when variable mode is selected on the encoder.

ARM consumption ~80% of one core (4 cores available 400% - this means something like 20% of total). However, only the capture is 11.25% so the h264 consumes 8.75%

RAM 1243/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [4%,7%,4%,5%]@-1 VDE 0 EDP limit 0
RAM 1247/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [6%,1%,5%,8%]@-1 VDE 0 EDP limit 0
RAM 1272/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [33%,24%,16%,34%]@-1 VDE 0 EDP limit 0
RAM 1273/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [17%,12%,13%,13%]@-1 VDE 0 EDP limit 0
RAM 1273/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,12%,5%,13%]@-1 VDE 0 EDP limit 0
RAM 1273/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [23%,15%,16%,11%]@-1 VDE 0 EDP limit 0
RAM 1273/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [23%,16%,11%,22%]@-1 VDE 0 EDP limit 0
RAM 1273/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [13%,8%,11%,16%]@-1 VDE 0 EDP limit 0
RAM 1277/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [24%,24%,20%,22%]@-1 VDE 0 EDP limit 0
RAM 1277/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,19%,19%,21%]@-1 VDE 0 EDP limit 0
RAM 1278/3854MB (lfb 442x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,16%,16%,19%]@-1 VDE 0 EDP limit 0
RAM 1278/3854MB (lfb 441x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,13%,17%,20%]@-1 VDE 0 EDP limit 0
RAM 1279/3854MB (lfb 441x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,23%,15%,20%]@-1 VDE 0 EDP limit 0
RAM 1279/3854MB (lfb 440x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,22%,18%,14%]@-1 VDE 0 EDP limit 0
RAM 1279/3854MB (lfb 440x4MB) SWAP 0/0MB (cached 0MB) cpu [16%,16%,17%,13%]@-1 VDE 0 EDP limit 0
RAM 1279/3854MB (lfb 440x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,18%,24%,23%]@-1 VDE 0 EDP limit 0
RAM 1279/3854MB (lfb 440x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,13%,7%,16%]@-1 VDE 0 EDP limit 0
RAM 1279/3854MB (lfb 439x4MB) SWAP 0/0MB (cached 0MB) cpu [25%,20%,26%,21%]@-1 VDE 0 EDP limit 0
RAM 1280/3854MB (lfb 439x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,15%,11%,13%]@-1 VDE 0 EDP limit 0
RAM 1277/3854MB (lfb 439x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,12%,9%,23%]@-1 VDE 0 EDP limit 0
RAM 1255/3854MB (lfb 439x4MB) SWAP 0/0MB (cached 0MB) cpu [13%,7%,3%,4%]@-1 VDE 0 EDP limit 0
RAM 1255/3854MB (lfb 439x4MB) SWAP 0/0MB (cached 0MB) cpu [4%,4%,6%,0%]@-1 VDE 0 EDP limit 0
RAM 1255/3854MB (lfb 439x4MB) SWAP 0/0MB (cached 0MB) cpu [10%,0%,3%,0%]@-1 VDE 0 EDP limit 0
  589 root      20   0  869160 101880  23112 S  45.0  2.6   2:37.29 nvcamera-daemon                                
 4854 ubuntu    20   0  264964   6932   4028 S   6.8  0.2   0:01.05 gst-launch-1.0            

The encoder exposes the following parameters

  control-rate        : Bitrate control method
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOMXVideoEncControlRate" Default: 1, "variable"
                           (0): disable          - Disable
                           (1): variable         - Variable
                           (2): constant         - Constant
                           (3): variable-skip-frames - Variable Skip Frames
                           (4): constant-skip-frames - Constant Skip Frames
  bitrate             : Target bitrate
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4000000 
  quant-i-frames      : Quantization parameter for I-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-p-frames      : Quantization parameter for P-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-b-frames      : Quantization parameter for B-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  quality-level       : Encoding quality-level
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 2 Default: 0 
  low-latency         : Whether or not set low latency attribute of encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshInterval: Set SliceIntraRefreshInterval
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 60 
  bit-packetization   : Whether or not Packet size is based upon Number Of bits
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  vbv-size            : virtual buffer size = vbv-size * (bitrate/fps)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 30 Default: 10 
  temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOmxVideoEncTemporalTradeoffType" Default: 0, "Do not drop frames"
                           (0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES
                           (1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES
                           (2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES
                           (3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES
                           (4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES
  insert-sps-pps      : Insert H.264 SPS, PPS at every IDR frame
                        flags: readable, writable
                        Boolean. Default: false
  no-B-Frames         : Do not use B Frames while encoding
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  slice-header-spacing: Slice Header Spacing number of macroblocks/bits in one packet
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Long. Range: 0 - 4294967295 Default: 0 
  profile             : Set profile for encode
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOmxVideoEncProfileType" Default: 1, "baseline"
                           (1): baseline         - GST_OMX_VIDENC_BASELINE_PROFILE
                           (2): main             - GST_OMX_VIDENC_MAIN_PROFILE
                           (8): high             - GST_OMX_VIDENC_HIGH_PROFILE

Dual H264 encoding

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \
streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testA.h264 \
streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testB.h264


ARM consumption ~81% of one core (4 cores available 400% - this means something like 20.25% of total). However, only the capture is 11.25% so the dual h264 consumes 9%

RAM 948/3854MB (lfb 539x4MB) SWAP 0/0MB (cached 0MB) cpu [2%,0%,0%,4%]@102 EMC 7%@408 AVP 57%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 952/3854MB (lfb 539x4MB) SWAP 0/0MB (cached 0MB) cpu [9%,9%,0%,5%]@1734 EMC 2%@1600 AVP 56%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 969/3854MB (lfb 539x4MB) SWAP 0/0MB (cached 0MB) cpu [24%,24%,35%,27%]@1224 EMC 15%@1065 AVP 41%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 970/3854MB (lfb 539x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,19%,19%,19%]@307 EMC 29%@665 AVP 63%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 970/3854MB (lfb 539x4MB) SWAP 0/0MB (cached 0MB) cpu [23%,18%,14%,14%]@1224 EMC 19%@1065 AVP 69%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 970/3854MB (lfb 538x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,19%,19%,17%]@403 EMC 32%@665 AVP 69%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 538x4MB) SWAP 0/0MB (cached 0MB) cpu [25%,16%,24%,18%]@1224 EMC 19%@1065 AVP 62%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 537x4MB) SWAP 0/0MB (cached 0MB) cpu [24%,19%,19%,21%]@307 EMC 32%@665 AVP 69%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 537x4MB) SWAP 0/0MB (cached 0MB) cpu [25%,21%,20%,18%]@307 EMC 19%@1065 AVP 69%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 536x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,24%,28%,25%]@1224 EMC 19%@1065 AVP 62%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 536x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,19%,21%,18%]@518 EMC 32%@665 AVP 62%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 535x4MB) SWAP 0/0MB (cached 0MB) cpu [23%,22%,24%,22%]@518 EMC 32%@665 AVP 69%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 535x4MB) SWAP 0/0MB (cached 0MB) cpu [24%,25%,22%,19%]@403 EMC 32%@665 AVP 69%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 534x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,17%,17%,25%]@1224 EMC 19%@1065 AVP 62%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 534x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,23%,21%,18%]@614 EMC 32%@665 AVP 62%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 533x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,22%,21%,21%]@1224 EMC 19%@1065 AVP 55%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 533x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,21%,19%,21%]@1224 EMC 19%@1065 AVP 70%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 533x4MB) SWAP 0/0MB (cached 0MB) cpu [17%,21%,16%,22%]@1224 EMC 19%@1065 AVP 63%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 532x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,21%,14%,27%]@307 EMC 32%@665 AVP 63%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 971/3854MB (lfb 532x4MB) SWAP 0/0MB (cached 0MB) cpu [16%,17%,20%,17%]@307 EMC 32%@665 AVP 71%@12 VDE 0 GR3D 0%

H264 UDP Streaming

server:

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \
! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh266enc low-latency=1 control-rate=2 bitrate=4000000 ! 'video/x-h266, stream-format=(string)byte-stream' ! \
h266parse ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

PC client:

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264 ! xvimagesink sync=false async=false -e

h264 parse missing

In case gstreamer complains when running the pipeline for missing the h264parse plugin, run the following command to install it:

sudo apt-get install gstreamer1.0-plugins-bad

Latency

  • 1080p30
  • H264
  • Bitrate 4Mbps
  • Latency=172ms

VP8 encoding

videotestsrc

ARM load: 156% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.

gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxvp8enc ! qtmux ! filesink location=test.mp4 -e

VLC can play the file correctly at 30fps and reports as codec: Google/On2's VP8 Video (VP80). The fan on top of the SoC didn't turn on.

nvcamerasrc


gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxvp8enc bitrate=8000000 ! qtmux ! filesink location=test.mp4 -e

VLC reports bitrate on ~8Mbps.

ARM consumption ~77% of one core (4 cores available 400% - this means something like 19.25% of total). However, only the capture is 11.25% so the VP8 consumes 8%

RAM 1288/3854MB (lfb 430x4MB) SWAP 0/0MB (cached 0MB) cpu [11%,1%,7%,2%]@-1 VDE 0 EDP limit 0
RAM 1292/3854MB (lfb 430x4MB) SWAP 0/0MB (cached 0MB) cpu [11%,23%,4%,4%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 430x4MB) SWAP 0/0MB (cached 0MB) cpu [33%,20%,45%,19%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 430x4MB) SWAP 0/0MB (cached 0MB) cpu [29%,24%,16%,22%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 430x4MB) SWAP 0/0MB (cached 0MB) cpu [24%,20%,25%,21%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 430x4MB) SWAP 0/0MB (cached 0MB) cpu [29%,26%,26%,21%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 430x4MB) SWAP 0/0MB (cached 0MB) cpu [30%,25%,24%,26%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 429x4MB) SWAP 0/0MB (cached 0MB) cpu [24%,23%,25%,21%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 429x4MB) SWAP 0/0MB (cached 0MB) cpu [17%,18%,26%,17%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 429x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,18%,20%,20%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 429x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,16%,14%,16%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 428x4MB) SWAP 0/0MB (cached 0MB) cpu [20%,16%,17%,19%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 428x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,9%,14%,19%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 428x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,12%,14%,17%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 428x4MB) SWAP 0/0MB (cached 0MB) cpu [17%,15%,16%,18%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 427x4MB) SWAP 0/0MB (cached 0MB) cpu [23%,18%,23%,15%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 427x4MB) SWAP 0/0MB (cached 0MB) cpu [12%,18%,18%,17%]@-1 VDE 0 EDP limit 0
RAM 1314/3854MB (lfb 427x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,18%,10%,14%]@-1 VDE 0 EDP limit 0
RAM 1292/3854MB (lfb 427x4MB) SWAP 0/0MB (cached 0MB) cpu [12%,10%,11%,9%]@-1 VDE 0 EDP limit 0
  589 root      20   0  948716 106776  23616 S  52.7  2.7   3:38.42 nvcamera-daemon                                
 5258 ubuntu    20   0  264572   7300   4188 S   8.8  0.2   0:01.44 gst-launch-1.0       

The encoder exposes the following parameters

  bitrate             : Target bitrate
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4000000 
  quant-i-frames      : Quantization parameter for I-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-p-frames      : Quantization parameter for P-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-b-frames      : Quantization parameter for B-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  quality-level       : Encoding quality-level
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 2 Default: 0 
  low-latency         : Whether or not set low latency attribute of encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshInterval: Set SliceIntraRefreshInterval
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 60 
  bit-packetization   : Whether or not Packet size is based upon Number Of bits
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  vbv-size            : virtual buffer size = vbv-size * (bitrate/fps)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 30 Default: 10 
  temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOmxVideoEncTemporalTradeoffType" Default: 0, "Do not drop frames"
                           (0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES
                           (1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES
                           (2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES
                           (3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES
                           (4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES

H265 encoding

VLC with H265 support

In order to support h265 you need VLC 2.2 in your computer to play the videos

sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get install vlc vlc-plugin-*

videotestsrc

ARM load: 156% but only videotestsrc consumes 100%. According to the NVIDIA forum when using videotestsrc the system is also doing memory copies to feed the encoder with NVMM memory buffers so that loads more the ARM.

gst-launch-1.0 videotestsrc num-buffers=150 ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh265enc ! filesink location=test.h265 -e

VLC can play the file correctly at 30fps and reports as codec: MPEG-H Part2/HEVC (H.265)(hevc). The fan on top of the SoC didn't turn on.

nvcamerasrc

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! omxh265enc ! filesink location=test.h265 -e

Codecvisa reports bitrate on 7.96Mbps when variable mode is selected on the encoder.

ARM consumption ~55% of one core (4 cores available 400% - this means something like 13.75% of total). However, only the capture is 11.25% so the h264 consumes 2.75%

RAM 1329/3854MB (lfb 418x4MB) SWAP 0/0MB (cached 0MB) cpu [5%,7%,4%,8%]@-1 VDE 0 EDP limit 0
RAM 1329/3854MB (lfb 418x4MB) SWAP 0/0MB (cached 0MB) cpu [4%,5%,3%,2%]@-1 VDE 0 EDP limit 0
RAM 1342/3854MB (lfb 418x4MB) SWAP 0/0MB (cached 0MB) cpu [35%,19%,7%,8%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 417x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,19%,25%,35%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 417x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,25%,18%,20%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 417x4MB) SWAP 0/0MB (cached 0MB) cpu [17%,18%,18%,17%]@-1 VDE 0 EDP limit 0
RAM 1351/3854MB (lfb 417x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,23%,26%,22%]@-1 VDE 0 EDP limit 0
RAM 1351/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,24%,11%,13%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [22%,24%,16%,16%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [20%,22%,14%,17%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,14%,15%,17%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [14%,19%,19%,12%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [28%,21%,17%,16%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,13%,21%,17%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [25%,12%,13%,23%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 416x4MB) SWAP 0/0MB (cached 0MB) cpu [20%,21%,17%,21%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,16%,18%,17%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,16%,18%,17%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [15%,12%,9%,16%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [16%,13%,16%,13%]@-1 VDE 0 EDP limit 0
RAM 1351/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,12%,10%,15%]@-1 VDE 0 EDP limit 0
RAM 1351/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [11%,18%,18%,14%]@-1 VDE 0 EDP limit 0
RAM 1351/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [19%,11%,16%,14%]@-1 VDE 0 EDP limit 0
RAM 1350/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [13%,16%,11%,9%]@-1 VDE 0 EDP limit 0
RAM 1334/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [6%,6%,14%,3%]@-1 VDE 0 EDP limit 0
RAM 1334/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [3%,0%,2%,6%]@-1 VDE 0 EDP limit 0
RAM 1334/3854MB (lfb 415x4MB) SWAP 0/0MB (cached 0MB) cpu [8%,4%,7%,0%]@-1 VDE 0 EDP limit 0
  589 root      20   0 1043228 110648  24268 S  46.4  2.8   4:22.56 nvcamera-daemon                                
 5715 ubuntu    20   0  274224   7420   4212 S   7.7  0.2   0:01.72 gst-launch-1.0                       

The encoder exposes the following parameters

  control-rate        : Bitrate control method
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOMXVideoEncControlRate" Default: 1, "variable"
                           (0): disable          - Disable
                           (1): variable         - Variable
                           (2): constant         - Constant
                           (3): variable-skip-frames - Variable Skip Frames
                           (4): constant-skip-frames - Constant Skip Frames
  bitrate             : Target bitrate
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4000000 
  quant-i-frames      : Quantization parameter for I-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-p-frames      : Quantization parameter for P-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-b-frames      : Quantization parameter for B-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  quality-level       : Encoding quality-level
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 2 Default: 0 
  low-latency         : Whether or not set low latency attribute of encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshInterval: Set SliceIntraRefreshInterval
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 60 
  bit-packetization   : Whether or not Packet size is based upon Number Of bits
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  vbv-size            : virtual buffer size = vbv-size * (bitrate/fps)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 30 Default: 10 
  temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOmxVideoEncTemporalTradeoffType" Default: 0, "Do not drop frames"
                           (0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES
                           (1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES
                           (2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES
                           (3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES
                           (4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES

Dual H265 encoding

gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \
streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testA.h264 \
streams. ! omxh264enc bitrate=8000000 ! 'video/x-h264, stream-format=(string)byte-stream' ! filesink location=testB.h264
gst-launch-1.0 -e nvcamerasrc fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! tee name=streams \
streams. ! omxh265enc bitrate=8000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! filesink location=testA.h265 \
streams. ! omxh265enc bitrate=8000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! filesink location=testB.h265

ARM consumption ~61% of one core (4 cores available 400% - this means something like 15.25% of total). However, only the capture is 11.25% so the dual h265 consumes 4%

RAM 950/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [0%,0%,0%,0%]@1224 EMC 3%@1065 AVP 64%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 950/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [2%,0%,0%,0%]@102 EMC 9%@408 AVP 64%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 951/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [0%,6%,0%,1%]@102 EMC 3%@1065 AVP 64%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 954/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [9%,7%,4%,14%]@102 EMC 3%@1065 AVP 56%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 968/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [32%,10%,40%,9%]@614 EMC 20%@665 AVP 44%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 968/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [26%,14%,20%,18%]@1224 EMC 16%@1065 AVP 58%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 969/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,18%,19%,17%]@1224 EMC 18%@1065 AVP 58%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 969/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [20%,17%,14%,7%]@403 EMC 29%@665 AVP 72%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 969/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [21%,10%,12%,11%]@1224 EMC 18%@1065 AVP 72%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 969/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [20%,16%,14%,11%]@1224 EMC 18%@1065 AVP 65%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 969/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [18%,6%,14%,13%]@1224 EMC 18%@1065 AVP 7%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 970/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [36%,15%,12%,14%]@1224 EMC 19%@1065 AVP 65%@12 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 952/3854MB (lfb 525x4MB) SWAP 0/0MB (cached 0MB) cpu [10%,14%,11%,10%]@102 EMC 10%@1065 AVP 58%@12 VDE 0 GR3D 0%@76 EDP limit 1734

H265 UDP Streaming

In order to start streaming H265 you need the h265 payloader, to do this you need to compile the latest version of gstreamer in you tegra X1. Please follow the instructions to do it. Same instructions can be run in your PC if you want to update the gstreamer version in your host. (in case of the host I had to recompile the good plugins to get the rtph265pay).

Currently it is not possible to receive the stream using the PC because there is not a h265 decoder available on gstreamer. In order to evaluate the performance let's receive the stream on the tegra as well.

export PKG_CONFIG_PATH=/home/$USER/gst_$VERSION/out/lib/pkgconfig
VERSION=1.8.0
export LD_LIBRARY_PATH=/home/$USER/gst_$VERSION/out/lib/


server:

CLIENT_IP=<IP_ADDRESS>
gst-launch-1.0 nvcamerasrc fpsRange="30 30" intent=3 ! nvvidconv flip-method=6 \
! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh265enc low-latency=1 control-rate=2 bitrate=4000000 ! 'video/x-h265, stream-format=(string)byte-stream' ! \
h265parse ! rtph265pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false async=false

client:

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H265,payload=96 ! rtph265depay \
! h265parse ! queue ! omxh265dec ! nvoverlaysink sync=false async=false -e

Latency

  • 1080p30
  • H265
  • Bitrate 4Mbps
  • Latency=258ms

References

https://devtalk.nvidia.com/default/topic/902541/jetson-tx1/low-latency-video-streaming-with-h-265/
https://devtalk.nvidia.com/default/topic/901967/jetson-tx1/gstreamer-rtp-h-265-elements-missing/

Power

https://devtalk.nvidia.com/default/topic/916735/jetson-tx1/jetson-tx1-power-requirements-and-power-management/

Price

https://devtalk.nvidia.com/default/topic/912064/jetson-tx1/are-there-plans-to-sell-tx1-processor-without-the-full-module/

Products

  • NVIDIA Shield TV
https://shield.nvidia.com/blog/tegra-x1-processor-and-shield
https://shield.nvidia.com/android-tv#intro-video
http://www.amazon.com/NVIDIA-SHIELD-Pro/dp/B00XO7AK1U?ascsubtag=s1458504729867wrusa52378&tag=52378_iceleadscom-20

Articles related

Tegra X1
Compile gstreamer on tegra X1
Compiling Tegra X1 source code