Sony IMX219 Linux Driver for Jetson Xavier

From RidgeRun Developer Wiki



Previous: Video_Capture_and_Display/Cameras Index Next: Video_Capture_and_Display/Software_Support


NVIDIA®Jetson Xavier™
Xavier
Introduction
SoM Overview
Carrier Boards
Software Support
JetPack 5.X
Getting Started
Compiling Source Code
Flashing the Board from Cmdline
Performance Tuning
Development in the Board
Getting into the Board
Installing Packages
Video Capture and Display
Description
Supported Cameras
Software Support
Processors
HD Audio and Video Subystem
Deep Learning Accelerator
Volta GPU
Interfaces
GStreamer Pipelines
Deep Learning
TensorRT
Deep Learning Accelerator
Deep Learning Tutorials
RidgeRun Products
Reference Documentation
Contact Us







IMX219 Features

The Sony IMX219 is a CMOS image sensor with the following features:

  • CSI2 serial data output (selection of 4 or 2 lanes)
  • Max. 30 frame/s in all-pixel scan mode
  • 180 frame/s 720p with 2x2 analog (special) binning, 60 frame/s @ 1080p with V-crop
  • Data rate: Max. 722 Mbps/lane(@4lane), 912 Mbps/Lane(@2lane)
  • Max resolution of 3280 (H) x 2464 (V) approx. 8.08 M pixels

RidgeRun has developed a driver for the Jetson Xavier platform with the following support:

  • L4T 31.1 and Jetpack 4.1
  • V4l2 Media controller driver
  • One camera capturing (TODO: to expand to 6 cameras)
  • Tested resolution 3280 x 2464 @ 15 fps
  • Tested resolution 720p @ 78 fps
  • Tested resolution 1640x1232 @ 30 fps
  • Tested resolution 820x616 @ 30 fps
  • Tested with J20 Auvidea board.
  • Capture with v4l2src and also with nvarguscamerasrc using the ISP.

Enabling the driver

In order to use this driver, you have to patch and compile the kernel source:

  • Follow the instructions in [1] to get the kernel source code (source_sync.sh script).
  • Once you have the source code, apply the following patch if you haven't yet.
4.1.1_j20_imx219.patch
  • Follow the instructions in [2] for building the kernel, and then flash the image [3].

Make sure to enable imx219 driver support: 

make menuconfig

-> Device Drivers                                                                                                                        
  -> Multimedia support                                                                                           
    -> NVIDIA overlay Encoders, decoders, sensors and other helper chips
       -> <*> IMX219 camera sensor support

You need to make sure to enable the J20 support: 

-> Device Drivers                                                                                                                        
  -> Multimedia support                                                                                           
    -> NVIDIA overlay Encoders, decoders, sensors and other helper chips 
       -> <*> Auvidea J20 Expansion board

Using the driver

V4L examples

In order to capture with V4L you can use v4l-utils and the application v4l2-ctl: 

v4l2-ctl -d /dev/video0 --set-fmt-video=width=3280,height=2464,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test_3280x2464.bayer

the image captured is in raw format (Bayer) so in order to be able to see the image you will need to convert it.

You can use the raw2rgbpnm tool to check all the buffers:

https://github.com/martinezjavier/raw2rgbpnm

So, run the following commands to download the tool and compile it: 

git clone git clone git@github.com:martinezjavier/raw2rgbpnm.git
cd raw2rgbpnm

Open the file raw2rgbpnm.c and change the line 489 with: 

int c = getopt(argc, argv, "a:b:f:ghs:wn");

This is to enable the option to extract multiple frames from a file. Now, you can build the application: 

make

This tool converts from GRBG10 to pnm. We capture RGGB in the IMX219, so you will see that the colors at the output of the image are wrong.

Check the buffers with: 

./raw2rgbpnm -f SGRBG10 -s 3296x2464 -b 5.0 -n test_3280x2464.bayer output_3280x2464


Gstreamer examples

Nvcamerasrc

  • 3280x2464
DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)3280, height=(int)2464, format=(string)NV12, framerate=(fraction)21/1' ! nvvidconv ! queue ! xvimagesink
  • 1640x1232
DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)1640, height=(int)1232, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! queue ! xvimagesink

This is an image captured with the above pipeline:

IMX219 capture with nvcamerasrc.


  • 1280x720
DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! queue ! xvimagesink
  • 820x616
DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)820, height=(int)616, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! queue ! xvimagesink

Performance

Using the following pipelines we can test the performance of the Jetson Xavier when doing capture:

V4l2
v4l2-ctl -d /dev/video0 --set-fmt-video=width=3280,height=2464,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap

The framerate for this resolution with v4l: 

nvidia@jetson-0423318029309:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=3280,height=2464,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap 
<<<<<<<<<<<<<<<<<<<< 18.81 fps
<<<<<<<<<<<<<<<< 17.32 fps
<<<<<<<<<<<<<<<< 16.88 fps
<<<<<<<<<<<<<<<< 16.62 fps
<<<<<<<<<<<<<<<< 16.50 fps
<<<<<<<<<<<<<<<< 16.41 fps
<<<<<<<<<<<<<<<< 16.33 fps
<<<<<<<<<<<<<<<< 16.29 fps
<<<<<<<<<<<<<<<< 16.24 fps
<<<<<<<<<<<<<<<< 16.21 fps
<<<<<<<<<<<<<<<< 16.19 fps
<<<<<<<<<<<<<<<< 16.16 fps
<<<<<<<<<<<<<<< 16.15 fps


We noticed that capturing with the max resolution 3280x2464, the cpu load consumption measured with tegrastats doesn't change considerably, and it remains almost the same:

  • Tegrastats capturing with v4l2-ctl and 3280x2464 resolution:
RAM 1826/15820MB (lfb 3352x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1826/15820MB (lfb 3352x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [0%@1190,0%@1190,4%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [1%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [1%@1190,0%@1190,0%@1190,2%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [1%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1888/15820MB (lfb 3340x4MB) CPU [1%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] 
RAM 1826/15820MB (lfb 3352x4MB) CPU [1%@1190,0%@1190,0%@1190,1%@1190,off,off,off,off] 
RAM 1826/15820MB (lfb 3352x4MB) CPU [0%@1190,0%@1190,0%@1190,2%@1190,off,off,off,off]


Nvcamerasrc
DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)3280, height=(int)2464, format=(string)NV12, framerate=(fraction)21/1' ! fakesink


The framerate for this resolution with nvarguscamerasrc and perf: 

nvidia@jetson-0423318029309:~$ DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)3280, height=(int)2464, format=(string)NV12, framerate=(fraction)21/1' ! perf ! fakesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :

GST_ARGUS: 3280 x 2464 FR = 21.000000 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 820 x 616 FR = 29.999999 fps; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3280 H = 2464 
   seconds to Run    = 0 
   Frame Rate = 21.000000 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST-PERF INFO -->  Timestamp: 0:30:53.091389373; Bps: 0; fps: 0.0 
GST-PERF INFO -->  Timestamp: 0:30:54.144861013; Bps: 911; fps: 16.14 
GST-PERF INFO -->  Timestamp: 0:30:55.151976993; Bps: 953; fps: 15.88 
GST-PERF INFO -->  Timestamp: 0:30:56.152766447; Bps: 960; fps: 16.0 
GST-PERF INFO -->  Timestamp: 0:30:57.156691528; Bps: 957; fps: 15.95 
GST-PERF INFO -->  Timestamp: 0:30:58.161355065; Bps: 956; fps: 15.93 
GST-PERF INFO -->  Timestamp: 0:30:59.164075813; Bps: 958; fps: 15.96 
GST-PERF INFO -->  Timestamp: 0:31:00.168307587; Bps: 956; fps: 15.93 
GST-PERF INFO -->  Timestamp: 0:31:01.172187921; Bps: 957; fps: 15.95 
GST-PERF INFO -->  Timestamp: 0:31:02.176199458; Bps: 956; fps: 15.93 
GST-PERF INFO -->  Timestamp: 0:31:03.179801283; Bps: 957; fps: 15.95 
GST-PERF INFO -->  Timestamp: 0:31:04.183208955; Bps: 957; fps: 15.95 
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:13.407925037
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...


  • Tegrastats capturing with nvarguscamerasrc and 3280x2464 resolution:
RAM 2421/15820MB (lfb 3194x4MB) CPU [9%@1190,6%@1190,5%@1190,10%@1190,off,off,off,off]
RAM 2423/15820MB (lfb 3194x4MB) CPU [7%@1190,5%@1190,6%@1190,6%@1190,off,off,off,off] 
RAM 2423/15820MB (lfb 3194x4MB) CPU [7%@1190,6%@1190,6%@1190,5%@1190,off,off,off,off] 
RAM 2423/15820MB (lfb 3194x4MB) CPU [6%@1190,7%@1190,3%@1190,9%@1190,off,off,off,off] 
RAM 2423/15820MB (lfb 3194x4MB) CPU [8%@1190,2%@1190,7%@1190,8%@1190,off,off,off,off] 
RAM 2424/15820MB (lfb 3194x4MB) CPU [8%@1190,8%@1190,7%@1190,11%@1190,off,off,off,off]
RAM 2425/15820MB (lfb 3194x4MB) CPU [5%@1190,4%@1190,5%@1190,7%@1190,off,off,off,off] 
RAM 2426/15820MB (lfb 3194x4MB) CPU [7%@1190,5%@1190,7%@1190,6%@1190,off,off,off,off] 
RAM 2426/15820MB (lfb 3194x4MB) CPU [9%@1190,7%@1190,8%@1190,5%@1190,off,off,off,off] 
RAM 2426/15820MB (lfb 3194x4MB) CPU [6%@1190,6%@1190,3%@1190,6%@1190,off,off,off,off]


These tests were done using the J20 board from Auvidea Getting started guide for Auvidea J20 board

Video Encoding Transport Stream 1640x1232@30fps

CAPS="video/x-raw(memory:NVMM), width=(int)1640, height=(int)1232, format=(string)NV12, framerate=(fraction)30/1"

DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! capsfilter caps="$CAPS" ! omxh264enc ! \
               mpegtsmux ! filesink location=test.ts

Snapshots

DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 num-buffers=2 ! 'video/x-raw(memory:NVMM), width=(int)1640, height=(int)1232, format=(string)NV12, \
framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw, width=(int)1640, height=(int)1232, format=(string)NV12, framerate=(fraction)30/1' ! multifilesink location=test_%d.yuv

Imx219 latency in Xavier

3280x2464 16fps glass to glass latency

Jetson Glass to glass latency test for 3280x2464 16fps IMX219 camera mode

Test pipeline:

nvidia@jetson-0423318029309:~$ DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),\
width=(int)3280, height=(int)2464, format=(string)NV12, framerate=(fraction)21/1' ! nvvidconv ! queue ! xvimagesink

Glass to glass latency = 215 ms

frames glass to glass latency measurement method

Contact Us

If you are interested in purchasing the driver, please post your inquiry at our Contact Us link.



Previous: Video_Capture_and_Display/Cameras Index Next: Video_Capture_and_Display/Software_Support



Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=Xavier/Video_Capture_and_Display/Cameras/IMX219&oldid=60706"