Jetson TX2 Sony IMX230 image sensor Linux driver

From RidgeRun Developer Wiki





Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help.


Driver List Information
Refer to the RidgeRun Linux Camera Drivers to meet all the list of Drivers available


Sony IM230 CMOS image sensor features

The IMX230 is a CMOS image sensor with the following features:

  • Phase Detection Auto Focus (PDAF)
  • Single Frame High Dynamic Range (HDR) with equivalent full pixels
  • High signal to noise ratio (SNR)
  • Full resolution @24fps (Normal / HDR). 4K2K @30fps (Normal / HDR) 1080p @60fps (Normal / HDR)
  • Independent flipping and mirroring
  • CSI-2 serial data output (MIPI 2lane/4lane, Max. 1.5Gbps/lane, D-PHY spec. ver. 1.1 compliant)
  • CSI2 serial data output (MIPI interface 2 lanes)
  • Output video format of RAW10/8, COMP8/6
  • Advanced Noise Reduction (Chroma noise reduction and RAW noise reduction)


Official Manufacturer IMX230 camera sensor documentation link: IMX230 21-megapixel product brief

RidgeRun has developed a driver for the Jetson TX1 platform with the following support:

  • Jetpack 3.1 and 2.3.1 support.
  • V4l2 Media controller driver
  • Tested resolution 5344x4016 @ 21 fps. Also cropped 4208x3120, 2672x2008 and 1920x1080 .
  • Output format: RAW10 Bayer BGGR pattern.
  • Leopard Imaging CB boards for TX1
  • Capture with v4l2src and nvcamerasrc
  • Support for triple video capture. (Tested using the 3 available ports LI CB)
  • BU64295/BU64297 VCM driver support

Enabling the driver

In order to use this driver, you have to patch and compile the kernel source, and there are two ways to do it:

Using RidgeRun SDK

Through the SDK you can easily patch the kernel and generate an image with the required changes to get the IMX230 sensor to work. In this wiki Getting Started Guide for Jetson TX1 you can find all the information required to build a Jetson TX1 SDK from scratch.

In order to add the IMX230 driver follow these steps:

  • Go to your SDK directory
  • Go to the kernel directory
  • Copy the patches in the patches directory
001-add-li-cb-board-driver.patch 
002-add-driver-imx230-camera.patch                    
003-imx230-custom-dtb-li-cb.patch
  • Modify the series file in the kernel directory. You have to add the 3 above patches.
  • Run make config and select the IMX230 driver in the Kernel Configuration like this:
-> Kernel Configuration
 -> Device Drivers                                                                                                                        
  -> Multimedia support                                                                                           
    -> Encoders, decoders, sensors, and other helper chips
       -> <*> IMX230 camera sensor support
  • Then make the SDK and install it following the Started Guide mentioned before

Using Jetpack

  • Once you have the source code, apply the patches to fix kernel error during compilation and add the support for the IMX230:
cd $DEVDIR/64_TX1/Linux_for_Tegra_64_tx1/sources/
PATCHES=<path to the tarball>
tar -xzf $PATCHES/imx230-driver-for-tegra-x1-patches.tar.gz
mv imx230-driver-for-tegra-x1-patches patches
quilt push -a

Please change the path in PATCHES for the one where you downloaded the tarball with the IMX230 patches.

Make sure to enable IMX230 and driver support in step 5:

-> Device Drivers                                                                                                                        
 -> Multimedia support                                                                                           
   -> Encoders, decoders, sensors, and other helper chips
      -> <*> Omnivision IMX230 camera sensor support

Using the driver (GStreamer examples)

The GStreamer version distributed with Jetpack doesn't support Bayer RAW10 only RAW8 so GStreamer needs to be patched in order to capture using v4l2src. Follow the steps in the following wiki page to add the support for RAW10:

Compile_gstreamer_on_Jetson_TX1_and_TX2

DISPLAY=:0 gst-launch-1.0 ...

Snapshots

In order to check the snapshot, you can use the following tool:

https://github.com/jdthomas/bayer2rgb

So, run the following commands to download the tool and compile it:

git clone git@github.com:jdthomas/bayer2rgb.git
cd bayer2rgb
make
cp bayer2rgb /usr/bin/

Bayer2rgb will convert naked (no header) Bayer grid data into RGB data. There are several choices of interpolation (though they all look essentially the same to my eye). It can output tiff files and can integrate with ImageMagick to output other formats.

  • 1920x1080
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=1 ! "video/x-bayer, format=bggr, width=1920, height=1080" \
! multifilesink location=test%d_1920x1080.bayer

Check the snapshot with:

./bayer2rgb --input=test#_1920x1080.bayer --output=data.tiff --width=5344 --height=4016 --bpp=16 --first=BGGR \
--method=BILINEAR --tiff

Use image_magik to convert the tiff to png:

convert data.tiff data.png

Snapshots with nvcamerasrc

The following pipeline will create a file for each captured frame. You can visualize the file on the following web page: http://rawpixels.net/

gst-launch-1.0 -v nvcamerasrc sensor-id=1 fpsRange="30 30" num-buffers=100 ! 'video/x-raw(memory:NVMM), width=(int)5344, \
height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw, width=(int)5344, \
height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! multifilesink location=test_%d.yuv

Single Capture

V4l2src

You can use the raw2rgbpnm tool to check all the buffers:

https://github.com/martinezjavier/raw2rgbpnm

So, run the following commands to download the tool and compile it:

git clone git clone git@github.com:martinezjavier/raw2rgbpnm.git
cd raw2rgbpnm

Open the file raw2rgbpnm.c and change line 489 with:

int c = getopt(argc, argv, "a:b:f:ghs:wn");

This is to enable the option to extract multiple frames from a file. Now, you can build the application:

make

In order to capture 10 buffers and save them in a file, you can run the following pipelines:

  • 5344x4016
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=10 ! "video/x-bayer, format=rggb, width=5344, height=4016" \
! filesink location=test_5344x4016.bayer

Check the buffers with:

./raw2rgbpnm -f SRGGB10 -s 5344x4016 -b 5.0 -n test_5344x4016.bayer output_5344x4016

Nvcamerasrc

  • 5344x4016
DISPLAY=:0 gst-launch-1.0 nvcamerasrc sensor-id=0 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, \
height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvegltransform ! nveglglessink -e

3 Camera Capture

The following image consists of a Jetson TX1 with three IMX230 cameras connected at the board

V4l2src

  • Pipeline for sextuple video capture using v4l2src, at 5344x4016 @21fps:
gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \
v4l2src device=/dev/video1 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \
v4l2src device=/dev/video2 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \

Nvcamerasrc

  • Pipeline for triple video capture using nvcamerasrc, at 5344x4016 @21fps:
DISPLAY=:0 gst-launch-1.0 nvcamerasrc exposure-time=0.006 sensor-id=0 fpsRange="30 30" auto-exposure=off  ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1"  ! queue !  mixer.sink_0  nvcamerasrc  auto-exposure=off exposure-time=0.006  sensor-id=1 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420,framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1"  ! queue ! mixer.sink_1 nvcamerasrc auto-exposure=off exposure-time=0.006 sensor-id=2 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv !  "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1" ! queue ! mixer.sink_2  videomixer background=1 name=mixer sink_0::xpos=0 sink_0::ypos=0  sink_1::xpos=640 sink_1::ypos=0 sink_2::xpos=320 sink_2::ypos=480 ! queue ! nvoverlaysink sync=false -v

Performance

For this test framerate was measured with vl4-ctl and perf using nvcamerasrc to capture

Framerate with v4l

This test was developed with v4l-util tools to capture a raw image and measure the framerate.

nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=5344,height=4016,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap
<<<<<<<<<<<<<<<<<<<<< 19.60 fps
<<<<<<<<<<<<<<<<<<<<< 20.39 fps
<<<<<<<<<<<<<<<<<<<<<< 20.72 fps
<<<<<<<<<<<<<<<<<<<<< 20.89 fps
<<<<<<<<<<<<<<<<<<<<< 20.95 fps
<<<<<<<<<<<<<<<<<<<<<< 21.02 fps
<<<<<<<<<<<<<<<<<<<<< 21.08 fps
<<<<<<<<<<<<<<<<<<<<< 21.12 fps
<<<<<<<<<<<<<<<<<<<<<< 21.15 fps
<<<<<<<<<<<<<<<<<<<<< 21.15 fps
<<<<<<<<<<<<<<<<<<<<< 21.18 fps
<<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.21 fps
<<<<<<<<<<<<<<<<<<<<<< 21.22 fps
<<<<<<<<<<<<<<<<<<<<< 21.22 fps
<<<<<<<<<<<<<<<<<<<<< 21.23 fps

The sensor provides the capture at 21fps with v4l.

Framerate with Nvcamerasrc and perf

These test was developed with nvcamerasrc, the results are showed below:

gst-launch-1.0 nvcamerasrc auto-exposure=1 aeLock=1 ! perf ! autovideosink
Setting pipeline to PAUSED ...

Available Sensor modes : 
5344 x 4016 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
4208 x 3120 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2672 x 2008 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1920 x 1080 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 0 WxH = 5344x4016 FrameRate = 21.000000 ...

New clock: GstSystemClock
GST-PERF INFO -->  Timestamp: 0:20:01.076040876; Bps: 0; fps: 0.0 
GST-PERF INFO -->  Timestamp: 0:20:02.085110495; Bps: 800; fps: 21.80 
GST-PERF INFO -->  Timestamp: 0:20:03.108538791; Bps: 789; fps: 21.50 
GST-PERF INFO -->  Timestamp: 0:20:04.141721657; Bps: 782; fps: 21.29 
GST-PERF INFO -->  Timestamp: 0:20:05.181324930; Bps: 777; fps: 21.17 
GST-PERF INFO -->  Timestamp: 0:20:06.218568207; Bps: 779; fps: 21.21 
GST-PERF INFO -->  Timestamp: 0:20:07.251035501; Bps: 782; fps: 21.31 
GST-PERF INFO -->  Timestamp: 0:20:08.282084777; Bps: 783; fps: 21.33 
GST-PERF INFO -->  Timestamp: 0:20:09.313816032; Bps: 783; fps: 21.33 
GST-PERF INFO -->  Timestamp: 0:20:10.346620670; Bps: 782; fps: 21.31 

With nvcamerasrc the framerate is 21fps.

Arm load while capturing

The results showed below was taken with one camera capturing and using the pipeline mentioned on the "Framerate with Nvcamerasrc and perf" section in this wiki

Single capture

These measures was taken with nvcamerasrc and only the capture system and one camera as pipeline below:

gst-launch-1.0 nvcamerasrc sensor-id=0 ! perf ! fakesink
RAM 1820/3976MB (lfb 224x4MB) CPU [1%@204,10%@204,3%@204,3%@204] 
RAM 1823/3976MB (lfb 224x4MB) CPU [6%@1734,25%@1734,11%@1734,24%@1734]
RAM 1875/3976MB (lfb 224x4MB) CPU [36%@306,12%@306,28%@306,8%@306] --------------> start capturing
RAM 1876/3976MB (lfb 224x4MB) CPU [17%@204,13%@204,12%@204,16%@204]
RAM 1876/3976MB (lfb 224x4MB) CPU [9%@306,14%@306,18%@306,21%@306] 
RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,10%@408,15%@408,19%@408]
RAM 1876/3976MB (lfb 224x4MB) CPU [16%@204,11%@204,12%@204,20%@204]
RAM 1876/3976MB (lfb 224x4MB) CPU [8%@408,16%@408,20%@408,24%@408] 
RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,10%@204,9%@204,15%@204] 
RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,13%@204,15%@204,18%@204]
RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,15%@408,11%@408,17%@408]
RAM 1876/3976MB (lfb 224x4MB) CPU [19%@102,14%@102,9%@102,21%@102] 
RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,18%@204,22%@204,19%@204]
RAM 1876/3976MB (lfb 224x4MB) CPU [17%@306,17%@306,17%@306,22%@306]
RAM 1876/3976MB (lfb 224x4MB) CPU [18%@408,19%@408,12%@408,24%@408]
RAM 1876/3976MB (lfb 224x4MB) CPU [12%@306,10%@306,12%@306,24%@306] -------------> End capturing
RAM 1821/3976MB (lfb 224x4MB) CPU [17%@1734,12%@1734,15%@1734,23%@1734]
RAM 1821/3976MB (lfb 224x4MB) CPU [2%@102,7%@102,4%@102,0%@102]

The average for each core while single capturing is: [15.57%, 13.71%, 15.14%, 19.14%] and the memory is 1876MB for the total system, for only the single capture the armload is [6.82%, 3.22%, 11.6%, 6.35%] for each core and takes 56MB of memory.

Dual capture

These measures was taken with nvcamerasrc and only the capture system and two cameras as the pipeline below:

gst-launch-1.0 nvcamerasrc sensor-id=0 auto-exposure=1 aeLock=1 ! perf ! fakesink nvcamerasrc sensor-id=1 auto-exposure=1 aeLock=1 ! perf ! fakesink
nvidia@tegra-ubuntu:~$ ./tegrastats 
RAM 1792/3976MB (lfb 199x4MB) CPU [4%@1224,12%@1224,0%@1224,7%@1224] 
RAM 1814/3976MB (lfb 199x4MB) CPU [12%@1734,6%@1734,49%@1734,0%@1734]
RAM 2425/3976MB (lfb 139x4MB) CPU [29%@1734,42%@1734,59%@1734,55%@1734] -----> Start Capture
RAM 2428/3976MB (lfb 137x4MB) CPU [18%@306,17%@306,23%@306,23%@306]
RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,28%@714,20%@714,30%@714]
RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,16%@408,22%@408,25%@408]
RAM 2429/3976MB (lfb 136x4MB) CPU [23%@204,24%@204,22%@204,25%@204]
RAM 2429/3976MB (lfb 136x4MB) CPU [17%@408,20%@408,23%@408,23%@408]
RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,18%@714,32%@714,28%@714]
RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,17%@408,20%@408,23%@408]
RAM 2429/3976MB (lfb 136x4MB) CPU [22%@612,21%@612,28%@612,28%@612]
RAM 2429/3976MB (lfb 136x4MB) CPU [13%@408,20%@408,24%@408,22%@408]
RAM 2431/3976MB (lfb 136x4MB) CPU [30%@1224,20%@1224,24%@1224,26%@1224]
RAM 2431/3976MB (lfb 136x4MB) CPU [23%@510,12%@510,20%@510,29%@510]
RAM 2431/3976MB (lfb 136x4MB) CPU [27%@510,18%@510,15%@510,26%@510]
RAM 2431/3976MB (lfb 136x4MB) CPU [21%@306,23%@306,19%@306,24%@306]  ---------> End Capture
RAM 1822/3976MB (lfb 223x4MB) CPU [11%@1224,28%@1224,20%@1224,11%@1224]
RAM 1822/3976MB (lfb 223x4MB) CPU [4%@102,6%@102,0%@102,4%@102]
RAM 1822/3976MB (lfb 223x4MB) CPU [1%@102,6%@102,0%@102,8%@102]
RAM 1822/3976MB (lfb 223x4MB) CPU [5%@102,6%@102,0%@102,1%@102]
RAM 1822/3976MB (lfb 224x4MB) CPU [5%@102,8%@102,0%@102,2%@102]

The average for each core while dual capturing is: [22.64%, 21.14%, 25.07%, 27.64%], and the memory is 2429MB for the total system, for only the dual capture the armload is [15.37%, 18.85%, 22.81%, 22.81%] for each core and takes 609MB of memory.



For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.


Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.




  Index