Sony IMX230 Linux driver
 |
|
|
|
Sony IM230 CMOS image sensor features
The IMX230 is a CMOS image sensor with the following features:
- Phase Detection Auto Focus (PDAF)
- Single Frame High Dynamic Range (HDR) with equivalent full pixels
- High signal to noise ratio (SNR)
- Full resolution @24fps (Normal / HDR). 4K2K @30fps (Normal / HDR) 1080p @60fps (Normal / HDR)
- Independent flipping and mirroring
- CSI-2 serial data output (MIPI 2lane/4lane, Max. 1.5Gbps/lane, D-PHY spec. ver. 1.1 compliant)
- CSI2 serial data output (MIPI interface 2 lanes)
- Output video format of RAW10/8, COMP8/6
- Advanced Noise Reduction (Chroma noise reduction and RAW noise reduction)
Official Manufacturer IMX230 camera sensor documentation link: IMX230 21-megapixel product brief
RidgeRun has developed a driver for the Jetson TX1 platform with the following support:
- Jetpack 3.1 and 2.3.1 support.
- V4l2 Media controller driver
- Tested resolution 5344x4016 @ 21 fps. Also cropped 4208x3120, 2672x2008 and 1920x1080 .
- Output format: RAW10 Bayer BGGR pattern.
- Leopard Imaging CB boards for TX1
- Capture with v4l2src and nvcamerasrc
- Support for triple video capture. (Tested using the 3 available ports LI CB)
- BU64295/BU64297 VCM driver support
Enabling the driver
In order to use this driver, you have to patch and compile the kernel source, and there are two ways to do it:
Using RidgeRun SDK
Through the SDK you can easily patch the kernel and generate an image with the required changes to get the IMX230 sensor to work. In this wiki Getting Started Guide for Jetson TX1 you can find all the information required to build a Jetson TX1 SDK from scratch.
In order to add the IMX230 driver follow these steps:
- Go to your SDK directory
- Go to the kernel directory
- Copy the patches in the patches directory
001-add-li-cb-board-driver.patch 002-add-driver-imx230-camera.patch 003-imx230-custom-dtb-li-cb.patch
- Modify the series file in the kernel directory. You have to add the 3 above patches.
- Run make config and select the IMX230 driver in the Kernel Configuration like this:
-> Kernel Configuration
-> Device Drivers
-> Multimedia support
-> Encoders, decoders, sensors, and other helper chips
-> <*> IMX230 camera sensor support
- Then make the SDK and install it following the Started Guide mentioned before
Using Jetpack
- Follow the instructions in Compiling_Tegra_X1/TX2_source_code (Downloading the code) to get the kernel source code.
- If you didn't have downloaded the toolchain yet, follow the instructions in Compiling_Tegra_TX1/TX2_source_code (Toolchain)
- Once you have the source code, apply the patches to fix kernel error during compilation and add the support for the IMX230:
cd $DEVDIR/64_TX1/Linux_for_Tegra_64_tx1/sources/ PATCHES=<path to the tarball> tar -xzf $PATCHES/imx230-driver-for-tegra-x1-patches.tar.gz mv imx230-driver-for-tegra-x1-patches patches quilt push -a
Please change the path in PATCHES for the one where you downloaded the tarball with the IMX230 patches.
- Follow the instructions in Compiling Jetson_TX1/TX2 source_code (Build Kernel) for building the kernel, and then flash the image. Omit steps 3 and 4 because you already applied the patches.
Make sure to enable IMX230 and driver support in step 5:
-> Device Drivers
-> Multimedia support
-> Encoders, decoders, sensors, and other helper chips
-> <*> Omnivision IMX230 camera sensor support
- To flash the built image into the board follow the instructions in Compiling_Jetson_TX1/TX2_source_code(Flashing_the_board)
Using the driver (GStreamer examples)
The GStreamer version distributed with Jetpack doesn't support Bayer RAW10 only RAW8 so GStreamer needs to be patched in order to capture using v4l2src. Follow the steps in the following wiki page to add the support for RAW10:
Compile_gstreamer_on_Jetson_TX1_and_TX2
 | Important Note: When you are accessing the board through serial or ssh and you want to run a pipeline to display with autovideosink, nveglglessink, xvimagesink or any other video sink, you have to run your pipeline with DISPLAY=:0 at the beginning of the description: |
DISPLAY=:0 gst-launch-1.0 ...
Snapshots
In order to check the snapshot, you can use the following tool:
https://github.com/jdthomas/bayer2rgb
So, run the following commands to download the tool and compile it:
git clone git@github.com:jdthomas/bayer2rgb.git cd bayer2rgb make cp bayer2rgb /usr/bin/
Bayer2rgb will convert naked (no header) Bayer grid data into RGB data. There are several choices of interpolation (though they all look essentially the same to my eye). It can output tiff files and can integrate with ImageMagick to output other formats.
- 1920x1080
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=1 ! "video/x-bayer, format=bggr, width=1920, height=1080" \ ! multifilesink location=test%d_1920x1080.bayer
Check the snapshot with:
./bayer2rgb --input=test#_1920x1080.bayer --output=data.tiff --width=5344 --height=4016 --bpp=16 --first=BGGR \ --method=BILINEAR --tiff
Use image_magik to convert the tiff to png:
convert data.tiff data.png
| Important Note 1: In general the first buffer contains very low light because the AWB algorithm of the sensor is calibrating, so we recommend using multifilesink to test debayer with a buffer above from number one. To obtain better image colors and bright quality, due to automatic sensor image calibration, we recommend to test debayer with a frame above number 10, to give time to the sensor to adjust the best image calibration parameters. |
| Important Note 2: The debayered image obtained as the output when using the "bayer2rgb" tool presents some kind of light saturation, viewed as multiple color pixels sections. This is a problem of the tool used to do the debayer process but led the users to verify that the driver and camera sensor is working fine. |
Snapshots with nvcamerasrc
The following pipeline will create a file for each captured frame. You can visualize the file on the following web page: http://rawpixels.net/
gst-launch-1.0 -v nvcamerasrc sensor-id=1 fpsRange="30 30" num-buffers=100 ! 'video/x-raw(memory:NVMM), width=(int)5344, \ height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw, width=(int)5344, \ height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! multifilesink location=test_%d.yuv
Single Capture
V4l2src
You can use the raw2rgbpnm tool to check all the buffers:
https://github.com/martinezjavier/raw2rgbpnm
So, run the following commands to download the tool and compile it:
git clone git clone git@github.com:martinezjavier/raw2rgbpnm.git cd raw2rgbpnm
Open the file raw2rgbpnm.c and change line 489 with:
int c = getopt(argc, argv, "a:b:f:ghs:wn");
This is to enable the option to extract multiple frames from a file. Now, you can build the application:
make
| Important Note: This tool converts from GRBG10 to pnm. We capture RGGB in the IMX230, so you will see that the colors at the output of the image are wrong. |
In order to capture 10 buffers and save them in a file, you can run the following pipelines:
- 5344x4016
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=10 ! "video/x-bayer, format=rggb, width=5344, height=4016" \ ! filesink location=test_5344x4016.bayer
Check the buffers with:
./raw2rgbpnm -f SRGGB10 -s 5344x4016 -b 5.0 -n test_5344x4016.bayer output_5344x4016
Nvcamerasrc
- 5344x4016
DISPLAY=:0 gst-launch-1.0 nvcamerasrc sensor-id=0 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, \ height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvegltransform ! nveglglessink -e
3 Camera Capture
The following image consists of a Jetson TX1 with three IMX230 cameras connected at the board
V4l2src
- Pipeline for sextuple video capture using v4l2src, at 5344x4016 @21fps:
gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \ v4l2src device=/dev/video1 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \ v4l2src device=/dev/video2 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \
Nvcamerasrc
- Pipeline for triple video capture using nvcamerasrc, at 5344x4016 @21fps:
DISPLAY=:0 gst-launch-1.0 nvcamerasrc exposure-time=0.006 sensor-id=0 fpsRange="30 30" auto-exposure=off ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1" ! queue ! mixer.sink_0 nvcamerasrc auto-exposure=off exposure-time=0.006 sensor-id=1 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420,framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1" ! queue ! mixer.sink_1 nvcamerasrc auto-exposure=off exposure-time=0.006 sensor-id=2 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1" ! queue ! mixer.sink_2 videomixer background=1 name=mixer sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=640 sink_1::ypos=0 sink_2::xpos=320 sink_2::ypos=480 ! queue ! nvoverlaysink sync=false -v
Performance
For this test framerate was measured with vl4-ctl and perf using nvcamerasrc to capture
Framerate with v4l
This test was developed with v4l-util tools to capture a raw image and measure the framerate.
nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=5344,height=4016,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap <<<<<<<<<<<<<<<<<<<<< 19.60 fps <<<<<<<<<<<<<<<<<<<<< 20.39 fps <<<<<<<<<<<<<<<<<<<<<< 20.72 fps <<<<<<<<<<<<<<<<<<<<< 20.89 fps <<<<<<<<<<<<<<<<<<<<< 20.95 fps <<<<<<<<<<<<<<<<<<<<<< 21.02 fps <<<<<<<<<<<<<<<<<<<<< 21.08 fps <<<<<<<<<<<<<<<<<<<<< 21.12 fps <<<<<<<<<<<<<<<<<<<<<< 21.15 fps <<<<<<<<<<<<<<<<<<<<< 21.15 fps <<<<<<<<<<<<<<<<<<<<< 21.18 fps <<<<<<<<<<<<<<<<<<<<<< 21.19 fps <<<<<<<<<<<<<<<<<<<<< 21.19 fps <<<<<<<<<<<<<<<<<<<<< 21.21 fps <<<<<<<<<<<<<<<<<<<<<< 21.22 fps <<<<<<<<<<<<<<<<<<<<< 21.22 fps <<<<<<<<<<<<<<<<<<<<< 21.23 fps
The sensor provides the capture at 21fps with v4l.
Framerate with Nvcamerasrc and perf
These test was developed with nvcamerasrc, the results are showed below:
gst-launch-1.0 nvcamerasrc auto-exposure=1 aeLock=1 ! perf ! autovideosink Setting pipeline to PAUSED ... Available Sensor modes : 5344 x 4016 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 4208 x 3120 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 2672 x 2008 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 1920 x 1080 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 0 WxH = 5344x4016 FrameRate = 21.000000 ... New clock: GstSystemClock GST-PERF INFO --> Timestamp: 0:20:01.076040876; Bps: 0; fps: 0.0 GST-PERF INFO --> Timestamp: 0:20:02.085110495; Bps: 800; fps: 21.80 GST-PERF INFO --> Timestamp: 0:20:03.108538791; Bps: 789; fps: 21.50 GST-PERF INFO --> Timestamp: 0:20:04.141721657; Bps: 782; fps: 21.29 GST-PERF INFO --> Timestamp: 0:20:05.181324930; Bps: 777; fps: 21.17 GST-PERF INFO --> Timestamp: 0:20:06.218568207; Bps: 779; fps: 21.21 GST-PERF INFO --> Timestamp: 0:20:07.251035501; Bps: 782; fps: 21.31 GST-PERF INFO --> Timestamp: 0:20:08.282084777; Bps: 783; fps: 21.33 GST-PERF INFO --> Timestamp: 0:20:09.313816032; Bps: 783; fps: 21.33 GST-PERF INFO --> Timestamp: 0:20:10.346620670; Bps: 782; fps: 21.31
With nvcamerasrc the framerate is 21fps.
Arm load while capturing
The results showed below was taken with one camera capturing and using the pipeline mentioned on the "Framerate with Nvcamerasrc and perf" section in this wiki
Single capture
These measures was taken with nvcamerasrc and only the capture system and one camera as pipeline below:
gst-launch-1.0 nvcamerasrc sensor-id=0 ! perf ! fakesink
RAM 1820/3976MB (lfb 224x4MB) CPU [1%@204,10%@204,3%@204,3%@204] RAM 1823/3976MB (lfb 224x4MB) CPU [6%@1734,25%@1734,11%@1734,24%@1734] RAM 1875/3976MB (lfb 224x4MB) CPU [36%@306,12%@306,28%@306,8%@306] --------------> start capturing RAM 1876/3976MB (lfb 224x4MB) CPU [17%@204,13%@204,12%@204,16%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [9%@306,14%@306,18%@306,21%@306] RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,10%@408,15%@408,19%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [16%@204,11%@204,12%@204,20%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [8%@408,16%@408,20%@408,24%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,10%@204,9%@204,15%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,13%@204,15%@204,18%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,15%@408,11%@408,17%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [19%@102,14%@102,9%@102,21%@102] RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,18%@204,22%@204,19%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [17%@306,17%@306,17%@306,22%@306] RAM 1876/3976MB (lfb 224x4MB) CPU [18%@408,19%@408,12%@408,24%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [12%@306,10%@306,12%@306,24%@306] -------------> End capturing RAM 1821/3976MB (lfb 224x4MB) CPU [17%@1734,12%@1734,15%@1734,23%@1734] RAM 1821/3976MB (lfb 224x4MB) CPU [2%@102,7%@102,4%@102,0%@102]
The average for each core while single capturing is: [15.57%, 13.71%, 15.14%, 19.14%] and the memory is 1876MB for the total system, for only the single capture the armload is [6.82%, 3.22%, 11.6%, 6.35%] for each core and takes 56MB of memory.
Dual capture
These measures was taken with nvcamerasrc and only the capture system and two cameras as the pipeline below:
gst-launch-1.0 nvcamerasrc sensor-id=0 auto-exposure=1 aeLock=1 ! perf ! fakesink nvcamerasrc sensor-id=1 auto-exposure=1 aeLock=1 ! perf ! fakesink
nvidia@tegra-ubuntu:~$ ./tegrastats RAM 1792/3976MB (lfb 199x4MB) CPU [4%@1224,12%@1224,0%@1224,7%@1224] RAM 1814/3976MB (lfb 199x4MB) CPU [12%@1734,6%@1734,49%@1734,0%@1734] RAM 2425/3976MB (lfb 139x4MB) CPU [29%@1734,42%@1734,59%@1734,55%@1734] -----> Start Capture RAM 2428/3976MB (lfb 137x4MB) CPU [18%@306,17%@306,23%@306,23%@306] RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,28%@714,20%@714,30%@714] RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,16%@408,22%@408,25%@408] RAM 2429/3976MB (lfb 136x4MB) CPU [23%@204,24%@204,22%@204,25%@204] RAM 2429/3976MB (lfb 136x4MB) CPU [17%@408,20%@408,23%@408,23%@408] RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,18%@714,32%@714,28%@714] RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,17%@408,20%@408,23%@408] RAM 2429/3976MB (lfb 136x4MB) CPU [22%@612,21%@612,28%@612,28%@612] RAM 2429/3976MB (lfb 136x4MB) CPU [13%@408,20%@408,24%@408,22%@408] RAM 2431/3976MB (lfb 136x4MB) CPU [30%@1224,20%@1224,24%@1224,26%@1224] RAM 2431/3976MB (lfb 136x4MB) CPU [23%@510,12%@510,20%@510,29%@510] RAM 2431/3976MB (lfb 136x4MB) CPU [27%@510,18%@510,15%@510,26%@510] RAM 2431/3976MB (lfb 136x4MB) CPU [21%@306,23%@306,19%@306,24%@306] ---------> End Capture RAM 1822/3976MB (lfb 223x4MB) CPU [11%@1224,28%@1224,20%@1224,11%@1224] RAM 1822/3976MB (lfb 223x4MB) CPU [4%@102,6%@102,0%@102,4%@102] RAM 1822/3976MB (lfb 223x4MB) CPU [1%@102,6%@102,0%@102,8%@102] RAM 1822/3976MB (lfb 223x4MB) CPU [5%@102,6%@102,0%@102,1%@102] RAM 1822/3976MB (lfb 224x4MB) CPU [5%@102,8%@102,0%@102,2%@102]
The average for each core while dual capturing is: [22.64%, 21.14%, 25.07%, 27.64%], and the memory is 2429MB for the total system, for only the dual capture the armload is [15.37%, 18.85%, 22.81%, 22.81%] for each core and takes 609MB of memory.
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.
ARTIFICIAL INTELLIGENCE
