Sony IMX230 Linux driver: Difference between revisions
Acervantes (talk | contribs) |
No edit summary |
||
(35 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
<seo title="IMX230 Driver | Linux driver for Jetson | <seo title=" Sony IMX230 Driver | Sony IMX230 Linux driver for NVIDIA Jetson | RidgeRun" titlemode="replace" metakeywords="GStreamer, Linux SDK, Linux BSP, Embedded Linux, Device Drivers, Nvidia, Xilinx, TI, NXP, Freescale, Embedded Linux driver development, Linux Software development, Embedded Linux SDK, Embedded Linux Application development, GStreamer Multimedia Framework, IMX230, Sony IMX230, V4L2 Driver, IMX2XX, IMX230 Jetson TX1, IMX230 Jetson TX2, Sony IMX230 Linux driver, Sony IMX230 NVIDIA Jetson" metadescription="Learn more about Sony IMX230 Linux driver for Nvidia Jetson from RidgeRun today."></seo> | ||
{{Driver-shoppingcart & NVIDIA logo & RR Contact}} | |||
< | <br> | ||
<td> | <td> | ||
{{GStreamer debug}} | |||
<td> | <td> | ||
{{message|title=Driver List Information|Refer to the [[RidgeRun_Linux_Camera_Drivers/Available_Camera_Drivers/List_of_Tested_Camera_Sensors|RidgeRun Linux Camera Drivers]] to meet all the list of Drivers available}} | |||
==Sony IM230 CMOS image sensor features== | |||
=IM230 | |||
The IMX230 is a CMOS image sensor with the following features: | The IMX230 is a CMOS image sensor with the following features: | ||
Line 31: | Line 24: | ||
Official Manufacturer IMX230 camera sensor documentation link: [https://www.sony-semicon.co.jp/products_en/IS/sensor1/img/products/ProductBrief_IMX230_20150427.pdf IMX230 21-megapixel product brief] | |||
RidgeRun has developed a driver for the Jetson TX1 platform with the following support: | RidgeRun has developed a driver for the Jetson TX1 platform with the following support: | ||
Line 45: | Line 37: | ||
* BU64295/BU64297 VCM driver support | * BU64295/BU64297 VCM driver support | ||
=Enabling the driver= | ==Enabling the driver== | ||
In order to use this driver, you have to patch and compile the kernel source, and there are two ways to do it: | In order to use this driver, you have to patch and compile the kernel source, and there are two ways to do it: | ||
==Using RidgeRun SDK== | ===Using RidgeRun SDK=== | ||
Through the SDK you can easily patch the kernel and generate an image with the required changes to get the IMX230 sensor to work. In this wiki [[ | Through the SDK you can easily patch the kernel and generate an image with the required changes to get the IMX230 sensor to work. In this wiki [[Getting Started Guide for Jetson TX1]] you can find all the information required to build a Jetson TX1 SDK from scratch. | ||
In order to add the IMX230 driver follow these steps: | In order to add the IMX230 driver follow these steps: | ||
* Go | * Go to your SDK directory | ||
* Go to the kernel directory | * Go to the kernel directory | ||
* Copy the patches in the patches directory | * Copy the patches in the patches directory | ||
Line 70: | Line 62: | ||
-> Device Drivers | -> Device Drivers | ||
-> Multimedia support | -> Multimedia support | ||
-> Encoders, decoders, sensors and other helper chips | -> Encoders, decoders, sensors, and other helper chips | ||
-> <*> IMX230 camera sensor support | -> <*> IMX230 camera sensor support | ||
</pre> | </pre> | ||
* Then '''make''' the SDK and install following the Started Guide mentioned before | * Then '''make''' the SDK and install it following the Started Guide mentioned before | ||
==Using Jetpack== | ===Using Jetpack=== | ||
* Follow the instructions in [ | * Follow the instructions in [[Compiling_Jetson_TX1/TX2_source_code#Downloading_the_code | Compiling_Tegra_X1/TX2_source_code (Downloading the code)]] to get the kernel source code. | ||
* If you didn't have | * If you didn't have downloaded the toolchain yet, follow the instructions in [[Compiling_Jetson_TX1/TX2_source_code#Toolchain | Compiling_Tegra_TX1/TX2_source_code (Toolchain)]] | ||
* Once you have the source code, apply the patches | * Once you have the source code, apply the patches to fix kernel error during compilation and add the support for the IMX230: | ||
<pre> | <pre> | ||
Line 94: | Line 86: | ||
Please change the path in PATCHES for the one where you downloaded the tarball with the IMX230 patches. | Please change the path in PATCHES for the one where you downloaded the tarball with the IMX230 patches. | ||
* Follow the instructions in [ | * Follow the instructions in [[Compiling_Jetson_TX1/TX2_source_code#Build_Kernel | Compiling Jetson_TX1/TX2 source_code (Build Kernel)]] for building the kernel, and then flash the image. Omit steps 3 and 4 because you already applied the patches. | ||
Make sure to enable IMX230 and driver support in | Make sure to enable IMX230 and driver support in step 5: | ||
<pre> | <pre> | ||
-> Device Drivers | -> Device Drivers | ||
-> Multimedia support | -> Multimedia support | ||
-> Encoders, decoders, sensors and other helper chips | -> Encoders, decoders, sensors, and other helper chips | ||
-> <*> Omnivision IMX230 camera sensor support | -> <*> Omnivision IMX230 camera sensor support | ||
</pre> | </pre> | ||
* To flash the built image into the board follow the instructions in [ | * To flash the built image into the board follow the instructions in [[Compiling_Jetson_TX1/TX2_source_code#Flashing_the_Tegra | Compiling_Jetson_TX1/TX2_source_code(Flashing_the_board)]] | ||
=Using the driver ( | ==Using the driver (GStreamer examples)== | ||
The | The GStreamer version distributed with Jetpack doesn't support Bayer RAW10 only RAW8 so GStreamer needs to be patched in order to capture using v4l2src. Follow the steps in the following wiki page to add the support for RAW10: | ||
[[Compile_gstreamer_on_Jetson_TX1_and_TX2]] | |||
{{Ambox | |||
|type=notice | |||
|small=left | |||
|issue='''Important Note:''' When you are accessing the board through serial or ssh and you want to run a pipeline to display with '''autovideosink''', '''nveglglessink''', '''xvimagesink''' or any other video sink, you have to run your pipeline with '''''DISPLAY=:0''''' at the beginning of the description: | |||
|style=width:unset; | |||
}} | |||
<pre style="background:#d6e4f1"> | <pre style="background:#d6e4f1"> | ||
DISPLAY=:0 gst-launch-1.0 ... | DISPLAY=:0 gst-launch-1.0 ... | ||
</pre> | </pre> | ||
=== Snapshots === | |||
== Snapshots == | |||
In order to check the snapshot, you can use the following tool: | In order to check the snapshot, you can use the following tool: | ||
Line 135: | Line 131: | ||
</pre> | </pre> | ||
Bayer2rgb will convert naked (no header) | Bayer2rgb will convert naked (no header) Bayer grid data into RGB data. There are several choices of interpolation (though they all look essentially the same to my eye). It can output tiff files and can integrate with ImageMagick to output other formats. | ||
*1920x1080 | *1920x1080 | ||
Line 156: | Line 152: | ||
</pre> | </pre> | ||
'''Important Note 1:''' In general the first buffer contains very low light because the AWB algorithm of the sensor is calibrating, so we recommend | {{Ambox | ||
|type=notice | |||
|small=left | |||
|issue='''Important Note 1:''' In general the first buffer contains very low light because the AWB algorithm of the sensor is calibrating, so we recommend using multifilesink to test debayer with a buffer above from number one. To obtain better image colors and bright quality, due to automatic sensor image calibration, we recommend to test debayer with a frame above number 10, to give time to the sensor to adjust the best image calibration parameters. | |||
|style=width:unset; | |||
}} | |||
'''Important Note 2:''' The debayered image obtained as the output when | {{Ambox | ||
|type=notice | |||
|small=left | |||
|issue='''Important Note 2:''' The debayered image obtained as the output when using the "bayer2rgb" tool presents some kind of light saturation, viewed as multiple color pixels sections. This is a problem of the tool used to do the debayer process but led the users to verify that the driver and camera sensor is working fine. | |||
|style=width:unset; | |||
}} | |||
=== Snapshots with nvcamerasrc === | ==== Snapshots with nvcamerasrc ==== | ||
The following pipeline will create a file for each captured frame. You can visualize the file | The following pipeline will create a file for each captured frame. You can visualize the file on the following web page: http://rawpixels.net/ | ||
<pre style="background:#d6e4f1"> | <pre style="background:#d6e4f1"> | ||
Line 169: | Line 175: | ||
</pre> | </pre> | ||
=== Single Capture === | |||
== | ==== V4l2src ==== | ||
=== | |||
You can use the raw2rgbpnm tool to check all the buffers: | You can use the raw2rgbpnm tool to check all the buffers: | ||
Line 184: | Line 189: | ||
</pre> | </pre> | ||
Open the file raw2rgbpnm.c and change | Open the file raw2rgbpnm.c and change line 489 with: | ||
<pre> | <pre> | ||
Line 196: | Line 201: | ||
</pre> | </pre> | ||
'''Important Note:''' This tool converts from GRBG10 to pnm. We capture RGGB in the IMX230, so you will see that the colors at the output of the image are wrong. | {{Ambox | ||
|type=notice | |||
|small=left | |||
|issue='''Important Note:''' This tool converts from GRBG10 to pnm. We capture RGGB in the IMX230, so you will see that the colors at the output of the image are wrong. | |||
|style=width:unset; | |||
}} | |||
In order to capture 10 buffers and save them in a file, you can run the following pipelines: | In order to capture 10 buffers and save them in a file, you can run the following pipelines: | ||
Line 213: | Line 222: | ||
</pre> | </pre> | ||
=== Nvcamerasrc === | ==== Nvcamerasrc ==== | ||
* 5344x4016 | * 5344x4016 | ||
<pre style="background:#d6e4f1"> | <pre style="background:#d6e4f1"> | ||
Line 220: | Line 229: | ||
</pre> | </pre> | ||
== | === 3 Camera Capture === | ||
The following image consists | The following image consists of a Jetson TX1 with three IMX230 cameras connected at the board | ||
==== V4l2src ==== | |||
=== V4l2src === | |||
* Pipeline for sextuple video capture using v4l2src, at 5344x4016 @21fps: | * Pipeline for sextuple video capture using v4l2src, at 5344x4016 @21fps: | ||
<pre style="background:#d6e4f1"> | <pre style="background:#d6e4f1"> | ||
Line 233: | Line 241: | ||
</pre> | </pre> | ||
=== Nvcamerasrc === | ==== Nvcamerasrc ==== | ||
* Pipeline for triple video capture using nvcamerasrc, at 5344x4016 @21fps: | * Pipeline for triple video capture using nvcamerasrc, at 5344x4016 @21fps: | ||
<pre style="background:#d6e4f1"> | <pre style="background:#d6e4f1"> | ||
Line 239: | Line 247: | ||
</pre> | </pre> | ||
= Performance = | == Performance == | ||
For this test framerate was measured with vl4-ctl and perf using nvcamerasrc to capture | For this test framerate was measured with vl4-ctl and perf using nvcamerasrc to capture | ||
== Framerate with v4l == | === Framerate with v4l === | ||
This test was developed with v4l-util tools to capture a raw image and | This test was developed with v4l-util tools to capture a raw image and measure the framerate. | ||
<pre> | <pre> | ||
nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=5344,height=4016,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap | nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=5344,height=4016,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap | ||
Line 268: | Line 276: | ||
The sensor provides the capture at 21fps with v4l. | The sensor provides the capture at 21fps with v4l. | ||
==Framerate with Nvcamerasrc and perf == | ===Framerate with Nvcamerasrc and perf === | ||
These test was developed with nvcamerasrc, the results are showed below: | These test was developed with nvcamerasrc, the results are showed below: | ||
<pre> | <pre> | ||
gst-launch-1.0 nvcamerasrc auto-exposure=1 aeLock=1 ! perf ! autovideosink | |||
Setting pipeline to PAUSED ... | Setting pipeline to PAUSED ... | ||
Line 281: | Line 289: | ||
Pipeline is live and does not need PREROLL ... | Pipeline is live and does not need PREROLL ... | ||
Setting pipeline to PLAYING ... | Setting pipeline to PLAYING ... | ||
NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 0 WxH = 5344x4016 FrameRate = 21.000000 ... | |||
New clock: GstSystemClock | New clock: GstSystemClock | ||
GST-PERF INFO --> Timestamp: 0:20:01.076040876; Bps: 0; fps: 0.0 | |||
GST-PERF INFO --> Timestamp: 0:20:02.085110495; Bps: 800; fps: 21.80 | |||
GST-PERF INFO --> Timestamp: 0:20:03.108538791; Bps: 789; fps: 21.50 | |||
GST-PERF INFO --> Timestamp: 0:20:04.141721657; Bps: 782; fps: 21.29 | |||
GST-PERF INFO --> Timestamp: 0:20:05.181324930; Bps: 777; fps: 21.17 | |||
GST-PERF INFO --> Timestamp: 0:20:06.218568207; Bps: 779; fps: 21.21 | |||
GST-PERF INFO --> Timestamp: 0:20:07.251035501; Bps: 782; fps: 21.31 | |||
GST-PERF INFO --> Timestamp: 0:20:08.282084777; Bps: 783; fps: 21.33 | |||
GST-PERF INFO --> Timestamp: 0:20:09.313816032; Bps: 783; fps: 21.33 | |||
GST-PERF INFO --> Timestamp: 0:20:10.346620670; Bps: 782; fps: 21.31 | |||
</pre> | |||
With nvcamerasrc the framerate is 21fps. | |||
=== Arm load while capturing === | |||
The results showed below was taken with one camera capturing and using the pipeline mentioned on the "Framerate with Nvcamerasrc and perf" section in this wiki | |||
====Single capture==== | |||
These measures was taken with nvcamerasrc and only the capture system and one camera as pipeline below: | |||
<pre> | |||
gst-launch-1.0 nvcamerasrc sensor-id=0 ! perf ! fakesink | |||
</pre> | |||
<pre> | |||
RAM 1820/3976MB (lfb 224x4MB) CPU [1%@204,10%@204,3%@204,3%@204] | |||
RAM 1823/3976MB (lfb 224x4MB) CPU [6%@1734,25%@1734,11%@1734,24%@1734] | |||
RAM 1875/3976MB (lfb 224x4MB) CPU [36%@306,12%@306,28%@306,8%@306] --------------> start capturing | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [17%@204,13%@204,12%@204,16%@204] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [9%@306,14%@306,18%@306,21%@306] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,10%@408,15%@408,19%@408] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [16%@204,11%@204,12%@204,20%@204] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [8%@408,16%@408,20%@408,24%@408] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,10%@204,9%@204,15%@204] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,13%@204,15%@204,18%@204] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,15%@408,11%@408,17%@408] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [19%@102,14%@102,9%@102,21%@102] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,18%@204,22%@204,19%@204] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [17%@306,17%@306,17%@306,22%@306] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [18%@408,19%@408,12%@408,24%@408] | |||
RAM 1876/3976MB (lfb 224x4MB) CPU [12%@306,10%@306,12%@306,24%@306] -------------> End capturing | |||
RAM 1821/3976MB (lfb 224x4MB) CPU [17%@1734,12%@1734,15%@1734,23%@1734] | |||
RAM 1821/3976MB (lfb 224x4MB) CPU [2%@102,7%@102,4%@102,0%@102] | |||
</pre> | </pre> | ||
The average for each core while single capturing is: [15.57%, 13.71%, 15.14%, 19.14%] and the memory is 1876MB for the total system, for only the single capture the armload is [6.82%, 3.22%, 11.6%, 6.35%] for each core and takes 56MB of memory. | |||
== | ====Dual capture==== | ||
These measures was taken with nvcamerasrc and only the capture system and two cameras as the pipeline below: | |||
<pre> | <pre> | ||
gst-launch-1.0 nvcamerasrc sensor-id=0 auto-exposure=1 aeLock=1 ! perf ! fakesink nvcamerasrc sensor-id=1 auto-exposure=1 aeLock=1 ! perf ! fakesink | |||
</pre> | |||
<pre> | |||
nvidia@tegra-ubuntu:~$ ./tegrastats | |||
RAM 1792/3976MB (lfb 199x4MB) CPU [4%@1224,12%@1224,0%@1224,7%@1224] | |||
RAM 1814/3976MB (lfb 199x4MB) CPU [12%@1734,6%@1734,49%@1734,0%@1734] | |||
RAM 2425/3976MB (lfb 139x4MB) CPU [29%@1734,42%@1734,59%@1734,55%@1734] -----> Start Capture | |||
RAM 2428/3976MB (lfb 137x4MB) CPU [18%@306,17%@306,23%@306,23%@306] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,28%@714,20%@714,30%@714] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,16%@408,22%@408,25%@408] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [23%@204,24%@204,22%@204,25%@204] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [17%@408,20%@408,23%@408,23%@408] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,18%@714,32%@714,28%@714] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,17%@408,20%@408,23%@408] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [22%@612,21%@612,28%@612,28%@612] | |||
RAM 2429/3976MB (lfb 136x4MB) CPU [13%@408,20%@408,24%@408,22%@408] | |||
RAM 2431/3976MB (lfb 136x4MB) CPU [30%@1224,20%@1224,24%@1224,26%@1224] | |||
RAM 2431/3976MB (lfb 136x4MB) CPU [23%@510,12%@510,20%@510,29%@510] | |||
RAM 2431/3976MB (lfb 136x4MB) CPU [27%@510,18%@510,15%@510,26%@510] | |||
RAM 2431/3976MB (lfb 136x4MB) CPU [21%@306,23%@306,19%@306,24%@306] ---------> End Capture | |||
RAM 1822/3976MB (lfb 223x4MB) CPU [11%@1224,28%@1224,20%@1224,11%@1224] | |||
RAM 1822/3976MB (lfb 223x4MB) CPU [4%@102,6%@102,0%@102,4%@102] | |||
RAM 1822/3976MB (lfb 223x4MB) CPU [1%@102,6%@102,0%@102,8%@102] | |||
RAM 1822/3976MB (lfb 223x4MB) CPU [5%@102,6%@102,0%@102,1%@102] | |||
RAM 1822/3976MB (lfb 224x4MB) CPU [5%@102,8%@102,0%@102,2%@102] | |||
</pre> | </pre> | ||
The average for each core while dual capturing is: [22.64%, 21.14%, 25.07%, 27.64%], and the memory is 2429MB for the total system, for only the dual capture the armload is [15.37%, 18.85%, 22.81%, 22.81%] for each core and takes 609MB of memory. | |||
{{ContactUs}} | |||
[[Category:Jetson]] | [[Category:Jetson]][[Category:Jetson V4L2 Drivers]][[Category:Sony]] |
Latest revision as of 21:24, 18 September 2024
Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help. |
Sony IM230 CMOS image sensor features
The IMX230 is a CMOS image sensor with the following features:
- Phase Detection Auto Focus (PDAF)
- Single Frame High Dynamic Range (HDR) with equivalent full pixels
- High signal to noise ratio (SNR)
- Full resolution @24fps (Normal / HDR). 4K2K @30fps (Normal / HDR) 1080p @60fps (Normal / HDR)
- Independent flipping and mirroring
- CSI-2 serial data output (MIPI 2lane/4lane, Max. 1.5Gbps/lane, D-PHY spec. ver. 1.1 compliant)
- CSI2 serial data output (MIPI interface 2 lanes)
- Output video format of RAW10/8, COMP8/6
- Advanced Noise Reduction (Chroma noise reduction and RAW noise reduction)
Official Manufacturer IMX230 camera sensor documentation link: IMX230 21-megapixel product brief
RidgeRun has developed a driver for the Jetson TX1 platform with the following support:
- Jetpack 3.1 and 2.3.1 support.
- V4l2 Media controller driver
- Tested resolution 5344x4016 @ 21 fps. Also cropped 4208x3120, 2672x2008 and 1920x1080 .
- Output format: RAW10 Bayer BGGR pattern.
- Leopard Imaging CB boards for TX1
- Capture with v4l2src and nvcamerasrc
- Support for triple video capture. (Tested using the 3 available ports LI CB)
- BU64295/BU64297 VCM driver support
Enabling the driver
In order to use this driver, you have to patch and compile the kernel source, and there are two ways to do it:
Using RidgeRun SDK
Through the SDK you can easily patch the kernel and generate an image with the required changes to get the IMX230 sensor to work. In this wiki Getting Started Guide for Jetson TX1 you can find all the information required to build a Jetson TX1 SDK from scratch.
In order to add the IMX230 driver follow these steps:
- Go to your SDK directory
- Go to the kernel directory
- Copy the patches in the patches directory
001-add-li-cb-board-driver.patch 002-add-driver-imx230-camera.patch 003-imx230-custom-dtb-li-cb.patch
- Modify the series file in the kernel directory. You have to add the 3 above patches.
- Run make config and select the IMX230 driver in the Kernel Configuration like this:
-> Kernel Configuration -> Device Drivers -> Multimedia support -> Encoders, decoders, sensors, and other helper chips -> <*> IMX230 camera sensor support
- Then make the SDK and install it following the Started Guide mentioned before
Using Jetpack
- Follow the instructions in Compiling_Tegra_X1/TX2_source_code (Downloading the code) to get the kernel source code.
- If you didn't have downloaded the toolchain yet, follow the instructions in Compiling_Tegra_TX1/TX2_source_code (Toolchain)
- Once you have the source code, apply the patches to fix kernel error during compilation and add the support for the IMX230:
cd $DEVDIR/64_TX1/Linux_for_Tegra_64_tx1/sources/ PATCHES=<path to the tarball> tar -xzf $PATCHES/imx230-driver-for-tegra-x1-patches.tar.gz mv imx230-driver-for-tegra-x1-patches patches quilt push -a
Please change the path in PATCHES for the one where you downloaded the tarball with the IMX230 patches.
- Follow the instructions in Compiling Jetson_TX1/TX2 source_code (Build Kernel) for building the kernel, and then flash the image. Omit steps 3 and 4 because you already applied the patches.
Make sure to enable IMX230 and driver support in step 5:
-> Device Drivers -> Multimedia support -> Encoders, decoders, sensors, and other helper chips -> <*> Omnivision IMX230 camera sensor support
- To flash the built image into the board follow the instructions in Compiling_Jetson_TX1/TX2_source_code(Flashing_the_board)
Using the driver (GStreamer examples)
The GStreamer version distributed with Jetpack doesn't support Bayer RAW10 only RAW8 so GStreamer needs to be patched in order to capture using v4l2src. Follow the steps in the following wiki page to add the support for RAW10:
Compile_gstreamer_on_Jetson_TX1_and_TX2
Important Note: When you are accessing the board through serial or ssh and you want to run a pipeline to display with autovideosink, nveglglessink, xvimagesink or any other video sink, you have to run your pipeline with DISPLAY=:0 at the beginning of the description: |
DISPLAY=:0 gst-launch-1.0 ...
Snapshots
In order to check the snapshot, you can use the following tool:
https://github.com/jdthomas/bayer2rgb
So, run the following commands to download the tool and compile it:
git clone git@github.com:jdthomas/bayer2rgb.git cd bayer2rgb make cp bayer2rgb /usr/bin/
Bayer2rgb will convert naked (no header) Bayer grid data into RGB data. There are several choices of interpolation (though they all look essentially the same to my eye). It can output tiff files and can integrate with ImageMagick to output other formats.
- 1920x1080
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=1 ! "video/x-bayer, format=bggr, width=1920, height=1080" \ ! multifilesink location=test%d_1920x1080.bayer
Check the snapshot with:
./bayer2rgb --input=test#_1920x1080.bayer --output=data.tiff --width=5344 --height=4016 --bpp=16 --first=BGGR \ --method=BILINEAR --tiff
Use image_magik to convert the tiff to png:
convert data.tiff data.png
Important Note 1: In general the first buffer contains very low light because the AWB algorithm of the sensor is calibrating, so we recommend using multifilesink to test debayer with a buffer above from number one. To obtain better image colors and bright quality, due to automatic sensor image calibration, we recommend to test debayer with a frame above number 10, to give time to the sensor to adjust the best image calibration parameters. |
Important Note 2: The debayered image obtained as the output when using the "bayer2rgb" tool presents some kind of light saturation, viewed as multiple color pixels sections. This is a problem of the tool used to do the debayer process but led the users to verify that the driver and camera sensor is working fine. |
Snapshots with nvcamerasrc
The following pipeline will create a file for each captured frame. You can visualize the file on the following web page: http://rawpixels.net/
gst-launch-1.0 -v nvcamerasrc sensor-id=1 fpsRange="30 30" num-buffers=100 ! 'video/x-raw(memory:NVMM), width=(int)5344, \ height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw, width=(int)5344, \ height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! multifilesink location=test_%d.yuv
Single Capture
V4l2src
You can use the raw2rgbpnm tool to check all the buffers:
https://github.com/martinezjavier/raw2rgbpnm
So, run the following commands to download the tool and compile it:
git clone git clone git@github.com:martinezjavier/raw2rgbpnm.git cd raw2rgbpnm
Open the file raw2rgbpnm.c and change line 489 with:
int c = getopt(argc, argv, "a:b:f:ghs:wn");
This is to enable the option to extract multiple frames from a file. Now, you can build the application:
make
Important Note: This tool converts from GRBG10 to pnm. We capture RGGB in the IMX230, so you will see that the colors at the output of the image are wrong. |
In order to capture 10 buffers and save them in a file, you can run the following pipelines:
- 5344x4016
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=10 ! "video/x-bayer, format=rggb, width=5344, height=4016" \ ! filesink location=test_5344x4016.bayer
Check the buffers with:
./raw2rgbpnm -f SRGGB10 -s 5344x4016 -b 5.0 -n test_5344x4016.bayer output_5344x4016
Nvcamerasrc
- 5344x4016
DISPLAY=:0 gst-launch-1.0 nvcamerasrc sensor-id=0 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, \ height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvegltransform ! nveglglessink -e
3 Camera Capture
The following image consists of a Jetson TX1 with three IMX230 cameras connected at the board
V4l2src
- Pipeline for sextuple video capture using v4l2src, at 5344x4016 @21fps:
gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \ v4l2src device=/dev/video1 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \ v4l2src device=/dev/video2 ! 'video/x-bayer,format=bggr,width=5344,height=4016' ! fakesink \
Nvcamerasrc
- Pipeline for triple video capture using nvcamerasrc, at 5344x4016 @21fps:
DISPLAY=:0 gst-launch-1.0 nvcamerasrc exposure-time=0.006 sensor-id=0 fpsRange="30 30" auto-exposure=off ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1" ! queue ! mixer.sink_0 nvcamerasrc auto-exposure=off exposure-time=0.006 sensor-id=1 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420,framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1" ! queue ! mixer.sink_1 nvcamerasrc auto-exposure=off exposure-time=0.006 sensor-id=2 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)5344, height=(int)4016, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! "video/x-raw,width=(int)640,height=(int)480,format=(string)I420,framerate=(fraction)30/1" ! queue ! mixer.sink_2 videomixer background=1 name=mixer sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=640 sink_1::ypos=0 sink_2::xpos=320 sink_2::ypos=480 ! queue ! nvoverlaysink sync=false -v
Performance
For this test framerate was measured with vl4-ctl and perf using nvcamerasrc to capture
Framerate with v4l
This test was developed with v4l-util tools to capture a raw image and measure the framerate.
nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=5344,height=4016,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap <<<<<<<<<<<<<<<<<<<<< 19.60 fps <<<<<<<<<<<<<<<<<<<<< 20.39 fps <<<<<<<<<<<<<<<<<<<<<< 20.72 fps <<<<<<<<<<<<<<<<<<<<< 20.89 fps <<<<<<<<<<<<<<<<<<<<< 20.95 fps <<<<<<<<<<<<<<<<<<<<<< 21.02 fps <<<<<<<<<<<<<<<<<<<<< 21.08 fps <<<<<<<<<<<<<<<<<<<<< 21.12 fps <<<<<<<<<<<<<<<<<<<<<< 21.15 fps <<<<<<<<<<<<<<<<<<<<< 21.15 fps <<<<<<<<<<<<<<<<<<<<< 21.18 fps <<<<<<<<<<<<<<<<<<<<<< 21.19 fps <<<<<<<<<<<<<<<<<<<<< 21.19 fps <<<<<<<<<<<<<<<<<<<<< 21.21 fps <<<<<<<<<<<<<<<<<<<<<< 21.22 fps <<<<<<<<<<<<<<<<<<<<< 21.22 fps <<<<<<<<<<<<<<<<<<<<< 21.23 fps
The sensor provides the capture at 21fps with v4l.
Framerate with Nvcamerasrc and perf
These test was developed with nvcamerasrc, the results are showed below:
gst-launch-1.0 nvcamerasrc auto-exposure=1 aeLock=1 ! perf ! autovideosink Setting pipeline to PAUSED ... Available Sensor modes : 5344 x 4016 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 4208 x 3120 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 2672 x 2008 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 1920 x 1080 FR=21.000000 CF=0x1009208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10 Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 0 WxH = 5344x4016 FrameRate = 21.000000 ... New clock: GstSystemClock GST-PERF INFO --> Timestamp: 0:20:01.076040876; Bps: 0; fps: 0.0 GST-PERF INFO --> Timestamp: 0:20:02.085110495; Bps: 800; fps: 21.80 GST-PERF INFO --> Timestamp: 0:20:03.108538791; Bps: 789; fps: 21.50 GST-PERF INFO --> Timestamp: 0:20:04.141721657; Bps: 782; fps: 21.29 GST-PERF INFO --> Timestamp: 0:20:05.181324930; Bps: 777; fps: 21.17 GST-PERF INFO --> Timestamp: 0:20:06.218568207; Bps: 779; fps: 21.21 GST-PERF INFO --> Timestamp: 0:20:07.251035501; Bps: 782; fps: 21.31 GST-PERF INFO --> Timestamp: 0:20:08.282084777; Bps: 783; fps: 21.33 GST-PERF INFO --> Timestamp: 0:20:09.313816032; Bps: 783; fps: 21.33 GST-PERF INFO --> Timestamp: 0:20:10.346620670; Bps: 782; fps: 21.31
With nvcamerasrc the framerate is 21fps.
Arm load while capturing
The results showed below was taken with one camera capturing and using the pipeline mentioned on the "Framerate with Nvcamerasrc and perf" section in this wiki
Single capture
These measures was taken with nvcamerasrc and only the capture system and one camera as pipeline below:
gst-launch-1.0 nvcamerasrc sensor-id=0 ! perf ! fakesink
RAM 1820/3976MB (lfb 224x4MB) CPU [1%@204,10%@204,3%@204,3%@204] RAM 1823/3976MB (lfb 224x4MB) CPU [6%@1734,25%@1734,11%@1734,24%@1734] RAM 1875/3976MB (lfb 224x4MB) CPU [36%@306,12%@306,28%@306,8%@306] --------------> start capturing RAM 1876/3976MB (lfb 224x4MB) CPU [17%@204,13%@204,12%@204,16%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [9%@306,14%@306,18%@306,21%@306] RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,10%@408,15%@408,19%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [16%@204,11%@204,12%@204,20%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [8%@408,16%@408,20%@408,24%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,10%@204,9%@204,15%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,13%@204,15%@204,18%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [12%@408,15%@408,11%@408,17%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [19%@102,14%@102,9%@102,21%@102] RAM 1876/3976MB (lfb 224x4MB) CPU [14%@204,18%@204,22%@204,19%@204] RAM 1876/3976MB (lfb 224x4MB) CPU [17%@306,17%@306,17%@306,22%@306] RAM 1876/3976MB (lfb 224x4MB) CPU [18%@408,19%@408,12%@408,24%@408] RAM 1876/3976MB (lfb 224x4MB) CPU [12%@306,10%@306,12%@306,24%@306] -------------> End capturing RAM 1821/3976MB (lfb 224x4MB) CPU [17%@1734,12%@1734,15%@1734,23%@1734] RAM 1821/3976MB (lfb 224x4MB) CPU [2%@102,7%@102,4%@102,0%@102]
The average for each core while single capturing is: [15.57%, 13.71%, 15.14%, 19.14%] and the memory is 1876MB for the total system, for only the single capture the armload is [6.82%, 3.22%, 11.6%, 6.35%] for each core and takes 56MB of memory.
Dual capture
These measures was taken with nvcamerasrc and only the capture system and two cameras as the pipeline below:
gst-launch-1.0 nvcamerasrc sensor-id=0 auto-exposure=1 aeLock=1 ! perf ! fakesink nvcamerasrc sensor-id=1 auto-exposure=1 aeLock=1 ! perf ! fakesink
nvidia@tegra-ubuntu:~$ ./tegrastats RAM 1792/3976MB (lfb 199x4MB) CPU [4%@1224,12%@1224,0%@1224,7%@1224] RAM 1814/3976MB (lfb 199x4MB) CPU [12%@1734,6%@1734,49%@1734,0%@1734] RAM 2425/3976MB (lfb 139x4MB) CPU [29%@1734,42%@1734,59%@1734,55%@1734] -----> Start Capture RAM 2428/3976MB (lfb 137x4MB) CPU [18%@306,17%@306,23%@306,23%@306] RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,28%@714,20%@714,30%@714] RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,16%@408,22%@408,25%@408] RAM 2429/3976MB (lfb 136x4MB) CPU [23%@204,24%@204,22%@204,25%@204] RAM 2429/3976MB (lfb 136x4MB) CPU [17%@408,20%@408,23%@408,23%@408] RAM 2429/3976MB (lfb 136x4MB) CPU [22%@714,18%@714,32%@714,28%@714] RAM 2429/3976MB (lfb 136x4MB) CPU [25%@408,17%@408,20%@408,23%@408] RAM 2429/3976MB (lfb 136x4MB) CPU [22%@612,21%@612,28%@612,28%@612] RAM 2429/3976MB (lfb 136x4MB) CPU [13%@408,20%@408,24%@408,22%@408] RAM 2431/3976MB (lfb 136x4MB) CPU [30%@1224,20%@1224,24%@1224,26%@1224] RAM 2431/3976MB (lfb 136x4MB) CPU [23%@510,12%@510,20%@510,29%@510] RAM 2431/3976MB (lfb 136x4MB) CPU [27%@510,18%@510,15%@510,26%@510] RAM 2431/3976MB (lfb 136x4MB) CPU [21%@306,23%@306,19%@306,24%@306] ---------> End Capture RAM 1822/3976MB (lfb 223x4MB) CPU [11%@1224,28%@1224,20%@1224,11%@1224] RAM 1822/3976MB (lfb 223x4MB) CPU [4%@102,6%@102,0%@102,4%@102] RAM 1822/3976MB (lfb 223x4MB) CPU [1%@102,6%@102,0%@102,8%@102] RAM 1822/3976MB (lfb 223x4MB) CPU [5%@102,6%@102,0%@102,1%@102] RAM 1822/3976MB (lfb 224x4MB) CPU [5%@102,8%@102,0%@102,2%@102]
The average for each core while dual capturing is: [22.64%, 21.14%, 25.07%, 27.64%], and the memory is 2429MB for the total system, for only the dual capture the armload is [15.37%, 18.85%, 22.81%, 22.81%] for each core and takes 609MB of memory.
RidgeRun Resources | |||||
Contact Us
|