View source for NVIDIA Jetson ISP Control

{{#seo:
|title=NVIDIA Jetson TX1 TX2 ISP Control with Nvcamerasrc
|title_mode=replace
|description={{{description|This page is an introduction on how to use ISP with the white balance property of the nvcamerasrc element. Read about this Bayer to YUV ISP converter.}}}
}}
{{Mpo-shoppingcart & NVIDIA logo & RR Contact}}
<br>
<td>
{{GStreamer debug}}
</td>

== NVIDIA Jetson ISP Control Description ==

This page is an introduction to changing the Jetson TX1/TX2/Xavier/Nano ISP configuration with the nvcamerasrc element.

=== NVIDIA ISP Overview ===
Image signal processor (ISP) has the ability to convert from Bayer to YUV. Jetson TX1/TX2/Xavier/Nano has two ISP, nvcamerasrc element was created by NVIDIA and it has access to the ISP.
<br>
<br>
[[Image:ISP.png|thumb|center|455px|ISP.png]] 
<div style="text-align: center;">'''Figure 1'''. ISP bayer to yuv</div> 
<br>

=== White balance modes on NVcamerasrc ===
White balance is the process of removing unrealistic color casts, that affect the color temperature of the photo, which refers to the relative warmth or coolness of white light. The human eyes can judge what is white under different light sources, however, digital cameras have great difficulty with auto white balance. Nvcamerasrc element has a white balance property (wbmode)

{| border="1" cellpadding="2"


{| style="margin: 1em auto 1em auto;" class="wikitable"
|+ Colors temperature effects with a light source
|-
!width="100"|Color Temperature (K)
!width="250"|Light Source
|-
| 1000-2000
| Candlelight 
|-
| 2500-3500
| Tungsten Bulb
|-
| 3000-4000 
| Sunrise/Sunset
|-
| 4000-5000
| Fluorescent Lamps
|-
| 5000-5500 
| Electronic Flash
|-
| 5000-6500
| Daylight with Clear Sky
|-
| 6500-8000 
| Moderately Overcast Sky
|-
| 9000-10000 
| Shades or Heavily Overcast Sky
|}


Nvcamerasrc can set 10 white balance modes (default wbmode=1  auto white balance ) to adjust the image colors as follows:

*                           (0): off              
*                           (1): auto             
*                           (2): incandescent     
*                           (3): fluorescent      
*                           (4): warm-fluorescent 
*                           (5): daylight         
*                           (6): cloudy-daylight  
*                           (7): twilight         
*                           (8): shade            
*                           (9): manual

All of those modes are tested with an IMX219 sensor using a J20 module for Jetson TX1 with the driver that Ridgerun created.

==== Off mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=0 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_0.jpeg|thumb|center|455px|imx219_0.jpeg]] 
<div style="text-align: center;">'''Figure 2'''. off white balance mode with imx219 sensor</div>

==== Auto mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=1 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_1.jpeg|thumb|center|455px|imx219_1.jpeg]] 
<div style="text-align: center;">'''Figure 3'''. auto white balance mode with imx219 sensor</div>

==== Incandecent mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=2 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_2.jpeg|thumb|center|455px|imx219_2.jpeg]] 
<div style="text-align: center;">'''Figure 4'''. incandecent white balance mode with imx219 sensor</div>

==== Fluorescent mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=3 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_3.jpeg|thumb|center|455px|imx219_3.jpeg]] 
<div style="text-align: center;">'''Figure 5'''. fluorescent white balance mode with imx219 sensor</div>

==== Warm-fluorescent mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=4 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_4.jpeg|thumb|center|455px|imx219_4.jpeg]] 
<div style="text-align: center;">'''Figure 6'''. warm-fluorescent white balance mode with imx219 sensor</div>

==== Daylight mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=5 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_5.jpeg|thumb|center|455px|imx219_5.jpeg]] 
<div style="text-align: center;">'''Figure 7'''. daylight white balance mode with imx219 sensor</div> 
<br>

==== Cloudy-daylight   mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=6 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_6.jpeg|thumb|center|455px|imx219_6.jpeg]]
<div style="text-align: center;">'''Figure 8'''. cloudy-daylight   white balance mode with imx219 sensor</div>
 
==== Twilight mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=7 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_7.jpeg|thumb|center|455px|imx219_7.jpeg]] 
<div style="text-align: center;">'''Figure 9'''. twilight white balance mode with imx219 sensor</div> 

==== Shade mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=8 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_8.jpeg|thumb|center|455px|imx219_8.jpeg]] 
<div style="text-align: center;">'''Figure 10'''. shade white balance mode with imx219 sensor</div> 

==== Manual mode ====

<pre>
gst-launch-1.0 nvcamerasrc wbmode=9 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:imx219_9.jpeg|thumb|center|455px|imx219_9.jpeg]] 
<div style="text-align: center;">'''Figure 11'''. Manual white balance mode with imx219 sensor</div> 

=== Other properties of nvcamerasrc ===

====Contrast====
Contrast is the separation between the darkest and brightest areas of the image. If you increase contrast, you make shadows darker and highlights brighter. Nvcamerasrc have the ability to increase and decrease contrast in the float interval [0-1]

=====Minimum contrast=====

<pre>
gst-launch-1.0 nvcamerasrc contrast=0 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)3280, height=(int)2464, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Contrast0.png|thumb|center|455px|Contrast0.png]] 
<div style="text-align: center;">'''Figure 21'''. Contrast set at 0 with nvcamerasrc element using imx219 sensor</div> 

=====Maximum contrast=====

<pre>
gst-launch-1.0 nvcamerasrc contrast=1 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)3280, height=(int)2464, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Contrast1.png|thumb|center|455px|Contrast1.png]] 
<div style="text-align: center;">'''Figure 22'''. Contrast set at 1 with nvcamerasrc element using imx219 sensor</div> 

====Saturation====

Saturation is the property to describe the intensity of color in the image. A saturated image has overly bright colors. Using nvcamerasrc you can set saturation, that property works in 0 - 2 float intervals. 

=====Minimum saturation=====
<pre>
gst-launch-1.0 nvcamerasrc saturation=0 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Saturation0.jpeg|thumb|center|455px|Saturation0.jpeg]] 
<div style="text-align: center;">'''Figure 12'''. Saturation set at 0 with nvcamerasrc element using imx219 sensor</div> 

=====Maximum saturation=====

<pre>
gst-launch-1.0 nvcamerasrc saturation=2 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Saturation2.jpeg|thumb|center|455px|Saturation2.jpeg]] 
<div style="text-align: center;">'''Figure 13'''. Saturation set at 2 with nvcamerasrc element using imx219 sensor</div> 

====Color effect====
With nvcamersrc you can create images with amazing color effects, it has 7 modes:
* (1): off 
* (2): mono          
* (3): negative      
* (4): solarize     
* (5): sepia         
* (6): posterize      
* (7): aqua

=====Off color effect=====

<pre>
gst-launch-1.0 nvcamerasrc color-effect=1 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Color-effect1.jpeg|thumb|center|455px|Color-effect1.jpeg]] 
<div style="text-align: center;">'''Figure 14'''. Color effect set at 1 with nvcamerasrc element using imx219 sensor</div> 

=====Mono color effect=====

<pre>
gst-launch-1.0 nvcamerasrc color-effect=2 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Color-effect2.jpeg|thumb|center|455px|Color-effect2.jpeg]] 
<div style="text-align: center;">'''Figure 15'''.Color effect set at 2 with nvcamerasrc element using imx219</div> 

=====Negative color effect=====

<pre>
gst-launch-1.0 nvcamerasrc color-effect=3 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Color-effect3.jpeg|thumb|center|455px|Color-effect3.jpeg]] 
<div style="text-align: center;">'''Figure 16'''. Saturation set at 3 with nvcamerasrc element using imx219 sensor</div> 

=====Solarize color effect=====

<pre>
gst-launch-1.0 nvcamerasrc color-effect=4 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Color-effect4.jpeg|thumb|center|455px|Color-effect4.jpeg]] 
<div style="text-align: center;">'''Figure 17'''. Saturation set at 4 with nvcamerasrc element using imx219 sensor</div> 

=====Sepia color effect=====

<pre>
gst-launch-1.0 nvcamerasrc color-effect=5 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Color-effect5.jpeg|thumb|center|455px|Color-effect5.jpeg]] 
<div style="text-align: center;">'''Figure 18'''. Saturation set at 5 with nvcamerasrc element using imx219 sensor</div> 

=====Posterize color effect=====

<pre>
gst-launch-1.0 nvcamerasrc color-effect=6 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Color-effect6.jpeg|thumb|center|455px|Color-effect6.jpeg]] 
<div style="text-align: center;">'''Figure 19'''. Saturation set at 6 with nvcamerasrc element using imx219 sensor</div> 

=====Aqua color effect=====

<pre>
gst-launch-1.0 nvcamerasrc color-effect=7 fpsRange="30.0 30.0" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvoverlaysink -e
</pre>

[[Image:Color-effect7.jpeg|thumb|center|455px|Color-effect7.jpeg]] 
<div style="text-align: center;">'''Figure 20'''. Saturation set at 7 with nvcamerasrc element using imx219 sensor</div> 

==== Scene ====

With nvcamersrc you can adjust the colors depending on the scene and the object that you want to focus on, it has 16 modes: 
*                           (0): face-priority   
*                           (1): action          
*                           (2): portrait        
*                           (3): landscape       
*                           (4): night           
*                           (5): night-portrait   
*                           (6): theatre          
*                           (7): beach            
*                           (8): snow             
*                           (9): sunset           
*                           (10): steady-photo     
*                           (11): fireworks        
*                           (12): sports           
*                           (13): party            
*                           (14): candle-light     
*                           (15): barcode

=== Custom ISP Configuration ===

Companies with access to the ISP can create a custom ISP file configuration to calibrate correctly a specific sensor based on its own parameters Below is the procedure to follow:

* 1) When you develop a driver on the Jetson (TX1/TX2/Xavier/Nano) you are able to capture it in 2 different ways. One is using the V4L2 standard API with applications such as GStreamer (v4l2src), v4l2-ctl or yavta. This is direct capture and the Jetson will not do any color conversion or change in the imaging, the format is kept as given by the sensor. The second way is using the NVIDIA proprietary software, such as GStreamer nvcamerasrc/nvarguscamerasrc elements or libargus, when using these applications, the frames will pass through the internal Jetson's ISP, performing the debayering process (conversion from RAW to YUV format) and color conversions/corrections based on ISP configurable parameters.

* 2) These ISP configurable parameters can be managed and changed in 2 ways, one is using the libargus or nvcamerasrc/nvarguscamerasrc properties and options (run gst-inspect-1.0 nvcamerasrc to see the available properties) such as white balance, auto exposure, saturation, and color effects, changing the ISP configuration at runtime and applying filters to the image. The second way is creating a file called camera_overrides.isp, which contains configuration and tuned parameters that the ISP uses to execute the internal algorithms.

* 3) This camera_overrides.isp file can be created with all the tuned configurations for a specific sensor or scene, in order for the images to look exactly how the customer wants them to look. NVIDIA provides access to the creation and tuning of this file only to ODMs such as D3 or Leopard Imaging, and they have all the tools and procedures to tune the ISP for any sensor, doing calibration on the Black level, Lens shading, White Balance, and Color correction. RidgeRun doesn't provide support on ISP tuning but only on the driver development side.

* 4) Once the driver is ready and the capture is properly done, in case the customer wants to tune up the image, they can contact [https://www.d3engineering.com/ D3 Engineering] or leopard imaging. You can mention RidgeRun recommendation on this task. For D3 Engineering, you can contact them directly or you can send us an email to  [mailto:support@ridgerun.com <u>'''support@ridgerun.com'''</u>] and we will put them in contact with you. 

In order to apply a new ISP calibration you will need to follow the next steps:

'''1.''' If there are any '''nvcam_cache_#.bin''' and '''serial_no_#.bin''' files under '''/var/nvidia/nvcam/settings''' on Jetson, delete them.
 sudo rm /var/nvidia/nvcam/settings/nvcam_cache_*
 sudo rm /var/nvidia/nvcam/settings/serial_no_*
'''2.''' Copy camera_overrides.isp in the correct location:
 cp '''<path>'''/camera_overrides.isp /var/nvidia/nvcam/settings
'''3.''' Do below commands:
 sudo chmod 664 /var/nvidia/nvcam/settings/camera_overrides.isp
 sudo chown root:root /var/nvidia/nvcam/settings/camera_overrides.isp

== ISP Bandwidth ==

When talking about capture, debayering (RAW to YUV conversion), and encoding in Jetson it is important to consider the bandwidth of each of the stages involved. Basically, there are 3 parts typically:
 MIPI CSI Bandwidth > Encoder Bandwidth > ISP Bandwidth. 
This section provides a brief summary about this bandwidth but more information can be found in the SoC datasheet which can be downloaded from:
 https://developer.nvidia.com/embedded/downloads 

=== Jetson Nano ===

Dedicated RAW to YUV processing engines processes up to 1400Mpix/s (up to 24MP sensor) | MIPI CSI 2.0 up to 1.5Gbps (per lane) | Support for x4 and x2 configurations (up to four active streams).

In the case of the encoder you can find statistics here at [[Jetson_Nano/Introduction/Overview|NVIDIA Jetson Nano SoM Overview]]

=== Jetson TX2 ===

Dedicated RAW to YUV processing engine process up to 1.4Gpix/s | MIPI CSI 2.0 up to 2.5Gbps (per lane) | Support for x4 and x2 configurations (up to 3 x4-lane or 6 x2-lane cameras)

In the case of the encoder, you can find statistics here at [[JetsonTX2/Getting Started | JetsonTX2 Getting Started]]

=== Jetson Xavier ===
At the moment of writing this section, it was not possible to find the ISP bandwidth for Xavier. However, it is possible to assume that it is similar to the TX2 (1.4Gpix/s). In [https://en.wikichip.org/wiki/nvidia/tegra/xavier external sources] references to 1.5Gpix/s can be found. The encoder is limited to 4x 4Kp60, so that is around 1.99Gpix/s. 

You can find statistics here at [[Xavier/Introduction/SoM_Overview | Xavier SoM_Overview]]

== MIPI CSI Bandwidth ==
It could be estimated with this formula (from TX1 datasheet):
<pre>
fps = (efficiency) * (#Lanes)*(Lane Speed in bps) /  ( (Width*Height) *(bpp) ) 
</pre>
As an example, for 1.5Gbps, 1 lane, and 0.85 efficiency due to blanking and protocol, 18.75fps should be possible for an 8MP sensor. However, it is important to mention that these numbers need to be verified with a proof of concept.

== Check ISP uses  ==
The ISP uses could be checked by using the <code> v4l2-ctl -d /dev/video# --get-ctrl bypass_mode </code> command. For example:

<pre>
v4l2-ctl -d /dev/video0 --get-ctrl bypass_mode
</pre>

If the ISP is used, the output should show the following:

<pre>
bypass_mode: 1
</pre>
{{ContactUs}}

[[Category:GStreamer]][[Category:Jetson]][[Category:JetsonNano]][[Category:JetsonTX2]][[Category:NVIDIA Xavier]][[Category:JetsonXavierNX]][[Category:NVIDIA Jetson Orin]]