Jetson TX1/TX2 Multi Camera Exposure Feedback Control

Introduction

This document is created with the goal to solve color differences capturing from multiple cameras using NVIDIA's ISP.

Nvcamerasrc overview

Nvcamerasrc element use the ISP to improve image colors and light exposure. By default when you run a pipeline with nvcamerasrc, the automatic gain and exposure controls are enabled.

The ISP use a configuration file created by NVIDIA and modify gain, coarse time and frame length controls on the driver to executes the auto-exposure. This means that when the auto-exposure property is enabled, the ISP processes the image calculating the gain and exposure values automatically and then it sets the new values to the v4l2-controls exposed by the driver like coarse time, gain and frame length depending on the minimum and maximum for those controls. If the auto-exposure is not enabled, that feedback from the ISP to the sensor is not applied.

Multicapture Issue

Drivers configured to capture from multiple cameras can use the auto-white balance and auto-exposure properties from Nvcamersrc element. However, the resulting images from the different sensors will have different colors depending on the light conditions. Despite all the cameras from different positions will use the same ISP configuration file, the external variables from the environment determine the correct colors for each image. Thus, the all cameras needs to point in the same direction in order to provide the same colors for the each image captured.

3 cameras capture with auto-exposure example

This example will capture from 3 cameras using auto-exposure mode.

gst-launch-1.0 \
nvcamerasrc sensor-id=0 fpsRange="30/30" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_0.ts \
nvcamerasrc sensor-id=1 fpsRange="30/30" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_1.ts \
nvcamerasrc sensor-id=2 fpsRange="30/30" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_2.ts

You can observe the the light difference between the center camera and the other two in the resultant three images below.

3 cameras capture with auto-exposure disabled example

Nvcamerasrc expose different properties to control the ISP, it's possible to disable automatic controls to remove this colors difference. To do that you need to disable and block auto-exposure (auto-exposure=1 aeLock=true)

This example will capture from 3 cameras using auto-exposure disabled.

gst-launch-1.0 \
nvcamerasrc auto-exposure=1 aeLock=true sensor-id=0 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_0.ts \
nvcamerasrc auto-exposure=1 aeLock=true sensor-id=1 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_1.ts \
nvcamerasrc auto-exposure=1 aeLock=true sensor-id=2 fpsRange="30 30" ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_2.ts

In the resultant images bellow you can see that the light exposure is the same for the three, but the ISP is not used and the colors are very dark. Also there is a little difference in white-balance between the three images.

3 cameras capture with auto-exposure and white-balance disabled example

Disabling auto-white-balance we can get the same colors for the three images.

This example will capture from 3 cameras using auto-exposure and auto-white-balance disabled.

gst-launch-1.0 \
nvcamerasrc wbmode=9 auto-exposure=1 aeLock=true sensor-id=0 fpsRange="30 30" ! 'video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080,format=(string)I420,framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_0.ts \
nvcamerasrc wbmode=9 auto-exposure=1 aeLock=true sensor-id=1 fpsRange="30 30" ! 'video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080, format=(string)I420,framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_1.ts \
nvcamerasrc wbmode=9 auto-exposure=1 aeLock=true sensor-id=2 fpsRange="30 30" ! 'video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080,format=(string)I420,framerate=(fraction)30/1' ! \
omxh264enc ! mpegtsmux ! filesink location=test_2.ts

In the resultant images bellow you can see that the light exposure and colors are the same for the three, but the ISP is not used and the colors are still very dark.

Controls Transfer

RidgeRun offers a mechanism to have the same gain and exposure levels for all the cameras by setting the middle camera in auto-exposure mode and reading the gain, coarse time and frame length controls directly from the driver, then, transferring the values to the other cameras with the auto-exposure disabled.

The following are results obtained with the Multi Camera Exposure Feedback Control:

3 camera capture with feedback tuning example

In this example, we capture from 3 cameras and transfer the controls from the center camera in auto-exposure mode to the left and right cameras in manual mode. For this implementation, left and right cameras has the auto-exposure disabled and blocked.

In the resultant 3 images bellow you can see that the light exposure is the same for the three. The controls are received from the center camera and applied to the other two with success. However a little difference in white-balance is presented.

3 camera capture with feedback tuning and auto-white-balance disabled example

In this example, we disable the auto-white-balance, capture from 3 cameras and transfer the controls from the center camera in auto-exposure mode to the left and right cameras in manual mode. For this implementation, left and right cameras has the auto-exposure disabled and blocked.

In the resultant 3 images bellow you can see that the light exposure and white-balance are the same for the three. The exposure level is configured by the script and the the colors are the same because the auto-white-balance is disabled.

For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.

Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.

❯