FPGA Image Signal Processor - FPGA ISP Accelerators - AutoWhiteBalance

Introduction

The FPGA-ISP Auto White Balance Accelerator is an FPGA accelerator that performs Auto White Balance for ARGB video. It can be used in cases where the video needs to be balanced but without sacrificing CPU power.

Supported caps

The FPGA-ISP AWB Accelerator is capable of managing the following image properties:

Input

Min resolution: 8x8
Max resolution: 4096x2160
Formats: ARGB

Output

Min resolution: 8x8
Max resolution: 4096x2160
Formats: ARGB

Algorithm overview

This FPGA-ISP implements the gray world algorithm. This algorithm assumes that in a well-balanced image the average of all the colors tends to a neutral gray. I this algorithm, the average of the R, G, and B components of every frame is calculated and then used to correct each individual pixel to accomplish the goal of an average neutral gray image. Although it gives good results in most situations it is not recommended for cases where one color predominates over the whole scene.

Example pipelines

In combination with RidgeRun's V4L2-FPGA, it is possible to create a V4L2 interface with GStreamer support, making even easier your computer vision application for embedded systems. Here is some example of pipelines to test the FPGA-ISP Auto White Balance.

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,format=ARGB,width=640,height=480 ! queue ! v4l2sink device=/dev/video2 -v

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=ARGB" ! perf ! videoconvert ! xvimagesink -v

Processing results

Here is an example of the results of applying the AWB algorithm. On the left, the unbalanced image, and on the right the same image after being processed by the AWB ISP.

Figure 1. Input image
Figure 2. Output (balanced) image

Benchmarks

These benchmarks are taken based on the following setup:

System: NVidia Jetson Xavier
FPGA: PicoEVB (Artix 7 XC7A50T CSG325 -2l)
OS: Ubuntu 18.04
PCI-e: v2.0 - 1 lane

CPU consumption

Table 1. CPU usage for FPGA-ISP AWB Accelerator
Resolution	CPU Usage (%)	Framerate (fps)
4k	35	12.9
1080p	18	31
720p	18	31

Note: The maximum framerate provided by the camera is 31fps.

You can reproduce these results by using the following pipelines:

Generator (Accelerator input)

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! "video/x-raw,width=3840,height=2160,format=RGBA" ! capssetter caps="video/x-raw,format=(string)ARGB,width=(int)3840,height=(int)2160,framerate=(fraction)30/1,multiview-mode=(string)mono,interlace-mode=(string)progressive,colorimetry=(string)1:1:5:4,pixel-aspect-ratio=(fraction)1/1" ! queue ! v4l2sink device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=3840, height=2160,format=ARGB" ! perf ! fakesink sync=false

Maximum framerate

Maximum framerate provided by video source

Table 1. Maximum framerate for ARGB
Resolution	Maximum framerate (fps)
4k	47
1080p	190.1
720p	441

ARGB

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=ARGB ! perf ! fakesink sync=false

Maximum framerate using AWB accelerator

Table 2. Maximum framerate for FPGA-ISP AWB Accelerator
Resolution	Maximum framerate (fps)
4k	12.5
1080p	49.5
720p	106.5

You can reproduce these results by using the following pipelines:

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=ARGB ! queue ! v4l2sink device=/dev/video2 sync=false

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=3840, height=2160,format=ARGB" ! perf ! fakesink sync=false

Known issues

1. GStreamer autonegotiation: The caps, such as width, height, and format, must be specified in the pipeline.

Previous: Modules/Debayer

Index

Next: Modules/ColorSpaceConversion

❯