FPGA Image Signal Processor - FPGA ISP Accelerators - Debayer

FPGA Image Signal Processor
Introduction
FPGA ISP Accelerators/Modules
Debayer Auto White Balance Color Space Conversion Histogram Equalizer GTU Interpolators Undistort Convolution Fast Fourier Transform
Getting the Code
How to get the code Evaluating FPGA-ISP Building and Installation Guide
Examples
FFMPEG Custom Accelerator
GStreamer Pipelines
Debayer Auto White Balance Color Space Conversion Histogram Equalizer GTU Undistort
Tested Platforms
PicoEVB Zynq 7000 series Zynq MP-SoC series LiteFury
Contact Us

Introduction

The FPGA-ISP Debayer Accelerator is an FPGA accelerator that performs image demosaicing captured in Bayer to RGB. One of the use cases is a Bayer camera connected to an embedded system, but this embedded system does not have an ISP or GPU. With this accelerator, you will easily integrate an accelerated demosaicing module in your platform without sacrificing any CPU resource.

Supported caps

The FPGA-ISP Debayer Accelerator is capable of managing the following image properties:

Input

Min resolution: 8x8
Max resolution: 4096x2160
Formats: RGGB, BGGR, GRBG, GBRG (8-bit bayer)

Output

Min resolution: 8x8
Max resolution: 4096x2160 (same as input)
Formats: ARGB (32-bit format)

Algorithm overview

There are several ways to perform Bayer demosaicing. One of the most popular is a bilinear interpolation. In the FPGA-ISP Debayer Accelerator, the bilinear interpolation is transformed into a convolution of color kernels, based on the approach. This makes the demosaicing much faster when combining this method with pixel streaming, saving area, and avoid any full memory copy into the FPGA, making this accelerator suitable to be implemented on FPGA without direct access to RAM.

Our accelerator is composed of a color filter (called Phi Filter), three convolutions, and a pixel joiner.

Debayer

Additionally, the convolution kernels are represented in Qs(0,16) fixed-point representation, which guarantees the precision on the numeric computation of each channel and helps to save area as much as possible.

Example pipelines

In combination with RidgeRun's V4L2-FPGA, it is possible to create a V4L2 interface with GStreamer support, making even easier your computer vision application for embedded systems. Here are some example of pipelines to test the FPGA-ISP Debayer.

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,format=ARGB,width=640,height=480 ! queue ! rgb2bayer ! "video/x-bayer,format=bggr" ! queue ! v4l2sink device=/dev/video2 -v

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=ARGB" ! perf ! videoconvert ! xvimagesink

Benchmarks

Table 1. Typical framerate of FPGA-ISP Debayer Accelerator. Based on

Resolution	Maximum framerate (fps)
4k	7,464
1080p	29,258
720p	65,023

These framerates are taken based on the following setup:

System: NVidia Jetson Xavier
FPGA: PicoEVB (Artix 7 XC7A50T CSG325 -2l)
OS: Ubuntu 18.04
PCI-e: v2.0 - 1 lane

You can reproduce these results by using the following pipelines:

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,format=ARGB,width=640,height=480 ! queue ! rgb2bayer ! "video/x-bayer,format=bggr" ! queue ! v4l2sink device=/dev/video2 -v

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=ARGB" ! perf ! fakesink sync=false

Known issues

1. GStreamer autonegotiation: The caps, such as width, height, and format, must be specified in the pipeline.