View source for FPGA Image Signal Processor/Modules/Debayer

<noinclude>
{{FPGA Image Signal Processor/Head|next=Modules/AutoWhiteBalance|previous=Modules}}
</noinclude>

{{DISPLAYTITLE:FPGA Image Signal Processor - FPGA ISP Accelerators - Debayer
|noerror}}

== Introduction ==

The FPGA-ISP Debayer Accelerator is an FPGA accelerator that performs image demosaicing captured in Bayer to RGB. One of the use cases is a Bayer camera connected to an embedded system, but this embedded system does not have an ISP or GPU. With this accelerator, you will easily integrate an accelerated demosaicing module in your platform without sacrificing any CPU resource.

== Supported caps ==

The FPGA-ISP Debayer Accelerator is capable of managing the following image properties:

=== Input ===

<pre>
Min resolution: 8x8
Max resolution: 4096x2160
Formats: RGGB, BGGR, GRBG, GBRG (8-bit bayer)
</pre>

=== Output ===

<pre>
Min resolution: 8x8
Max resolution: 4096x2160 (same as input)
Formats: ARGB (32-bit format)
</pre>

== Algorithm overview ==

There are several ways to perform Bayer demosaicing. One of the most popular is a bilinear interpolation. In the FPGA-ISP Debayer Accelerator, the bilinear interpolation is transformed into a convolution of color kernels, based on the <ref>https://hal.archives-ouvertes.fr/hal-00683233/document</ref> approach. This makes the demosaicing much faster when combining this method with pixel streaming, saving area, and avoid any full memory copy into the FPGA, making this accelerator suitable to be implemented on FPGA without direct access to RAM.

Our accelerator is composed of a color filter (called Phi Filter), three convolutions, and a pixel joiner.

[[File:ISP Debayer Algorithm.png|600px|frameless|center|Debayer]]

Additionally, the convolution kernels are represented in Qs(0,16) fixed-point representation, which guarantees the precision on the numeric computation of each channel and helps to save area as much as possible.

== Example pipelines ==

In combination with [https://developer.ridgerun.com/wiki/index.php?title=V4L2_FPGA RidgeRun's V4L2-FPGA], it is possible to create a V4L2 interface with GStreamer support, making even easier your [https://www.ridgerun.com/computer-vision computer vision] application for embedded systems. Here are some example of pipelines to test the FPGA-ISP Debayer.

=== Generator (Accelerator input) ===
<syntaxhighlight lang=bash>
gst-launch-1.0 videotestsrc ! video/x-raw,format=ARGB,width=640,height=480 ! queue ! rgb2bayer ! "video/x-bayer,format=bggr" ! queue ! v4l2sink device=/dev/video2 -v 
</syntaxhighlight>

=== Sink (Accelerator output) ===
<syntaxhighlight lang=bash>
gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=ARGB" ! perf ! videoconvert ! xvimagesink
</syntaxhighlight>

== Benchmarks ==

{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;"
|+ align="top" style="caption-side: top" | Table 1. Typical framerate of FPGA-ISP Debayer Accelerator. Based on <ref>https://developer.ridgerun.com/wiki/index.php?title=V4L2_FPGA/Examples/Pass_Through</ref>
|-
! Resolution 
! Maximum framerate (fps)
|-
| 4k
| 7,464
|-
| 1080p
| 29,258
|-
| 720p
| 65,023
|}

These framerates are taken based on the following setup:

<pre>
System: NVidia Jetson Xavier
FPGA: PicoEVB (Artix 7 XC7A50T CSG325 -2l)
OS: Ubuntu 18.04
PCI-e: v2.0 - 1 lane
</pre>

You can reproduce these results by using the following pipelines:

''' Generator (Accelerator input) '''
<syntaxhighlight lang=bash>
gst-launch-1.0 videotestsrc ! video/x-raw,format=ARGB,width=640,height=480 ! queue ! rgb2bayer ! "video/x-bayer,format=bggr" ! queue ! v4l2sink device=/dev/video2 -v 
</syntaxhighlight>

''' Sink (Accelerator output) '''
<syntaxhighlight lang=bash>
gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=ARGB" ! perf ! fakesink sync=false
</syntaxhighlight>

== Known issues ==

1. '''GStreamer autonegotiation''': The caps, such as width, height, and format, must be specified in the pipeline.

<noinclude>
{{FPGA Image Signal Processor/Foot|Modules|Modules/AutoWhiteBalance}}
</noinclude>