FPGA Image Signal Processor - FPGA ISP Accelerators - Undistort

FPGA Image Signal Processor
Introduction
FPGA ISP Accelerators/Modules
Debayer Auto White Balance Color Space Conversion Histogram Equalizer GTU Interpolators Undistort Convolution Fast Fourier Transform
Getting the Code
How to get the code Evaluating FPGA-ISP Building and Installation Guide
Examples
FFMPEG Custom Accelerator
GStreamer Pipelines
Debayer Auto White Balance Color Space Conversion Histogram Equalizer GTU Undistort
Tested Platforms
PicoEVB Zynq 7000 series Zynq MP-SoC series LiteFury
Contact Us

Introduction

The FPGA-ISP Undistort allows you to fix lens distortion from the images retrieved by a camera based on the camera matrices and the distortion parameters. It performs the same operation to the OpenCV undistort and fisheye::undistortImage functions.

This accelerator performs the inverse mapping from the undistorted (fixed) image to the input image (distorted), and also produces the undistorted image at the accelerator's output, interpolating with Nearest Neighbor (we are currently working on integrating the bilinear interpolator.

The FPGA-ISP Undistort can work as any of the following options: module and accelerator.

Supported caps

The FPGA-ISP Interpolators are capable of managing the following image properties:

Maximum input: 2047x2047
Maximum output: 2047x2047
Pixels-per-clock: 2 (monochannel and multichannel) - limited by the interpolator
Number of channels: monochannel, multichannel
Formats: Any (the pixel size is templated) unless planar
Pixel-arrangement support: interleaved

Algorithm overview

The interpolators are modules. You can configure them according to your application. The module I/O is presented below:

Input

• Image pointer: the reference to the RAM (BRAM if FPGA does not have one)
• Input/Output image dimensions: width/height in pixels
• Camera matrix (intrinsic): represented in Fixed Point Qs11.20
• Enhanced camera matrix: represented in Fixed Point Qs11.20 (called newcameramatrix in OpenCV)
• Distortion coefficients: represented in Fixed Point Qs11.20. This accelerator supports 5 distortion parameters (k1,k2,p1,p2,k3) for Brown-Conrady and 4 distortion parameters (k1,k2,k3,k4) for Fisheye Model.
• Pixel-width: templated (up to 32 bits)

You can generate the camera matrices and the distortion coefficients by using OpenCV calibration function calibrateCamera().

For more information about camera calibration, please refer to .

Output

• Out stream: Depending on the Pixel-width

Public members

• Camera matrix struct: camera_matrix_t
• Distortion coefficients struct: distortion_t

Algorithms

The algorithms are based on the Brown–Conrady model:

• OpenCV undistort
• Optical distortion

and on the Fisheye model:

• OpenCV fisheye
• Fisheye lens

Some of the equations have been simplified and tailored to the application.

Results

• 

  Figure 1. Input image

• 

  Figure 2. Output (fisheye undistorted) image

Note: The results could vary depending on the camera matrix and distortion parameters. Accurate parameters may result in better image correction.

Benchmarks

Since the PicoEVB doesn't support 1080p images because of memory availability, the following performances are theoretical and approximate based on the performance obtained by the latency reported by the synthesizer:

Table 1. Typical framerate of FPGA-ISP Undistort Accelerator (Brown-Conrady). Based on

Resolution	Maximum framerate (ARGB-NN)	Maximum framerate (GRAY8-NN)
4k	N/A	N/A
1080p	21.5	42
720p	44.73	87,39

These estimates were generated by the latency produced by the RTL simulation.

Disclaimer: The accelerator performance shown above is intended to give an idea of the peak performance for a PicoEVB (PCI-e 2.0 1 lane) with more DSP available and a RAM attached to it. With a better device, this performance can be reached, leading to better results.

Known issues

1. GStreamer autonegotiation: The caps, such as width, height, and format, must be specified in the pipeline.

2. Numerical precision: Since the resolution is lower than using floating-point numbers, different results might be obtained.