FPGA Image Signal Processor - FPGA ISP Accelerators - ColorSpaceConversion

FPGA Image Signal Processor
Introduction
FPGA ISP Accelerators/Modules
Debayer Auto White Balance Color Space Conversion Histogram Equalizer GTU Interpolators Undistort Convolution Fast Fourier Transform
Getting the Code
How to get the code Evaluating FPGA-ISP Building and Installation Guide
Examples
FFMPEG Custom Accelerator
GStreamer Pipelines
Debayer Auto White Balance Color Space Conversion Histogram Equalizer GTU Undistort
Tested Platforms
PicoEVB Zynq 7000 series Zynq MP-SoC series LiteFury
Contact Us

Introduction

This module provides conversions between commonly used ways to express colors. It provides conversions between color spaces such as UYVY and ARGB. The color conversions are defined through the GStreamer capabilities and v4l2 elements.

Supported caps

The FPGA-ISP Color Space Conversion Accelerator is capable of managing the following image properties:

Input

Min resolution: 8x8
Max resolution: 4096x2160
Formats: ARGB, UYVY

Output

Min resolution: 8x8
Max resolution: 4096x2160
Formats: ARGB, UYVY, GRAY8

Algorithm overview

ARGB to UYVY

The ARGB2UYVY conversion is based on these equations

Y =  (0.257 * R) + (0.504 * G) + (0.098 * B) + 16
U = -(0.148 * R) - (0.291 * G) + (0.439 * B) + 128
V =  (0.439 * R) - (0.368 * G) - (0.071 * B) + 128

and is being approximated using a fixed point with 14 bits for the decimal part.

The YUV values are distributed as V4L2 format standard for V4L2_PIX_FMT_UYVY.

ARGB to GRAY8

The ARGB2GREY conversion is based on these equations:

Y = (0.299 * R) + (0.587 * G) + (0.114 * B)

and is being approximated using a Qu(8,14) fixed point format (8-bit integer, 10-bit decimal)

The YUV values are distributed as V4L2 format standard for V4L2_PIX_FMT_GREY.

UYVY to ARGB

The UYVY2ARGB conversion is based on these equations

R = Y + 1.4075 * (V - 128)
G = Y - 0.3455 * (U - 128) - (0.7169 * (V - 128))
B = Y + 1.7790 * (U - 128)

and is being approximated using a fixed point with 14 bits for the decimal part.

The ARGB values are distributed as V4L2 format standard for V4L2_PIX_FMT_ARGB32.

Example Pipelines

In combination with RidgeRun's V4L2-FPGA, it is possible to create a V4L2 interface with GStreamer support, making even easier your computer vision application for embedded systems. Here are some example pipelines to test the FPGA-ISP Color Space Converter.

UYVYtoARGB

Generator (Accelerator input)

gst-launch-1.0 videotestsrc is-live=true  pattern=0 ! "video/x-raw,width=(int)640,height=(int)480,format=UYVY" ! v4l2sink sync=false device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=ARGB" ! perf ! videoconvert ! xvimagesink

Note: This pipeline uses a videoconvert element to display using through xvimagesink element.

ARGBtoUYVY

Generator (Accelerator input)

gst-launch-1.0 videotestsrc is-live=true  pattern=0 ! "video/x-raw,width=(int)640,height=(int)480,format=ARGB" ! v4l2sink sync=false device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=UYVY" ! perf ! videoconvert ! xvimagesink

Note: This pipeline uses a videoconvert element to display using through xvimagesink element.

Processing results

This is a comparative table between the pre/post-converted images for allowed conversions by the CSC accelerator.

• 

  Figure 1. UYVY source image

• 

  Figure 2. ARGB converted image

• 

  Figure 3. ARGB source image

• 

  Figure 4. UYVY converted image

Benchmarks

These benchmarks are taken based on the following setup:

System: NVidia Jetson Xavier
FPGA: PicoEVB (Artix 7 XC7A50T CSG325 -2l)
OS: Ubuntu 18.04
PCI-e: v2.0 - 1 lane

CPU consumption

Table 1. CPU usage for ARGB to UYVY

Resolution	CPU Usage (%)	Framerate (fps)
4k	34.7	13.1
1080p	19.1	31
720p	18.6	31

Table 2. CPU usage for UYVY to ARGB

Resolution	CPU Usage (%)	framerate (fps)
4k	35.2	13.2
1080p	18.1	31
720p	19.2	31

Note: The maximum framerate provided by the camera is 31fps.

You can reproduce these results by using the following pipelines:

• ARGB to UYVY

Generator (Accelerator input)

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! "video/x-raw,width=3840,height=2160,format=RGBA" ! capssetter caps="video/x-raw,format=(string)ARGB,width=(int)3840,height=(int)2160,framerate=(fraction)30/1,multiview-mode=(string)mono,interlace-mode=(string)progressive,colorimetry=(string)1:1:5:4,pixel-aspect-ratio=(fraction)1/1" ! queue ! v4l2sink device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=UYVY" ! perf ! fakesink sync=false

• UYVY to ARGB

Generator (Accelerator input)

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! "video/x-raw,width=3840,height=2160,format=UYVY" ! capssetter caps="video/x-raw,format=(string)UYVY,width=(int)3840,height=(int)2160,framerate=(fraction)30/1,multiview-mode=(string)mono,interlace-mode=(string)progressive,colorimetry=(string)bt601,pixel-aspect-ratio=(fraction)1/1" ! queue ! v4l2sink device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=ARGB" ! perf ! fakesink sync=false

Maximum framerate

Maximum framerate provided by video source

Table 4. Maximum framerate for ARGB

Resolution	Maximum framerate (fps)
4k	47
1080p	190.1
720p	441

Table 5. Maximum framerate for UYVY

Resolution	Maximum framerate (fps)
4k	25.8
1080p	100.6
720p	217

• ARGB

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=ARGB ! perf ! fakesink sync=false

• UYVY

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=UYVY ! perf ! fakesink sync=false

Maximum framerate using the CSC accelerator

Table 4. Maximum framerate for ARGB to UYVY

Resolution	Maximum framerate (fps)
4k	12.6
1080p	47.2
720p	105.5

Table 5. Maximum framerate for UYVY to ARGB

Resolution	Maximum framerate (fps)
4k	12.6
1080p	51.5
720p	112.5

You can reproduce these results by using the following pipelines:

• ARGB to UYVY

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=ARGB ! queue ! v4l2sink device=/dev/video2 sync=false

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=UYVY" ! perf ! fakesink sync=false

• UYVY to ARGB

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=UYVY ! queue ! v4l2sink device=/dev/video2 sync=false

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=ARGB" ! perf ! fakesink sync=false

Known issues

1. GStreamer autonegotiation: The caps, such as width, height, and format, must be specified in the pipeline.