FPGA Image Signal Processor - FPGA ISP Accelerators - ColorSpaceConversion

From RidgeRun Developer Wiki



Previous: Modules/AutoWhiteBalance Index Next: Modules/HistogramEqualizer




Introduction

This module provides conversions between commonly used ways to express colors. It provides conversions between color spaces such as UYVY and ARGB. The color conversions are defined through the GStreamer capabilities and v4l2 elements.

Supported caps

The FPGA-ISP Color Space Conversion Accelerator is capable of managing the following image properties:

Input

Min resolution: 8x8
Max resolution: 4096x2160
Formats: ARGB, UYVY

Output

Min resolution: 8x8
Max resolution: 4096x2160
Formats: ARGB, UYVY, GRAY8

Algorithm overview

ARGB to UYVY

The ARGB2UYVY conversion is based on these equations

Y =  (0.257 * R) + (0.504 * G) + (0.098 * B) + 16
U = -(0.148 * R) - (0.291 * G) + (0.439 * B) + 128
V =  (0.439 * R) - (0.368 * G) - (0.071 * B) + 128

and is being approximated using a fixed point with 14 bits for the decimal part.

The YUV values are distributed as V4L2 format standard for V4L2_PIX_FMT_UYVY.

ARGB to GRAY8

The ARGB2GREY conversion is based on these equations:

Y = (0.299 * R) + (0.587 * G) + (0.114 * B)

and is being approximated using a Qu(8,14) fixed point format (8-bit integer, 10-bit decimal)

The YUV values are distributed as V4L2 format standard for V4L2_PIX_FMT_GREY.

UYVY to ARGB

The UYVY2ARGB conversion is based on these equations

R = Y + 1.4075 * (V - 128)
G = Y - 0.3455 * (U - 128) - (0.7169 * (V - 128))
B = Y + 1.7790 * (U - 128)

and is being approximated using a fixed point with 14 bits for the decimal part.

The ARGB values are distributed as V4L2 format standard for V4L2_PIX_FMT_ARGB32.

Example Pipelines

In combination with RidgeRun's V4L2-FPGA, it is possible to create a V4L2 interface with GStreamer support, making even easier your computer vision application for embedded systems. Here are some example pipelines to test the FPGA-ISP Color Space Converter.

UYVYtoARGB

Generator (Accelerator input)

gst-launch-1.0 videotestsrc is-live=true  pattern=0 ! "video/x-raw,width=(int)640,height=(int)480,format=UYVY" ! v4l2sink sync=false device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=ARGB" ! perf ! videoconvert ! xvimagesink

Note: This pipeline uses a videoconvert element to display using through xvimagesink element.

ARGBtoUYVY

Generator (Accelerator input)

gst-launch-1.0 videotestsrc is-live=true  pattern=0 ! "video/x-raw,width=(int)640,height=(int)480,format=ARGB" ! v4l2sink sync=false device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=640, height=480,format=UYVY" ! perf ! videoconvert ! xvimagesink

Note: This pipeline uses a videoconvert element to display using through xvimagesink element.

Processing results

This is a comparative table between the pre/post-converted images for allowed conversions by the CSC accelerator.


Benchmarks

These benchmarks are taken based on the following setup:

System: NVidia Jetson Xavier
FPGA: PicoEVB (Artix 7 XC7A50T CSG325 -2l)
OS: Ubuntu 18.04
PCI-e: v2.0 - 1 lane

CPU consumption

Table 1. CPU usage for ARGB to UYVY
Resolution CPU Usage (%) Framerate (fps)
4k 34.7 13.1
1080p 19.1 31
720p 18.6 31
Table 2. CPU usage for UYVY to ARGB
Resolution CPU Usage (%) framerate (fps)
4k 35.2 13.2
1080p 18.1 31
720p 19.2 31

Note: The maximum framerate provided by the camera is 31fps.

You can reproduce these results by using the following pipelines:

  • ARGB to UYVY

Generator (Accelerator input)

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! "video/x-raw,width=3840,height=2160,format=RGBA" ! capssetter caps="video/x-raw,format=(string)ARGB,width=(int)3840,height=(int)2160,framerate=(fraction)30/1,multiview-mode=(string)mono,interlace-mode=(string)progressive,colorimetry=(string)1:1:5:4,pixel-aspect-ratio=(fraction)1/1" ! queue ! v4l2sink device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=UYVY" ! perf ! fakesink sync=false
  • UYVY to ARGB

Generator (Accelerator input)

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! "video/x-raw,width=3840,height=2160,format=UYVY" ! capssetter caps="video/x-raw,format=(string)UYVY,width=(int)3840,height=(int)2160,framerate=(fraction)30/1,multiview-mode=(string)mono,interlace-mode=(string)progressive,colorimetry=(string)bt601,pixel-aspect-ratio=(fraction)1/1" ! queue ! v4l2sink device=/dev/video2

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=ARGB" ! perf ! fakesink sync=false

Maximum framerate

Maximum framerate provided by video source

Table 4. Maximum framerate for ARGB
Resolution Maximum framerate (fps)
4k 47
1080p 190.1
720p 441
Table 5. Maximum framerate for UYVY
Resolution Maximum framerate (fps)
4k 25.8
1080p 100.6
720p 217
  • ARGB
gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=ARGB ! perf ! fakesink sync=false
  • UYVY
gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=UYVY ! perf ! fakesink sync=false

Maximum framerate using the CSC accelerator

Table 4. Maximum framerate for ARGB to UYVY
Resolution Maximum framerate (fps)
4k 12.6
1080p 47.2
720p 105.5
Table 5. Maximum framerate for UYVY to ARGB
Resolution Maximum framerate (fps)
4k 12.6
1080p 51.5
720p 112.5

You can reproduce these results by using the following pipelines:

  • ARGB to UYVY

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=ARGB ! queue ! v4l2sink device=/dev/video2 sync=false

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=UYVY" ! perf ! fakesink sync=false
  • UYVY to ARGB

Generator (Accelerator input)

gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,format=UYVY ! queue ! v4l2sink device=/dev/video2 sync=false

Sink (Accelerator output)

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw,width=3840,height=2160,format=ARGB" ! perf ! fakesink sync=false

Known issues

1. GStreamer autonegotiation: The caps, such as width, height, and format, must be specified in the pipeline.


Previous: Modules/AutoWhiteBalance Index Next: Modules/HistogramEqualizer