Xilinx UltraScale Plus VPSS V4L2 Linux Video Processing Driver

Introduction

The Xilinx UltraScale+ Video Processing Subsystem (VPSS) is a hardware accelerator supporting 4K UHD video processing including motion adaptive deinterlacer, scaler, color space conversion, chroma resampler, and format converter.

The VPSS hardware accelerator supports:

4K UHD (3840 × 2160) at 60 fps
RGB, YUV 4:4:4, YUV 4:2:2 and YUV 4:2:0 color spaces
8, 10, 12 and 16 bits per component
Deinterlacing
Scaling
Color space conversion between supported formats
Chroma resampling between supported color spaces
Frame rate conversion

At the time this wiki page was written, Xilinx has developed two software approaches to handle the VPSS hardware accelerator:

Bare metal drivers: These drivers have a comprehensive support to handle the VPSS capabilities. The driver set provides an API to control and configure the desired formats, start, stop, reset the IP, change parameters as brightness, saturation, PIP, subcores configurations and all the necessary for the correct hardware accelerator initialization and control. The configuration is obtained from the Vivado generated files describing the IPs in the FPGA Programmable Logic.

V4L2 single mode driver: these drivers provides an interface between the hardware accelerator and V4L2 applications. Each driver supports one feature, sucha s scaler-only support. The rest of the VPSS hardware accelerator features are not available. A device tree node is using for configuring the single mode drivers.

The V4L2 framework is the standard Linux kernel video processing API. V4L2 is useful for embedded applications and user space frameworks as GStreamer, VLC, and OpenCV. RidgeRun developed a new V4L2 Linux driver with the support for handling the Xilinx UltraScale+ VPSS features, providing developers a mechanism for utilizing these features using standard tools accessing the simple V4L2 Linux kernel interface. The RidgeRun VPSS V4L2 driver uses the bare metal driver API as the bottom layer to control the VPSS hardware accelerator.

Features

The new VPSS driver has the following features:

An easy-to-use initial configuration through a Device Tree node. Once the VPSS IP is set in place, you can generate the device tree node depending on the configuration specified by the Vivado generated files. This provides complete control over all the possible configurations and use cases.
Scaling: With a simple V4L2 pad configuration the IP can do the scaling taking in consideration the supported modes limited by the bare metal drivers.
Color Space Conversion: Also configuring the pad formats, the VPSS is capable to convert between RGB and YUV formats.
Deinterlacing: Each pad's field can be configured independently and the IP is able to handle the deinterlacing depending on that configuration.
Frame Rate Conversion: Each pad's framerate can be configured independently. If they have different framerate the conversion is made.

Architecture

The driver has a 3 layers design, each with independent functionality:

V4l2 Layer

This layer is in charge of manage mainly all the V4L2 stuffs and the device tree parsing. The functions are:

Kernel driver registration
V4l2 media entity handling
V4L2 pad formats management
Expose pad and subdevice operations
Device Tree parsing for all the VPSS and subcores configurations. This configuration will be explained with details in below sections.

Interface layer

This layer can be considered as a bridge between the V4L2 world and the real VPSS driver implementation and handling done by the bare metal drivers. As we already mentioned, the bare metal drivers exposed through their APIs all the required functions to handle easily the main actions to control the VPSS. So far, this layer has implemented the following functions:

xvpss_reset: VPSS reset
xvpss_start: Enable the VPSS to start processing frames.
xvpss_stop: Stop the VPSS streaming.
xvpss_initialize: Using the configuration set by the user in the device tree, the VPSS mode and resources are initialized.
xvpss_streams_configuration: This function takes in consideration the configuration done in the V4L2 layer in order to parse all the parameters needed by the bare metal drivers. Interlace, Color formats, Resolution and Framerate are translated from v4l2 space to bare metal drivers space. Once the configuration is parsed and verified, the input and output streams are configured.
xvpss_set_subsystem_configuration: This function lets the bare metal drivers know and register the VPSS configuration set by the user through the Device Tree.

As future work, more functions will be added to handle special features like Brightness, Gain and Saturation Control.

Bare Metal drivers layer

This is the bottom layer, and it was not created by RidgeRun, it is a compilation of specific drivers for all the VPSS components:

Horizontal and Vertical Scalers
Deinterlacer
Color Space Converter
Vertical and Horizontal Chroma Resamplers
Letterbox
AXI Switches and AXI VDMA controllers
Gpios
And finally the VPSS controller, which takes control of all the features exposed by the others components.

The main change made by RidgeRun is to change the way the components are configured. Initially it was made through the files generated by Vivado, however this approach is not kernel-friendly, extensible or easy to change for other modes and topologies. That's why the Device Tree configuration was implemented.

Configuration

Each component within the VPSS IP has an special configuration and parameters that can be handled depending on the Vivado setup, to enable or disable different features in the IPs. The bare metal drivers configure this parameters using the files generated by Vivado, however this is not a good approach for drivers embedded in the kernel, and it is less configurable. Instead RidgeRun uses a VPSS Device Tree node to configure each component and the VPSS itself.

To fill the node you only have to take a look to the configuration files generated by Vivado (xparameters.h). Then search for each component configuration and fill it depending on the values set.

VPSS configuration

The parameters allowed to be configured are:

Topology: VPSS Topology configured. Possible configurations:
- Scaler-only
- Full Fledged
- Deinterlacer-only
- CSC-only
- Vcresampler-only
- Hcresampler-only

Color Depth: Processing precision of the data pipe. Represents the Max-data-width for the components
Pix Per Clock: The pixels per clock property of the IP
Number of Components: Number of components used in the VPSS configuration
Maximum Width: The maximum number of columns supported
Maximum Height: The maximum number of rows supported
Has MADI: Motion Adaptive Deinterlacer Flag
Framebuf Address: DDR address for Frame Buffer (required for Deinterlacer and Full Fledged configurations)

Testing

For testing, it depends on the specific PL configuration and system designed by each developer. However, in order to configure the VPSS for the desired configuration, RidgeRun has added new features to the media-ctl application. With this support you are able to configure each pad resolution, color space, field mode (Interlaced or Progressive) and framerate. With the input and output pads formats settings, the VPSS will understand what kind of processing it needs to perform.

For example, using these two commands:

media-ctl -d /dev/media0 -V '"b0100000.vpss_full":0 [fmt:RBG888/1920x540/INTERLACED@1/50]'
media-ctl -d /dev/media0 -V '"b0100000.vpss_full":1 [fmt:UYVY/480x272/NONE@1/60]'

It means that the VPSS will perform 4 operations: