Zynq Ultrascale+ Capture settings and Gstreamer pipelines

Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help.

Introduction

The Xilinx's Zynq Ultrascale+ board combines the power of ARM processors with the flexibility of FPGAs. You can see more info about the board on this wiki Xilinx Zynq Ultrascale Overview. Also you can give it a try to the SDK developed by RidgeRun, which is still on progress Getting Started Guide for Xilinx Zynq Ultrascale.

Zynq board has the capacity for capturing video through HDMI and SDI interfaces, also it can generate video using a Test Pattern Generator used as an IP core in the FPGA. By now, the capture subsystem is working only for the TPG, so the settings and examples would be explained using this video source.

Video Subsystem

Xilinx has special drivers and device tree bindings for using and configuring the video subsystem. Xilinx video IP cores process video streams by acting as video sinks and/or sources. They are connected by links through their input and output ports, creating a video pipeline, the whole pipeline is represented by an AMBA (Advanced Microcontroller Bus Architecture) child node in the DT. The main components are:

Xilinx Video IP Pipeline (VIPP)

Xilinx video IP pipeline processes video streams through one or more Xilinx video IP cores.
The DT node of the VIPP represents as a top level node of the pipeline and defines mappings between DMAs and the video IP cores.
It is the interface in the ARM when processing streams into or out the FPGA, it can be seen as a video device but it is actually an interface, so the devices in the FPGA must be configured manually and they can't be configured through the VIPP.
When a video pipeline is created, a /dev/media0 device is created, and the entities properties can be accessed through the media device.

Xilinx Video Test Pattern Generator (TPG)

It is an special module for generating patterns with different features such as:

Foreground pattern:
- No Overlay
- Moving Box
- Cross Hairs
Color Mask
Motion speed
Cross Hairs Rows
Cross Hairs Columns
Box Size
Box Color

All those options can be configured before starting the streaming.

Getting the code

In order to get the support for adding the TPG functionality and test the video capture subsystem please contact us: support@ridgerun.com

TPG Settings

The TPG can be seen as a v4l-subdev which can be configured through the v4l2 API. By now, 720P and 1080P resolutions have been tested, as well as UYVY color format. In order to configure first install these three applications in your file system:

media-ctl
v4l2-utils
yavta (Can be get from this repository: Yavta)

First, check the topology of the media device:

media-ctl -d /dev/media0 -p

Then, lets configure the TPG:

#For 720P
media-ctl -d /dev/media0 -V '"a0050000.tpg":0 [fmt:UYVY/1280x720]'

#For 1080p
media-ctl -d /dev/media0 -V '"a0050000.tpg":0 [fmt:UYVY/1920x1080]'

If you wanted to check the controls set in the TPG:

./yavta --no-query -l /dev/v4l-subdev0

You should see something like:

Device /dev/v4l-subdev0 opened.
control 0x00980001 User Controls min 0 max 0 step 0 default 0 current n/a.
control 0x0098c903 Test Pattern: Color Mask min 0 max 7 step 0 default 0 current 0.
control 0x0098c907 Test Pattern: Motion Speed min 0 max 255 step 1 default 4 current 4.
control 0x0098c908 Test Pattern: Cross Hairs Row min 0 max 4095 step 1 default 100 current 100.
control 0x0098c909 Test Pattern: Cross Hairs Colum min 0 max 4095 step 1 default 100 current 100.
control 0x0098c90a Test Pattern: Zplate Horizontal min 0 max 65535 step 1 default 30 current 30.
control 0x0098c90b Test Pattern: Zplate Horizontal min 0 max 65535 step 1 default 0 current 0.
control 0x0098c90c Test Pattern: Zplate Vertical S min 0 max 65535 step 1 default 1 current 1.
control 0x0098c90d Test Pattern: Zplate Vertical S min 0 max 65535 step 1 default 0 current 0.
control 0x0098c90e Test Pattern: Box Size min 0 max 4095 step 1 default 50 current 50.
control 0x0098c90f Test Pattern: Box Color(RGB/YCb min 0 max 16777215 step 1 default 0 current 0.
control 0x0098c912 Test Pattern: Foreground Patter min 0 max 2 step 1 default 0 current 1.
0: No Overlay
1: Moving Box
2: Cross Hairs
control 0x009e0001 Image Source Controls min 0 max 0 step 0 default 0 current n/a.
control 0x009e0901 Vertical Blanking min 3 max 8159 step 1 default 100 current 100.
control 0x009e0902 Horizontal Blanking min 3 max 8159 step 1 default 100 current 100.
control 0x009f0001 Image Processing Controls min 0 max 0 step 0 default 0 current n/a.
control 0x009f0903 Test Pattern min 0 max 16 step 1 default 9 current 9.
17 controls found.

Lets modify the foreground pattern and set the Moving Box (1). As you can see the control identifier is 0x0098c912

yavta --no-query -w '0x0098c912 1' /dev/v4l-subdev0

Finally, configure the video interface:

#For 720P
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=720,pixelformat='YUYV'

#For 1080P
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat='YUYV'

Capture

Yavta capture

After configure the system, a quick test using Yavta could be done:

#For 720P
yavta -n 3 -c10 -f YUYV -s 1280x720 --skip 7 -Ftest.raw /dev/video0

#For 1080P
yavta -n 3 -c10 -f YUYV -s 1920x1080 --skip 7 -Ftest.raw /dev/video0

Here, 10 frames are captured but only the last 3 of them are stored in the file system with the name test.raw

You should see something like this:

GStreamer pipelines

A simple pipeline for testing capture with 720p:

gst-launch-1.0 -v v4l2src device=/dev/video0 do-timestamp=true num-buffers=120 ! "video/x-raw, format=(string)YUY2, \
framerate=(fraction)30/1, width=(int)1280, height=(int)720, interlace-mode=(string)progressive, \
pixel-aspect-ratio=(fraction)1/1"  ! perf ! fakesink silent=false

For decoding, we need to convert the incoming frames to a format accepted by the x264enc:

gst-launch-1.0 -v v4l2src device=/dev/video0 do-timestamp=true ! "video/x-raw, format=(string)YUY2, \
framerate=(fraction)30/1, width=(int)1280, height=(int)720, interlace-mode=(string)progressive, \
pixel-aspect-ratio=(fraction)1/1" ! videoconvert ! "video/x-raw, width=(int)1280, height=(int)720, \
framerate=(fraction)30/1, format=(string)Y42B, interlace-mode=(string)progressive, \
pixel-aspect-ratio=(fraction)1/1" ! perf print-arm-load=true ! queue ! x264enc ! fakesink sync=true silent=true -v

However, the performance is very bad, due to the software color space conversion and the software H264 encoding

Timestamp: 0:13:30.051924570; Bps: 11407957; fps: 6.18; CPU: 27; 
Timestamp: 0:13:31.183599780; Bps: 11407957; fps: 6.18; CPU: 26; 
Timestamp: 0:13:32.315170550; Bps: 11407957; fps: 6.18; CPU: 27; 
Timestamp: 0:13:33.446805550; Bps: 11407957; fps: 6.18; CPU: 27; 
Timestamp: 0:13:34.578261650; Bps: 11407957; fps: 6.18; CPU: 27;

Using 1080p, a simple pipeline looks like:

gst-launch-1.0 -v v4l2src device=/dev/video0 do-timestamp=true num-buffers=120 ! "video/x-raw, format=(string)YUY2, \
framerate=(fraction)30/1, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, \
pixel-aspect-ratio=(fraction)1/1"  ! perf ! fakesink silent=false

And trying to encode the incoming frames:

gst-launch-1.0 -v v4l2src device=/dev/video0 do-timestamp=true ! "video/x-raw, format=(string)YUY2, \
framerate=(fraction)30/1, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, \
pixel-aspect-ratio=(fraction)1/1" ! videoconvert ! "video/x-raw, width=(int)1920, height=(int)1080, \
framerate=(fraction)30/1, format=(string)Y42B, interlace-mode=(string)progressive, \
pixel-aspect-ratio=(fraction)1/1" ! perf print-arm-load=true ! queue ! x264enc ! fakesink sync=true silent=true -v

However, as with the 720p, the performance is very bad:

Timestamp: 0:34:39.850551400; Bps: 11414311; fps: 2.75; CPU: 26; 
Timestamp: 0:34:40.940849990; Bps: 11414311; fps: 2.75; CPU: 27; 
Timestamp: 0:34:42.031124610; Bps: 11414311; fps: 2.75; CPU: 26; 
Timestamp: 0:34:43.121232580; Bps: 11414311; fps: 2.75; CPU: 27; 
Timestamp: 0:34:44.211363480; Bps: 11414311; fps: 2.75; CPU: 27;

For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.

Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.

❯