Fast GStreamer overlay element

From RidgeRun Developer Wiki
Revision as of 16:39, 16 March 2016 by Spalli (talk | contribs)

Description

emb-overlay is a gstreamer element that can be used to overlay: images, text and/or time and date over video streams or photos without using lots of floating point arithmetic. This is necessary to get good performance when the processor doesn't contain an FPU.

GStreamer embedded overlay element

After building the SDK and installing the images onto your target hardware, verify the embedded overlay element is available: 

gst-inspect | grep emboverlay

With the expected output being: 

emboverlayplugin:  emboverlay: overlay for embedded systems

To see the properties that are supported, run: 

gst-inspect emboverlay

which will display an output similar to: 

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: null Current: "emboverlay0"
  logo                : Overlayed image path, must be a png file
                        flags: readable, writable
                        String. Default: null Current: null
  text                : Overlayed text to display
                        flags: readable, writable
                        String. Default: null Current: null
  logo-offseth        : Overlayed image horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  logo-offsetv        : Overlayed image vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  logo-transp         : Overlayed image transparency. Enable = 1 Disable = 0
                        flags: readable, writable
                        Integer. Range: 0 - 1 Default: 0 Current: 0
  text-offseth        : Overlayed text horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-offsetv        : Overlayed text vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-font-height    : Overlayed text font height
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  text-color          : Overlayed text color. Black = 0 Red = 1 Green = 2 Blue = 3 White = 4
                        flags: readable, writable
                        Integer. Range: 0 - 4 Default: 0 Current: 0
  time-offseth        : Overlayed time horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-offsetv        : Overlayed time vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-font-height    : Overlayed time font size
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 Current: 0
  time-color          : Overlayed time font color. Black = 0 Red = 1 Green = 2 Blue = 3 White = 4
                        flags: readable, writable
                        Integer. Range: 0 - 4 Default: 0 Current: 0
  time-param          : Overlayed time parameters %b %Y; %H:%M:%S will render: month year; hour:minutes:seconds
                        flags: readable, writable
                        String. Default: null Current: null

The (x,y) origin is the upper left corner.

Fonts and Graphics

Any Cairo compatible font can use used. Any graphic that can be stored in the PNG file format can be used. For graphics, a color in the PNG file can be used to indicate transparency.

The time consumed to overlay onto each video frame is dependent on two factors:

  • Size of text and graphic
  • If the bitmap version the text and/or graphic in the correct color space is already cached.

The fast GStreamer overlay element caches the text and graphics in a color space that can be directly applied to each video frame. If the text and graphics doesn't change from one frame to the next, then the time to overlay text/graphics on the frame is just the bitblit time to process each pixel associated with other overlay regions.

If the text or graphics is being changed quickly, then there is the addition delay to render the text in the choosen font and perform any needed color space conversion.

GStreamer Pipelines

Next you will find various pipelines showing the emboverlay element capabilities, in this case the element was tested using a Leopardboard DM365 with a 5Mp sensor camera.

You can use emboverlay for text and/or date and/or logo overlay on each video frame.

Video overlay examples

Overlay text

Colored text can be entered using hexadecimal values in the following format: 

0xRRGGBB

R stands for Red, G for Green, and B for Blue, with the possible values for each color being: 0-FF

Examples: 

0xFFFFFF = White
0x000000 = Black
0xFF0000 = Red
0x00FF00 = Green
0x0000FF = Blue 
0xFF8000 = Orange

Any other combination are possible, the same principle applies to the color of the border of the text, letter color and border are chosen with the caps: "text-color", "text-border".

Example pipeline: 

gst-launch v4l2src ! video/x-raw-yuv, format=\(fourcc\)NV12, width=640, height=480  ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=50 text-color=0xFFFFFF \
text-border=0x000000 text="Hello overlay" ! TIDmaiVideoSink

Note: Test above pipeline by inserting 'enable-text=true' emb-overlay element property.

Overlay date

emboverlay date has the same capabilities as the Unix date command, letter and border color can be chosen with the caps "time-color" and "time-border", using hexadecimal numbers with the format 0xRRGGBB, for example: 

gst-launch v4l2src ! video/x-raw-yuv, format=\(fourcc\)NV12, width=640, height=480 ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 /
time-color=0xFFFFFF time-param="%b %Y %H:%M:%S" time-border=0x000000 ! TIDmaiVideoSink

Note: Test above pipeline by inserting 'enable-time=true' emb-overlay element property.

Setting the time and date on the target

Unless your hardware has a battery backed up real time clock (RTC) and you have the Linux kernel configured to set the wall clock at power on, your time and date will start at 1970 when you turn on your target hardware. You can check the time/date setting by running 

date

You can also use the date command to set the current time and date. The format of the argument to set the time/date is MMDDhhmm[[CC]YY][.ss] For example, if you want to set the time/date to June 22nd, 2013 at 1:32:19 pm, use 

date 062213332013.19

You can run below pipline to view date/time overlay with camera capture DVI 720p output: 

 gst-launch v4l2src ! video/x-raw-yuv, format=\(fourcc\)NV12, width=1280, height=720 ! \
            emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0x000000 time-param="+DATE: %m/%d/%y TIME:%H:%M:%S" time-border=0x000000 enable-time=true ! \
            TIDmaiVideoSink videoOutput=DVI sync=false accelFrameCopy=true videoStd=720P_60

The only supported image file format is PNG. The image size in the file must be have even dimensions. 

gst-launch v4l2src ! dmaiaccel ! video/x-raw-yuv, format=\(fourcc\)NV12, width=640, height=480 ! emboverlay logo="t2.png" logo-offsetv=250 logo-offseth=300 /
logo-transp=0 ! TIDmaiVideoSink

Note: Test above pipeline by inserting 'enable-logo=true' emb-overlay element property.

Mixing emboverlay text/logo/date

Emboverlay can overlay text and/or date and/or logo, the following is an example of a pipe with date and logo: 

gst-launch v4l2src -e num-buffers=300 ! video/x-raw-yuv, format=\(fourcc\)NV12, width=640, height=480 ! emboverlay text-offsetv=40 text-offseth=40 / 
text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" time-offsetv=200 time-offseth=500 time-font-height=50 time-color=0x000000 /
time-param="%M:%S" time-border=0xFFFFFF ! dmaiperf ! TIDmaiVideoSink sync=false enable-last-buffer=false

Picture overlay examples

The following logos are used for the picture examples. The logo is 312 x 28 pixels in size and uses white (0xFFFFFF) as the transparency color.

This logo has transparency background, including the circle in the triangle being the transparency color.

This logo has transparency background, with the circle in the triangle being white.


Tested Pipelines with DVI Output:

Time overlay :
gst-launch v4l2src ! video/x-raw-yuv, format=\(fourcc\)NV12, width=1280, height=720 ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0x000000 /
time-param="%b %Y %H:%M:%S" time-border=0x000000 enable-time=true ! TIDmaiVideoSink videoOutput=DVI sync=false accelFrameCopy=true videoStd=720P_60
Text overlay :
gst-launch v4l2src ! video/x-raw-yuv, format=\(fourcc\)NV12, width=1280, height=720  ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=50 text-color=0xFFFFFF / 
text-border=0x000000 text="Hello overlay" enable-text=true ! TIDmaiVideoSink videoOutput=DVI sync=false accelFrameCopy=true videoStd=720P_60
Image overlay :
gst-launch v4l2src ! dmaiaccel ! video/x-raw-yuv, format=\(fourcc\)NV12, width=640, height=480 ! emboverlay logo= "/usr/reticle1.png" logo-offsetv=250 logo-offseth=300 / 
logo-transp=0 enable-logo=true ! TIDmaiVideoSink videoOutput=DVI sync=false accelFrameCopy=true videoStd=720P_60


Building and installing on Ubuntu host

Make sure $DEVDIR is not set otherwise you will end up using the wrong pkg-config. 

cd proprietary/gst-emb-overlay/src
autoreconf --verbose --force --install
./configure  --prefix=/usr
make
make install
gst-inspect emboverlay

Emboverlay versus cairo overlay

cairo overlay

Works in I420 color space. In order to use cairo overlay, code modifications needs to be done. GStreamer DMAI encoders work in NV12 color space.

cairo posses three gstreamer elements capable of:

  1. Blit text
  2. Blit time
  3. Render

cairo overlay element properties:

cairo text: Adds text strings on top of a video buffer

  1. name
  2. text
  3. shaded-background
  4. valign
  5. halign
  6. xpad
  7. ypad
  8. deltax
  9. deltay
  10. font-desc

cairo time: Overlays the time on a video stream

  1. name
  2. qos (handle quality of service)

cairo render: Encodes streams using Cairo

  1. name

emboverlay

RR Emboverlay works in NV12, this allows to use emboverlay with Dmai encoders and TIDmaiVideoSink

Emboverlay is one gstreamer element capable of:

  • Blit text
  • Blit time
  • Blit logo

emboverlay element properties:

Emboverlay is just one gstreamer element, nevertheless it will be document as a text element, time element and logo element

emboverlay text:

  1. text
  2. text-offseth
  3. text-offsetv
  4. text-font-height
  5. text-color
  6. text-border

emboverlay time:

  1. time-offsetv
  2. time-offseth
  3. time-font-height
  4. time-color
  5. time-border
  6. time-param

emboverlay logo:

  1. logo (png file with/without transparence)
  2. logo-offseth
  3. logo-offsetv
  4. logo-transp


emboverlay versus cairo overlay fps test

fps tests where done in a LeopardBoard DM365 with a 5Mp video camera and overlaying just text

No overlay

Aprox: 29-30 fps

cairotextoverlay

Aprox: 12 fps

emboverlay

Aprox: 29-30 fps

Retrieved from "http://developer.ridgerun.com/wiki/index.php?title=Fast_GStreamer_overlay_element&oldid=8855"