Fast GStreamer overlay element

From RidgeRun Developer Wiki

Problems running the pipelines shown on this page?
Please see our GStreamer Debugging guide for help.

Description

Emboverlay is a GStreamer element that can be used to overlay: images, text, and/or time and date over video streams or photos without using lots of floating point arithmetic. This is necessary to get good performance when the processor doesn't contain an FPU.

Dependencies

Emboverlay requires GStreamer. You may install it with the following command: 

sudo apt install \
gstreamer1.0-x \
libgstreamer1.0-dev \
libgstreamer-plugins-base1.0-dev \
gstreamer1.0-plugins-bad \
libgstreamer-plugins-bad1.0-dev

GStreamer embedded overlay element

After building the SDK and installing the images onto your target hardware, verify the embedded GStreamer overlay element is available: 

gst-inspect-1.0 | grep emboverlay

With the expected output being: 

emboverlay:  emboverlay: Embedded Overlay Element

To see the properties that are supported, run: 

gst-inspect-1.0 emboverlay

which will display an output similar to: 

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "emboverlay0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false
  font                : Font typeface such as Sans
                        flags: readable, writable
                        String. Default: "Sans"
  italic              : Enable or disable italic style on text
                        flags: readable, writable
                        Boolean. Default: false
  bold                : Enable or disable bold style on text
                        flags: readable, writable
                        Boolean. Default: false
  logo                : Overlayed image path, must be a png file
                        flags: readable, writable
                        String. Default: null
  logo-offseth        : Overlayed image horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  logo-offsetv        : Overlayed image vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  logo-transparency   : Enable or disable logo transparency.
                        flags: readable, writable
                        Boolean. Default: false
  logo-scale          : Scale factor for overlay image.  Scale less than 1.0 shrinks image, range 0.1 to 10.0
                        flags: readable, writable
                        Float. Range:             0,1 -              10 Default:               1 
  logo-autofit        : Enable or disable auto logo placement so it is fully visible
                        flags: readable, writable
                        Boolean. Default: false
  logo-enable         : Enable or disable logo rendering
                        flags: readable, writable
                        Boolean. Default: false
  text                : Overlayed text to display
                        flags: readable, writable
                        String. Default: null
  text-offseth        : Overlayed text horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text-offsetv        : Overlayed text vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text-font-height    : Overlayed text font height
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text-color          : Overlayed text color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text-border         : Overlayed text border color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text-enable         : Enable or disable text rendering
                        flags: readable, writable
                        Boolean. Default: false
  text2               : Overlayed text2 to display
                        flags: readable, writable
                        String. Default: null
  text2-offseth       : Overlayed text2 horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text2-offsetv       : Overlayed text2 vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text2-font-height   : Overlayed text2 font height
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text2-color         : Overlayed text2 color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text2-border        : Overlayed text2 border color. RGB format 0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  text2-enable        : Enable or disable text2 rendering
                        flags: readable, writable
                        Boolean. Default: false
  time-pattern        : Overlayed time pattern %b %d %Y; %H:%M:%S will render: month day year; hour:minutes:seconds
                        flags: readable, writable
                        String. Default: null
  time-offseth        : Overlayed time horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  time-offsetv        : Overlayed time vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  time-font-height    : Overlayed time font size
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  time-color          : Overlayed time font color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  time-border         : Overlayed time border color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  time-enable         : Enable or disable time rendering
                        flags: readable, writable
                        Boolean. Default: false
  frame-number        : Starting frame number to overlay
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 1 
  frame-offseth       : Overlayed frame horizontal offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  frame-offsetv       : Overlayed frame vertical offset
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  frame-font-height   : Overlayed frame font size
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  frame-color         : Overlayed frame font color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  frame-border        : Overlayed frame border color. RGB format color=0xRRGGBB
                        flags: readable, writable
                        Integer. Range: -2147483648 - 2147483647 Default: 0 
  frame-enable        : Enable or disable frame rendering
                        flags: readable, writable
                        Boolean. Default: false

The (x,y) origin is the upper left corner.

Fonts and Graphics

Any Cairo compatible font can use used. Any graphic that can be stored in the PNG file format can be used. For graphics, a color in the PNG file can be used to indicate transparency.

The time consumed to overlay onto each video frame is dependent on two factors:

  • Size of text and graphic
  • If the bitmap version the text and/or graphic in the correct color space is already cached.

The fast GStreamer overlay element caches the text and graphics in a color space that can be directly applied to each video frame. If the text and graphics don't change from one frame to the next, then the time to overlay text/graphics on the frame is just the bitblit time to process each pixel associated with other overlay regions.

If the text or graphics is being changed quickly, then there is an additional delay to render the text in the chosen font and perform any needed color space conversion.

Adding new fonts

To get the list of installed fonts on the system, run the following command 

fc-list | awk '{split($0,a,":"); print a[2]'} | sort | uniq

The name of the font that should be used is exactly as it appears on the list generated by the previous command. To generate a sample image with the "Ubuntu" font run the following command: 

gst-launch-1.0 videotestsrc num-buffers=1 ! video/x-raw,format=NV12,width=1920,height=1080 ! queue ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="The quick brown fox" text-enable=true font="Ubuntu" ! queue ! filesink location=ubuntu.ts

Some fonts can be displayed as a family, these cases are identified by the comma character (,) in the font's name. Example: 

Lato
Lato,Lato Black
Lato,Lato Hairline
Lato,Lato Heavy
Lato,Lato Light
Lato,Lato Medium
Lato,Lato Semibold
Lato,Lato Thin

In the above example, Lato is a valid font that can be used with GstEmboverlay but Lato is also a family name that owns the other 7 fonts. GstEmboverlay supports any of the mentioned fonts, if you want to use Lato Hairline you don't need to specify the family name, only the font name, pipeline example: 

gst-launch-1.0 videotestsrc num-buffers=1 ! video/x-raw,format=NV12,width=1920,height=1080 ! queue ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="The quick brown fox" text-enable=true font="Lato Hairline" ! queue ! filesink location=lato_hairline.ts

To view the installed systems or to install new fonts the gnome-font-viewer can be used.

To install the program on Ubuntu: 

sudo apt install gnome-font-viewer

To run the program: 

gnome-font-viewer

GStreamer Pipelines for IMX6

Next, you will find various pipelines showing the emboverlay element capabilities, in this case, the element was tested on the IMX6 platform.

You can use emboverlay for text, time, date, logo, and frame-number overlay on each video frame.

Video overlay examples

Text Overlay

Colored text can be entered using hexadecimal values in the following format: 

0xRRGGBB

R stands for Red, G for Green, and B for Blue, with the possible values for each color being: 0-FF

Examples: 

0xFFFFFF = White
0x000000 = Black
0xFF0000 = Red
0x00FF00 = Green
0x0000FF = Blue 
0xFF8000 = Orange

Any other combination is possible, the same principle applies to the color of the border of the text, letter color and border are chosen with the caps: "text-color", "text-border". The text style can be adjusted as needed, for Bold and Italic styles just set the properties "bold" or "italic" to true.

Example pipeline using normal style: 

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480 ! queue ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=50 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" text-enable=true font="Ubuntu" ! queue ! imxvpuenc_h264 ! mpegtsmux ! filesink location=test_text_overlay.ts
Text overlay using normal style

Example pipeline using Bold and Italic styles: 

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480 ! queue ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=50 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" text-enable=true font="Ubuntu" italic=true bold=true ! queue ! imxvpuenc_h264 ! mpegtsmux ! filesink location=test_text_overlay.ts
Text overlay using Italic and Bold styles

Date Overlay

emboverlay date has the same capabilities as the Unix date command, letter and border color can be chosen with the caps "time-color" and "time-border", using hexadecimal numbers with the format 0xRRGGBB, for example: 

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480 ! queue ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0xFFFFFF time-pattern="%b %Y %H:%M:%S" time-border=0x000000 time-enable=true ! queue ! imxvpuenc_h264 ! mpegtsmux ! filesink location=test_date_overlay.ts

Time and date Overlay

Unless your hardware has a battery backed-up real-time clock (RTC) and you have the Linux kernel configured to set the wall clock at power on, your time and date will start at 1970 when you turn on your target hardware. You can check the time/date setting by running 

date

You can also use the date command to set the current time and date. The format of the argument to set the time/date is MMDDhhmm[[CC]YY][.ss] For example, if you want to set the time/date to June 22nd, 2013 at 1:32:19 pm, use 

date 062213332013.19

You can run below pipeline to view date/time overlay: 

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480 ! queue ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0x000000 time-pattern="DATE: %m/%d/%y TIME:%H:%M:%S" time-border=0x000000 time-enable=true ! queue ! imxvpuenc_h264 ! mpegtsmux ! filesink location=test_date_time_overlay.ts

Image Overlay

The only supported image file format is PNG. The image size in the file must be have even dimensions. 

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=1280,height=720 ! queue ! emboverlay logo="RidgeRun.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! queue ! imxvpuenc_h264 ! mpegtsmux ! filesink location=test_image_overlay.ts

Mixed Overlay (text/logo/date)

Emboverlay can overlay text/date/logo in the same pipeline, the following is an example: 

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480 ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" text-enable=true time-offsetv=200 time-offseth=400 time-font-height=50 time-color=0x000000 time-pattern="%M:%S" time-border=0xFFFFFF time-enable=true logo="RidgeRun.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! imxvpuenc_h264 ! mpegtsmux ! filesink location=test_mix_overlay.ts

The logo used for the picture examples is 312 x 28 pixels in size and uses white (0xFFFFFF) as the transparency color. This logo has a transparency background, including the circle in the triangle being the transparency color.

Frame Number Overlay

Emboverlay element can overlay frame number, the following pipeline is an example: 

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=1280,height=720 ! queue ! emboverlay frame-offseth=40 frame-offsetv=40 frame-font-height=50 frame-color=0xFFFFFF frame-border=0x000000 frame-number=0 frame-enable=true ! queue ! imxvpuenc_h264 ! mpegtsmux ! filesink location=test_frame_number_overlay.ts

Video overlay Performance

Following you will find a series of pipelines using iMX6 hardware resources, with different emboverlay configuration and the corresponding performance results. All the test where run in a SabreSD Quad, capturing from the camera ov5640 mipi.

RidgeRun.png is an image of 1420x369.

DATE OVERLAY TO DISPLAY

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 !  emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0xFFFFFF time-pattern="%b %Y %H:%M:%S" time-border=0x000000 time-enable=true !  perf  print-arm-load=true ! imxv4l2sink

perf: perf0; timestamp: 0:09:17.739652084; bps: 746496000.000; mean_bps: 716636160.000; fps: 30.529; mean_fps: 30.046; cpu: 3; 
INFO:21.3 / 99:99:99.
perf: perf0; timestamp: 0:09:18.774840282; bps: 746496000.000; mean_bps: 718058057.143; fps: 29.946; mean_fps: 30.041; cpu: 4; 
INFO:22.3 / 99:99:99.
perf: perf0; timestamp: 0:09:19.807630621; bps: 746496000.000; mean_bps: 719350690.909; fps: 30.016; mean_fps: 30.039; cpu: 8; 
INFO:23.4 / 99:99:99.
perf: perf0; timestamp: 0:09:20.840615735; bps: 746496000.000; mean_bps: 720530921.739; fps: 30.010; mean_fps: 30.038; cpu: 5; 
INFO:24.5 / 99:99:99.
perf: perf0; timestamp: 0:09:21.873426667; bps: 746496000.000; mean_bps: 721612800.000; fps: 30.015; mean_fps: 30.037; cpu: 6; 
INFO:25.5 / 99:99:99.
perf: perf0; timestamp: 0:09:22.906231215; bps: 746496000.000; mean_bps: 722608128.000; fps: 30.015; mean_fps: 30.036; cpu: 4; 
INFO:26.5 / 99:99:99.
perf: perf0; timestamp: 0:09:23.939103223; bps: 746496000.000; mean_bps: 723526892.308; fps: 30.013; mean_fps: 30.035; cpu: 6; 
INFO:27.5 / 99:99:99.
perf: perf0; timestamp: 0:09:24.972092224; bps: 746496000.000; mean_bps: 724377600.000; fps: 30.010; mean_fps: 30.034; cpu: 7; 
INFO:28.6 / 99:99:99.
perf: perf0; timestamp: 0:09:26.004914875; bps: 746496000.000; mean_bps: 725167542.857; fps: 30.015; mean_fps: 30.034; cpu: 3; 
INFO:29.6 / 99:99:99.
perf: perf0; timestamp: 0:09:27.037792304; bps: 746496000.000; mean_bps: 725903006.897; fps: 30.013; mean_fps: 30.033; cpu: 5;

TIME & DATE OVERLAY TO DISPLAY

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 !  emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0x000000 time-pattern="DATE: %m/%d/%y TIME:%H:%M:%S" time-border=0x000000 time-enable=true ! perf  print-arm-load=true ! imxv4l2sink

perf: perf0; timestamp: 0:13:21.091886381; bps: 746496000.000; mean_bps: 642816000.000; fps: 29.998; mean_fps: 30.102; cpu: 5; 
INFO:07.8 / 99:99:99.
perf: perf0; timestamp: 0:13:22.092078714; bps: 746496000.000; mean_bps: 657627428.571; fps: 29.994; mean_fps: 30.087; cpu: 6; 
INFO:08.9 / 99:99:99.
perf: perf0; timestamp: 0:13:23.128191899; bps: 746496000.000; mean_bps: 668736000.000; fps: 29.920; mean_fps: 30.066; cpu: 4; 
INFO:09.9 / 99:99:99.
perf: perf0; timestamp: 0:13:24.128240200; bps: 746496000.000; mean_bps: 677376000.000; fps: 29.999; mean_fps: 30.058; cpu: 8; 
INFO:10.9 / 99:99:99.
perf: perf0; timestamp: 0:13:25.161603271; bps: 746496000.000; mean_bps: 684288000.000; fps: 29.999; mean_fps: 30.052; cpu: 5; 
INFO:11.9 / 99:99:99.
perf: perf0; timestamp: 0:13:26.194880485; bps: 746496000.000; mean_bps: 694656000.000; fps: 30.002; mean_fps: 30.048; cpu: 6; 
INFO:12.9 / 99:99:99.
perf: perf0; timestamp: 0:13:27.195047922; bps: 746496000.000; mean_bps: 698643692.308; fps: 29.995; mean_fps: 30.043; cpu: 5; 
INFO:13.9 / 99:99:99.
perf: perf0; timestamp: 0:13:28.228228773; bps: 746496000.000; mean_bps: 702061714.286; fps: 30.004; mean_fps: 30.040; cpu: 5; 
INFO:14.9 / 99:99:99.
perf: perf0; timestamp: 0:13:29.261612448; bps: 746496000.000; mean_bps: 705024000.000; fps: 29.999; mean_fps: 30.037; cpu: 8; 
INFO:15.9 / 99:99:99.
perf: perf0; timestamp: 0:13:30.261644359; bps: 746496000.000; mean_bps: 707616000.000; fps: 29.999; mean_fps: 30.035; cpu: 7;


IMAGE OVERLAY TO DISPLAY

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 ! emboverlay logo="RidgeRun.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! perf  print-arm-load=true ! imxv4l2sink

perf: perf0; timestamp: 0:30:27.344429931; bps: 746496000.000; mean_bps: 711854682.353; fps: 29.997; mean_fps: 29.301; cpu: 17; 
INFO:51.9 / 99:99:99.
perf: perf0; timestamp: 0:30:28.377718824; bps: 721612800.000; mean_bps: 712042338.462; fps: 28.066; mean_fps: 29.276; cpu: 19; 
INFO:52.9 / 99:99:99.
perf: perf0; timestamp: 0:30:29.422520234; bps: 746496000.000; mean_bps: 712692407.547; fps: 29.671; mean_fps: 29.284; cpu: 17; 
INFO:53.9 / 99:99:99.
perf: perf0; timestamp: 0:30:30.441589158; bps: 746496000.000; mean_bps: 713318400.000; fps: 30.420; mean_fps: 29.306; cpu: 17; 
INFO:55.0 / 99:99:99.
perf: perf0; timestamp: 0:30:31.475937667; bps: 746496000.000; mean_bps: 713921629.091; fps: 29.971; mean_fps: 29.318; cpu: 17; 
INFO:56.0 / 99:99:99.
perf: perf0; timestamp: 0:30:32.487874752; bps: 746496000.000; mean_bps: 714503314.286; fps: 28.658; mean_fps: 29.306; cpu: 16; 
INFO:57.0 / 99:99:99.
perf: perf0; timestamp: 0:30:33.487951270; bps: 721612800.000; mean_bps: 714628042.105; fps: 29.998; mean_fps: 29.319; cpu: 19; 
INFO:58.1 / 99:99:99.
perf: perf0; timestamp: 0:30:34.511117511; bps: 746496000.000; mean_bps: 715177489.655; fps: 30.298; mean_fps: 29.336; cpu: 17; 
INFO:59.1 / 99:99:99.
perf: perf0; timestamp: 0:30:35.522628189; bps: 746496000.000; mean_bps: 715708311.864; fps: 29.659; mean_fps: 29.342; cpu: 18; 
INFO:00.1 / 99:99:99.
perf: perf0; timestamp: 0:30:36.544478696; bps: 746496000.000; mean_bps: 716221440.000; fps: 30.337; mean_fps: 29.359; cpu: 17;


MIXED OVERLAY TO DISPLAY

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 ! emboverlay  text-offsetv=40 text-offseth=40 text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" text-enable=true time-offsetv=200 time-offseth=400 time-font-height=50 time-color=0x000000 time-pattern="%M:%S" time-border=0xFFFFFF time-enable=true logo="RidgeRun.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! perf  print-arm-load=true ! imxv4l2sink

perf: perf0; timestamp: 0:32:50.822063510; bps: 696729600.000; mean_bps: 628715520.000; fps: 27.940; mean_fps: 27.643; cpu: 16; 
INFO:16.5 / 99:99:99.
perf: perf0; timestamp: 0:32:51.869116122; bps: 721612800.000; mean_bps: 634521600.000; fps: 29.607; mean_fps: 27.774; cpu: 18; 
INFO:17.5 / 99:99:99.
perf: perf0; timestamp: 0:32:52.888813964; bps: 696729600.000; mean_bps: 638180894.118; fps: 26.478; mean_fps: 27.693; cpu: 16; 
INFO:18.5 / 99:99:99.
perf: perf0; timestamp: 0:32:53.919953969; bps: 696729600.000; mean_bps: 641433600.000; fps: 30.064; mean_fps: 27.832; cpu: 21; 
INFO:19.5 / 99:99:99.
perf: perf0; timestamp: 0:32:54.931583659; bps: 696729600.000; mean_bps: 644343915.789; fps: 25.701; mean_fps: 27.714; cpu: 17; 
INFO:20.5 / 99:99:99.
perf: perf0; timestamp: 0:32:55.955369192; bps: 671846400.000; mean_bps: 645719040.000; fps: 29.303; mean_fps: 27.797; cpu: 17; 
INFO:21.6 / 99:99:99.
perf: perf0; timestamp: 0:32:56.969140223; bps: 696729600.000; mean_bps: 648148114.286; fps: 23.674; mean_fps: 27.591; cpu: 16; 
INFO:22.6 / 99:99:99.
perf: perf0; timestamp: 0:32:57.988766998; bps: 622080000.000; mean_bps: 646963200.000; fps: 27.461; mean_fps: 27.585; cpu: 17; 
INFO:23.6 / 99:99:99.
perf: perf0; timestamp: 0:32:58.988867460; bps: 696729600.000; mean_bps: 649126956.522; fps: 29.997; mean_fps: 27.695; cpu: 20; 
INFO:24.6 / 99:99:99.
perf: perf0; timestamp: 0:33:00.022253756; bps: 746496000.000; mean_bps: 653184000.000; fps: 28.063; mean_fps: 27.711; cpu: 18;

DATE OVERLAY TO FILE

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 !  queue ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0xFFFFFF time-pattern="%b %Y %H:%M:%S" time-border=0x000000 time-enable=true !  perf  print-arm-load=true ! queue ! vpuenc_h264 ! mpegtsmux ! queue !  filesink location=test_date_overlay.ts

perf: perf0; timestamp: 0:37:01.541128773; bps: 746496000.000; mean_bps: 735049728.000; fps: 30.018; mean_fps: 30.030; cpu: 7; 
INFO:51.5 / 99:99:99.
perf: perf0; timestamp: 0:37:02.541171319; bps: 746496000.000; mean_bps: 735274164.706; fps: 29.999; mean_fps: 30.029; cpu: 6; 
INFO:52.5 / 99:99:99.
perf: perf0; timestamp: 0:37:03.574551646; bps: 746496000.000; mean_bps: 735489969.231; fps: 29.999; mean_fps: 30.028; cpu: 9; 
INFO:53.5 / 99:99:99.
perf: perf0; timestamp: 0:37:04.607776299; bps: 746496000.000; mean_bps: 735697630.189; fps: 30.003; mean_fps: 30.028; cpu: 6; 
INFO:54.5 / 99:99:99.
perf: perf0; timestamp: 0:37:05.641148282; bps: 746496000.000; mean_bps: 735897600.000; fps: 29.999; mean_fps: 30.027; cpu: 6; 
INFO:55.5 / 99:99:99.
perf: perf0; timestamp: 0:37:06.641274477; bps: 746496000.000; mean_bps: 736090298.182; fps: 29.996; mean_fps: 30.027; cpu: 8; 
INFO:56.6 / 99:99:99.
perf: perf0; timestamp: 0:37:07.674558449; bps: 746496000.000; mean_bps: 736276114.286; fps: 30.001; mean_fps: 30.026; cpu: 6; 
INFO:57.6 / 99:99:99.
perf: perf0; timestamp: 0:37:08.674585300; bps: 746496000.000; mean_bps: 736455410.526; fps: 29.999; mean_fps: 30.026; cpu: 8; 
INFO:58.6 / 99:99:99.
perf: perf0; timestamp: 0:37:09.725150163; bps: 746496000.000; mean_bps: 736628524.138; fps: 29.508; mean_fps: 30.017; cpu: 6; 
INFO:59.6 / 99:99:99.
perf: perf0; timestamp: 0:37:10.738326849; bps: 746496000.000; mean_bps: 736795769.492; fps: 30.597; mean_fps: 30.027; cpu: 6;


TIME & DATE OVERLAY TO FILE

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 !  queue ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0x000000 time-pattern="DATE: %m/%d/%y TIME:%H:%M:%S" time-border=0x000000 time-enable=true ! perf  print-arm-load=true ! queue ! vpuenc_h264 ! mpegtsmux ! queue ! filesink location=test_date_time_overlay.ts

perf: perf0; timestamp: 0:38:27.990823949; bps: 746496000.000; mean_bps: 718058057.143; fps: 29.994; mean_fps: 30.073; cpu: 9; 
INFO:21.9 / 99:99:99.
perf: perf0; timestamp: 0:38:28.990862770; bps: 746496000.000; mean_bps: 719350690.909; fps: 29.999; mean_fps: 30.069; cpu: 10; 
INFO:22.9 / 99:99:99.
perf: perf0; timestamp: 0:38:30.024142397; bps: 746496000.000; mean_bps: 720530921.739; fps: 30.002; mean_fps: 30.066; cpu: 10; 
INFO:23.9 / 99:99:99.
perf: perf0; timestamp: 0:38:31.057601896; bps: 746496000.000; mean_bps: 721612800.000; fps: 29.996; mean_fps: 30.063; cpu: 10; 
INFO:25.0 / 99:99:99.
perf: perf0; timestamp: 0:38:32.057684984; bps: 746496000.000; mean_bps: 722608128.000; fps: 29.998; mean_fps: 30.060; cpu: 12; 
INFO:26.0 / 99:99:99.
perf: perf0; timestamp: 0:38:33.057792944; bps: 746496000.000; mean_bps: 723526892.308; fps: 29.997; mean_fps: 30.058; cpu: 10; 
INFO:27.0 / 99:99:99.
perf: perf0; timestamp: 0:38:34.057871104; bps: 746496000.000; mean_bps: 724377600.000; fps: 29.998; mean_fps: 30.056; cpu: 10; 
INFO:28.0 / 99:99:99.
perf: perf0; timestamp: 0:38:35.057954133; bps: 746496000.000; mean_bps: 725167542.857; fps: 29.998; mean_fps: 30.053; cpu: 9; 
INFO:29.0 / 99:99:99.
perf: perf0; timestamp: 0:38:36.058041366; bps: 746496000.000; mean_bps: 725903006.897; fps: 29.997; mean_fps: 30.051; cpu: 7; 
INFO:30.0 / 99:99:99.
perf: perf0; timestamp: 0:38:37.058101800; bps: 746496000.000; mean_bps: 726589440.000; fps: 29.998; mean_fps: 30.050; cpu: 10;


IMAGE OVERLAY TO FILE

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 !  queue ! emboverlay logo="RidgeRun.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! perf  print-arm-load=true ! queue ! vpuenc_h264 ! mpegtsmux ! queue ! filesink location=test_image_overlay.ts

perf: perf0; timestamp: 0:39:30.937931965; bps: 746496000.000; mean_bps: 661893120.000; fps: 30.021; mean_fps: 30.313; cpu: 18; 
INFO:11.2 / 99:99:99.
perf: perf0; timestamp: 0:39:31.938094314; bps: 746496000.000; mean_bps: 669584290.909; fps: 29.995; mean_fps: 30.281; cpu: 18; 
INFO:12.2 / 99:99:99.
perf: perf0; timestamp: 0:39:32.938521568; bps: 746496000.000; mean_bps: 675993600.000; fps: 29.987; mean_fps: 30.255; cpu: 14; 
INFO:13.2 / 99:99:99.
perf: perf0; timestamp: 0:39:33.983390609; bps: 746496000.000; mean_bps: 681416861.538; fps: 29.669; mean_fps: 30.206; cpu: 17; 
INFO:14.2 / 99:99:99.
perf: perf0; timestamp: 0:39:35.004940653; bps: 746496000.000; mean_bps: 686065371.429; fps: 30.346; mean_fps: 30.217; cpu: 19; 
INFO:15.2 / 99:99:99.
perf: perf0; timestamp: 0:39:36.004998854; bps: 771379200.000; mean_bps: 691752960.000; fps: 29.998; mean_fps: 30.201; cpu: 16; 
INFO:16.2 / 99:99:99.
perf: perf0; timestamp: 0:39:37.005109277; bps: 721612800.000; mean_bps: 693619200.000; fps: 29.997; mean_fps: 30.187; cpu: 14; 
INFO:17.3 / 99:99:99.
perf: perf0; timestamp: 0:39:38.005217918; bps: 771379200.000; mean_bps: 698193317.647; fps: 29.997; mean_fps: 30.175; cpu: 16; 
INFO:18.3 / 99:99:99.
perf: perf0; timestamp: 0:39:39.021377723; bps: 746496000.000; mean_bps: 700876800.000; fps: 29.523; mean_fps: 30.137; cpu: 16; 
INFO:19.2 / 99:99:99.
perf: perf0; timestamp: 0:39:40.054240740; bps: 746496000.000; mean_bps: 703277810.526; fps: 30.014; mean_fps: 30.130; cpu: 20;

MIXED OVERLAY TO FILE

gst-launch-1.0 imxv4l2src device=/dev/video1 ! video/x-raw, width=1920, height=1080, framerate=30/1 ! emboverlay  text-offsetv=40 text-offseth=40 text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" text-enable=true time-offsetv=200 time-offseth=400 time-font-height=50 time-color=0x000000 time-pattern="%M:%S" time-border=0xFFFFFF time-enable=true logo="RidgeRun.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! perf  print-arm-load=true ! queue ! vpuenc_h264 ! mpegtsmux ! queue ! filesink location=test_mixed_overlay.ts

perf: perf0; timestamp: 0:40:51.132849004; bps: 696729600.000; mean_bps: 679164988.235; fps: 29.996; mean_fps: 29.534; cpu: 20; 
INFO:18.6 / 99:99:99.
perf: perf0; timestamp: 0:40:52.166909130; bps: 771379200.000; mean_bps: 684288000.000; fps: 29.979; mean_fps: 29.560; cpu: 22; 
INFO:19.6 / 99:99:99.
perf: perf0; timestamp: 0:40:53.199632053; bps: 746496000.000; mean_bps: 687562105.263; fps: 28.081; mean_fps: 29.478; cpu: 19; 
INFO:20.6 / 99:99:99.
perf: perf0; timestamp: 0:40:54.199672192; bps: 696729600.000; mean_bps: 688020480.000; fps: 29.999; mean_fps: 29.505; cpu: 20; 
INFO:21.6 / 99:99:99.
perf: perf0; timestamp: 0:40:55.200411948; bps: 746496000.000; mean_bps: 690805028.571; fps: 29.978; mean_fps: 29.529; cpu: 21; 
INFO:22.6 / 99:99:99.
perf: perf0; timestamp: 0:40:56.233213233; bps: 746496000.000; mean_bps: 693336436.364; fps: 30.015; mean_fps: 29.552; cpu: 19; 
INFO:23.6 / 99:99:99.
perf: perf0; timestamp: 0:40:57.261965790; bps: 746496000.000; mean_bps: 695647721.739; fps: 30.134; mean_fps: 29.579; cpu: 22; 
INFO:24.6 / 99:99:99.
perf: perf0; timestamp: 0:40:58.262849254; bps: 746496000.000; mean_bps: 697766400.000; fps: 29.974; mean_fps: 29.596; cpu: 20; 
INFO:25.6 / 99:99:99.
perf: perf0; timestamp: 0:40:59.266694163; bps: 746496000.000; mean_bps: 699715584.000; fps: 27.893; mean_fps: 29.525; cpu: 17; 
INFO:26.6 / 99:99:99.
perf: perf0; timestamp: 0:41:00.302570587; bps: 696729600.000; mean_bps: 699600738.462; fps: 27.030; mean_fps: 29.425; cpu: 17;

GStreamer Pipelines for IMX8

The pipelines and considerations for the IMX8 resemble the IMX6's, some slight changes are introduced due to a different H264 encoding element being used:

Video overlay examples

Text Overlay

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480 ! queue ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=50 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" text-enable=true font="Ubuntu" ! queue ! vpuenc_h264 ! mpegtsmux ! filesink location=test_text_overlay.ts

Date Overlay

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480 ! queue ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0xFFFFFF time-pattern="%b %Y %H:%M:%S" time-border=0x000000 time-enable=true ! queue ! vpuenc_h264 ! mpegtsmux ! filesink location=test_date_overlay.ts

Time and date Overlay

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=1280,height=720 ! queue ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0x000000 time-pattern="DATE: %m/%d/%y TIME:%H:%M:%S" time-border=0x000000 time-enable=true ! queue ! vpuenc_h264 ! mpegtsmux ! filesink location=test_date_time_overlay.ts

Image Overlay

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=1280,height=720 ! queue ! emboverlay logo="RidgeRun.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! queue ! vpuenc_h264 ! mpegtsmux ! filesink location=test_image_overlay.ts

Mixed Overlay (text/logo/date)

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=1280,height=720 ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="Hello overlay" text-enable=true time-offsetv=200 time-offseth=400 time-font-height=50 time-color=0x000000 time-pattern="%M:%S" time-border=0xFFFFFF time-enable=true logo="RidgeRunLogo.png" logo-offsetv=250 logo-offseth=300 logo-transparency=0 logo-enable=true ! vpuenc_h264 ! mpegtsmux ! filesink location=test_mix_overlay.ts

Frame Number Overlay

gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=1280,height=720 ! queue ! emboverlay frame-offseth=40 frame-offsetv=40 frame-font-height=50 frame-color=0xFFFFFF frame-border=0x000000 frame-number=0 frame-enable=true ! queue ! vpuenc_h264 ! mpegtsmux ! filesink location=test_frame_number_overlay.ts

GStreamer pipelines for NVIDIA Jetson

The following pipeline presents a Date Overlay using nvvidconv. 

gst-launch-1.0 nvarguscamerasrc num-buffers=90 \
! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12,framerate=(fraction)30/1' \
! nvvidconv ! 'video/x-raw' ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0xFFFFFF time-pattern="%b %Y %H:%M:%S" time-border=0x000000 time-enable=true \
! queue ! omxh264enc ! mpegtsmux ! filesink location=test_date_overlay.ts

GStreamer pipelines for x86

The following pipeline presents text overlay to display: 

gst-launch-1.0 videotestsrc is-live=true ! 'video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1' ! queue ! emboverlay text-offsetv=40 text-offseth=40 text-font-height=80 text-color=0xFFFFFF text-border=0x000000 text="The quick brown fox" text-enable=true font="Ubuntu" ! queue ! fpsdisplaysink sync=false

The following pipeline records a video with time overlay: 

gst-launch-1.0 videotestsrc is-live=true ! 'video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1' ! queue ! emboverlay time-offsetv=40 time-offseth=40 time-font-height=50 time-color=0xFFFFFF time-pattern="%b %Y %H:%M:%S" time-border=0x000000 time-enable=true ! queue ! x264enc ! mpegtsmux ! filesink location=test_date_overlay.ts

Building and installing on Ubuntu host

In order to build the project run the following commands. Note that the libdir variable corresponds to the Ubuntu 64 bits system. 

./autogen.sh
./configure  --prefix=/usr --libdir=/usr/lib/x86_64-linux-gnu/
make
sudo make install
gst-inspect-1.0 emboverlay

Building on Yocto

RidgeRun offers a Yocto layer containing RidgeRun commonly used packages. You can download this package from https://github.com/RidgeRun/meta-ridgerun.
It contains a recipe to build the plugin but you need to purchase Embedded Overlay with full source code, from https://shop.ridgerun.com/products/gstreamer-fast-text-graphics-overlay-1?_pos=2&_sid=9cfb1680e&_ss=r

Adding meta-ridgerun to your Yocto build

First you need to copy meta-ridgerun in your sources directory: 

cp -r meta-ridgerun $YOCTO_DIRECTORY/sources/

Then add the RidgeRun meta layer to your bblayers.conf file. First, go to the build configuration directory 

cd $YOCTO_DIRECTORY/build/conf/

Open the bblayers.conf file and add the RidgeRun meta layer path $YOCTO_DIRECTORY/sources/meta-ridgerun to BBLAYERS

Building Embedded Overlay

  • Once you have access to the repository, please open gst-emboverlay_1.0.bb in $YOCTO_DIRECTORY/sources/meta-ridgerun/recipes-multimedia/gstreamer/
  • Modify the following line in SRC_URI with the correct emboverlay URL by changing <Customer-Directory> with your own.
git://git@gitlab.com/RidgeRun/orders/<Customer-Directory>/gst-emboverlay.git;protocol=ssh;branch=${SRCBRANCH} \
  • Make sure you have added your ssh key to your GitLab account. For more information: SSH Information
  • Finally build recipe:
bitbake gst-emboverlay

SSH Information

The recipe will fetch the repository using your ssh key, thus it's necessary to add the key to the Gitlab page.
For adding it, go to Settings->SSH Keys and add your key, if don't have one you can find there a link about how to generate it.

SHH Issue
If the GitLab key hasn't added to your list of known hosts on the PC, you will have fetch errors when trying to build an Embedded Overlay recipe.
One easy way to add the key is when cloning the repository for the first time from GitLab, it will ask if you want to add the key to your list of known hosts.

Example: 

git clone git@gitlab.com:RidgeRun/orders/<Customer-Directory>/gst-emboverlay.git

Emboverlay versus cairo overlay

cairo overlay

Works in I420 color space. In order to use cairo overlay, code modifications need to be done. GStreamer DMAI encoders work in NV12 color space.

cairo posses three GStreamer elements capable of:

  1. Blit text
  2. Blit time
  3. Render

cairo overlay element properties:

cairo text: Adds text strings on top of a video buffer

  1. name
  2. text
  3. shaded-background
  4. valign
  5. halign
  6. xpad
  7. ypad
  8. deltax
  9. deltay
  10. font-desc

cairo time: Overlays the time on a video stream

  1. name
  2. qos (handle quality of service)

cairo render: Encodes streams using Cairo

  1. name

emboverlay

RR Emboverlay works in NV12, this allows to use emboverlay with Dmai encoders and TIDmaiVideoSink

Emboverlay is one GStreamer element capable of:

  • Blit text
  • Blit time
  • Blit logo

emboverlay element properties:

Emboverlay is just one GStreamer element, nevertheless, it will be documented as a text element, time element, and logo element

emboverlay text:

  1. text
  2. text-offseth
  3. text-offsetv
  4. text-font-height
  5. text-color
  6. text-border

emboverlay time:

  1. time-offsetv
  2. time-offseth
  3. time-font-height
  4. time-color
  5. time-border
  6. time-pattern

emboverlay logo:

  1. logo (png file with/without transparence)
  2. logo-offseth
  3. logo-offsetv
  4. logo-transparency

emboverlay versus cairo overlay fps test

fps tests were done in a LeopardBoard DM365 with a 5Mp video camera and overlaying just text

No overlay

Aprox: 29-30 fps

cairotextoverlay

Approx: 12 fps

emboverlay

Aprox: 29-30 fps


RidgeRun Resources

Quick Start Client Engagement Process RidgeRun Blog Homepage
Technical and Sales Support RidgeRun Online Store RidgeRun Videos Contact Us
RidgeRun.ai: Artificial Intelligence | Generative AI | Machine Learning

Contact Us

Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering informations are available in RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page.
Retrieved from "http://developer.ridgerun.com/wiki/index.php?title=Fast_GStreamer_overlay_element&oldid=48996"