DM36x Video Stabilization

From RidgeRun Developer Connection
Jump to: navigation, search

RR Eval SDK download button.png RR Pre built demo image download.png RR Contact Us.png


Video Stabilization (vs) is a video enhancement technique which aims to remove shaky motion from videos. We offer a video stabilization solution for dm36x that uses the TI's video stabilization library. This solution moves the position of the upper-left corner of the frame in the opposite direction of the camera movement.

The TI's video stabilization library provides an algorithm for video stabilization. This algorithm takes the data provided by the Boundary Signal Calculator module for the current and previous frame and returns the dimensions and coordinates of the upper-left corner of a sub-frame in the video frame. The changes in the corner coordinates compensates the camera movement.

The Boundary Signal Calculator(BSC) is a hardware module part of the dm36x Image Pipe (Ipipe). The BSC generates vectors of sum of rows and columns pixels. This hardware module is programmable to fit with the frame's properties. This hardware is part of the image processing path for a raw CCD/CMOS data, so BSC hardware accelerated video stabilization can only be used with sources that provide Bayer data.

Video stabilization algorithm compensates the camera movement which is up to +/- 10% of the sub-frame width in horizontal direction and sub-frame height in vertical direction. Therefore, when we use the video stabilization we capture an image of 120% the desired output image's width and height.

There is a video stabilization video showing the technology in action.


RidgeRun video stabilization support is composed by three elements:

  • A kernel support for the BSC module.
  • Ipiped functionality for BSC configuration.
  • A gstreamer plugin for video stabilization.

BSC kernel support

The kernel patch includes the BSC module in the previewer driver. That allows you to specify the desired configuration parameters for the BSC hardware module.

If the BSC is configured and enabled before a video streaming is initialized, it will attach the BSC interrupt(IPIPE_INT_BSC). This interrupt is triggered when boundary signal calculation is finished for each frame. Since BSC is a sub-module of the Ipipe it always operates on the input video frame. To keep the relation between the frame the BSC calculations, when the interrupt occurs the BSC data is copy to the end of the current frame.

BSC in ipiped

In order to apply video stabilization to a video, the BSC module must be configured first. The ipiped offers the capability of configure the BSC according to the image's width and height. It enables the BSC module and defines the parameters that uses the module maximum capacity and best fits the image dimensions.

Also, the corresponding ipipe-client command to access this functionality is provided: config-vs. e.g config-vs 768 576

This command is not build by default with the ipiped because it requires the kernel be patched to supported. If you applied the video_stabilize_support.patch to the kernel and want to use the command, you need to uncomment the lines:

   ipiped_VALAFLAGS += --define=BSC_ENABLE
  ipipe_client_VALAFLAGS += --define=BSC_ENABLE

in the and recompile the ipiped.

Video Stabilization plugin

The gstreamer plugin for video stabilization is called dm365vs. dm365vs adds the capability of reduce the video shaking in a preview pipeline.

The video stabilization needs to move a sub-frame in the video frame in the opposite direction to the camera movement. Therefore the dm365vs negotiates a transform of the image capabilities to reduce the input image. The plugin looks for an input image's width of 120% of the output image's width and for an input image's height of 120% of the output image's height. e.g an input frame of 768x576 results in an output frame of 640x480

During the initialization this plugin configures the TI's vs algorithm with the parameters of the BSC and the input image's dimensions.

When each buffer arrives to the dm365vs, it inputs the BSC data attached to the buffer and the BSC data of the previous buffer in the TI's video stabilization algorithm. The VS algorithm returns the coordinates of the upper left corner of a sub-frame and size of a sub-frame in the video frame and the dm365vs adjust this results to the negotiated output image's size. The calculated position of the upper-left corner of the frame is pushed in the output dmai transport frame.


First configure BSC module and be sure to set the ipipe mode to continuous mode first:

ipipe-client set-previewer-mode cont
ipipe-client config-vs 768 576 

Then run the video stabilization with your video pipeline:

 gst-launch v4l2src chain-ipipe=false always-copy=false ! video/x-raw-yuv,format=\(fourcc\)NV12, width=768, height=576 ! dm365vs !  TIDmaiVideoSink

720p streaming example

ipipe-client set-previewer-mode cont
ipipe-client config-vs 1536 864 

Board pipeline with video stabilization:

gst-launch -e v4l2src always-copy=FALSE chain-ipipe=false ! video/x-raw-yuv,format=\(fourcc\)NV12, width=1536, height=864 ! queue ! dmaiaccel ! dm365vs ! dmaiperf ! dmaienc_h264 encodingpreset=2 ratecontrol=2 intraframeinterval=23 idrinterval=46 targetbitrate=3000000 ! rtph264pay ! udpsink port=$HOST_PORT host=$HOST_IP sync=false enable-last-buffer=false

Host receiving streaming pipeline:

gst-launch -v udpsrc port=3000 ! 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, sprop-parameter-sets=(string)"Z2QAKK2EBUViuKxUdCAqKxXFYqOhAVFYrisVHQgKisVxWKjoQFRWK4rFR0ICorFcVio6ECSFITk8nyfk/k/J8nm5s00IEkKQnJ5Pk/J/J+T5PNzZprQCgC3I\,aO48sA\=\=", payload=(int)96' ! rtph264depay ! decodebin ! xvimagesink sync=false async=false