Library Integration for IMU - Computing the Stabilization

< >

RidgeRun Video Stabilization Library

Table of Contents [Sticky]

• Basics and Foundation
  • What is Video Stabilization
  • Video Stabilization Process using IMU
  • Stabilization Algorithms with IMU
• Getting Started
  • Supported Platforms, Sensors and Backends
  • Evaluating the Library
  • Getting the Code
  • Building the Library
• API Reference
  • Library Architecture
  • Adding New Sensors
  • Adding Algorithms
  • Adding Stabilization Algorithm
  • Adding Backends
  • API Documentation
  • Examples Guidebook
• Library Integration for IMU
  • Preparing the Video
  • Preparing the IMU Measurements
  • Measurement Integration
  • Interpolation
  • Computing the Stabilization
  • Video Undistortion
  • Example Application
• GStreamer
  • Add Support for Video Timestamps
  • Add Support for IMU Sensors
  • Video Stabilizer Element
  • Example Pipelines
• Useful Links
  • BMI160 Setup
  • Performance
• Contact Us

Introduction

After determining the angular position of the current frame, it is time to determine the distortion angle. The distortion angle represents how much the image deviated from a natural movement (a natural movement has a low frequency). Therefore, the distortion angles are often unwanted high-frequency noise and create waviness in the video.

Stabilization Principle

The RidgeRun Video Stabilization Library computes the distortion based on forward-backwards filtering processing using the sleep interpolation between consecutive samples and a smoothing factor. This is known as the Exponential Smoothing.

The basic representation of the Exponential Smoothing is given by:

$s_{t}=\alpha x_{t}+(1-\alpha )s_{t-1}$

where $\alpha$ is the smoothing factor.

The process of smoothing is performed forward in time. Then, using the computed data, it is computed backwards.

Overall, the RidgeRun Stabilization Library requires at least three interpolated samples to perform the stabilization: one from the past, the current, and the following, leading to a non-causal filter. This means that stabilization will introduce a latency of two image frames. As a limitation, the first frame will always be unstabilized, whereas the second frame will be stabilized at the time when the third frame should be processed.

Using the Stabilizers

The stabilizers require a special set of configurations for each algorithm; however, their usage is uniform in terms of API. Currently, the library contains two algorithms:

• Spherical Exponential: the most basic one.
• Fixed Horizon: an enhanced version of the Spherical Exponential. Similar to the Horizon Lock.

Creating the Settings

Each algorithm has a special set of settings:

• Spherical Exponential: SphericalExponentialParams
• Fixed Horizon: FixedHorizonParams

Depending on the algorithm, different members must be configured. For instance:

SphericalExponentialParams:

// Smoothing factor:
double smoothing = 0.206;

auto params = std::make_shared<SphericalExponentialParams>(smoothing);

FixedHorizonParams:

// Smoothing factor:
double smoothing = 0.101;
// Angle
double angle = M_PI;

auto params = std::make_shared<FixedHorizonParams>();
params->smoothing = smoothing;
params->horizon_angle = angle;

Creating the Stabilizer Instance

The creation of the stabilizer instance is through the factory available in IStabilizer.

// Assume that the settings match the implementation
std::shared_ptr<StabilizerParams> params;

// Create the instance
auto stabilizer = IStabilizer::Build(kSphericalExponential, params);

In case of providing a wrong set of parameters, it will throw an exception during construction.

Using the Stabilizer

Once created and configured, it is possible to use the Apply() method as in the former algorithms. In this case, the input vector of quaternions must have at least three values: one past sample, the present sample and a future sample.

// Assume that the interpolation has filled the structure and has three quaternions
std::vector<std::pair<Quaternion<Double>, uint64_t>> interpolated;

// Correction
Quaternion<double> correction;

// Apply the correction
stabilizer->Apply(correction, interpolated, 0);

// Delete the first sample
interpolated.erase(interpolated.begin());

The Apply() method do not alter the interpolated. The first sample must be deleted to have room for the following.

Apart from the correction, the quaternion must be rotated (multiply the conjugated of the correction by the current interpolated sample) against the input sample to compute the correction:

// Assume that the interpolation has filled the structure and has two quaternions
// The first is already deleted
std::vector<std::pair<Quaternion<Double>, uint64_t>> interpolated;

// Correction
Quaternion<double> correction;

// Rotate between: this quaternion can be used by the undistort
auto rotated = correction.Conjugate() * interpolated.at(0);

The resulting quaternion is ready to use by the undistort.

Result

Inspecting the result, the interpolated plot is the following:

Interpolated Plot

It indicates the position of the system at a given time. The stabilization captures the noise, which is often small angles:

Angles to Stabilize Angles

These plots are the result of converting the quaternion to Euler Angles.

< >