Basics and Foundation - Video Stabilization Algorithms with IMU

Table of Contents [Sticky]

Algorithms with IMU

Here we introduce some of the algorithms that are essential parts of the overall video stabilization process.

Integration

In this context the use of an integration is done to determine the orientation of an object from its angular velocity, which is the rate of change of its orientation over time. Essentially, it transforms the angular velocity (rotation rate) into the actual orientation.

However, this approach does not incorporate data from accelerometers, which measure linear acceleration. Accelerometer data can be fused with gyroscope data (which provides the angular velocity) in a process known as sensor fusion. This fusion can yield a more accurate estimate of the object’s orientation.

Simple integration

The simple integration algorithm takes only the gyroscope measurements which are the rate of change in rotation to obtain the given rotation, using an initial value of orientation.

However, this approach does not incorporate data from accelerometers, which measure linear acceleration. Accelerometer data can be fused with gyroscope data (which provides the angular velocity) in a process known as sensor fusion. This fusion can yield a more accurate estimate of the object’s orientation.

VQF

The Versatile Quaternion Based Filter or VQF is proposed in the paper Highly Accurate IMU Orientation Estimation with Bias Estimation and Magnetic Disturbance Rejection and it uses a gyroscope bias estimation algorithm and an algorithm for magnetic disturbance detection and rejection.

The full version of the algorithm, includes additional features such as rest detection, gyroscope bias estimation, and magnetic disturbance rejection. Notably, the gyroscope bias estimation method in VQF avoids reliance on magnetometer corrections, which enhances its resilience against magnetic disturbances. Instead, the bias estimation is based solely on the disagreement between strapdown integration and accelerometer measurements during motion. This design choice helps maintain accuracy and robustness in challenging environments.

Madgwick

This algorithm uses a quaternion representation, allowing accelerometer and magnetometer data to be used in an analytically derived and optimised gradient-descent algorithm and it proposed in the paper An Efficient Orientation Filter for Inertial and Inertial/Magnetic Sensor Arrays.

Performance evaluation compared this filter with a proprietary Kalman-based algorithm used in orientation sensors. The results demonstrate that the filter achieves higher levels of accuracy than the Kalman-based method. Notably, the filter's low computational load and ability to operate at low sampling rates present new opportunities for real-time applications with IMU and MARG sensor arrays.

Complementary

The complementary algorithm is proposed in the paper A Quaternion-Based Orientation Filter for IMUs and MARGs and it uses a algebraic solution of a system to obtain a quaternion estimation.

Incorporating a complementary filter, the system analyzes signals in the frequency domain to combine them effectively. By applying high-pass filtering on gyroscope data (affected by low-frequency noise) and low-pass filtering on accelerometer data (affected by high-frequency noise), the filter aims to achieve an all-pass and noise-free attitude estimation. This complementary filtering process is crucial for accurate attitude estimation from IMU readings.

Interpolation

Sensor data and video frames times might not be synchronized, to smooth the orientation at each frame we need the orientation values for each corresponding frame. This is why we need to interpolate the values for the sensor to obtain the orientation values at each frame.

Spherical linear interpolation (Slerp)

Slerp is a method of interpolation on the surface of a unit sphere. Given two points on the sphere, SLERP provides a smooth curve that follows the shortest path on the sphere’s surface between these two points. The speed along this curve is constant, which makes SLERP particularly useful for creating smooth transitions.

Slerp is often used to interpolate between two orientations or rotations. Slerp is often used with quaternions, allowing to create smooth rotational motion between two orientations.

Smooth orientation

Spherical Exponential Smoothing

Exponential smoothing is a time series forecasting method that uses weighted averages of past observations. The weights decrease exponentially as the observations get older, hence the name "exponential smoothing". It’s a powerful method that can handle data with a systematic trend or seasonal component. We apply the slerp interpolation to the exponential smoothing to obtain more stable orientations.

Horizon lock

The Horizon Lock technique leverages data from both the gyroscope and accelerometer. Specifically, it utilizes accelerometer data to determine the direction of gravity. This information is instrumental in stabilizing footage along the horizon.

Undistortion

In order to apply the required transformation to the original frames a rotation is computed between the unstable and stable orientations. This rotation is then transformed into a rotation matrix, serving as a rectification transformation. This transformation rectifies the footage from the unstable space (original orientation) to the stable space (desired orientation). This process is analogous to the undistortion performed on raw images captured by a camera, which corrects for lens distortions. Following this, a set of mapping functions are derived for the image. These functions account for transformations such as translations, rotations, scalings, and cropping, ultimately resulting in stabilized footage.