ElectronicStabilizer Class

Table of Contents [Sticky]

Introduction

The ElectronicStabilizer class provides an end-to-end video stabilization pipeline driven by IMU measurements. It collects IMU samples, integrates them into camera orientations, interpolates the orientation timeline at video frame timestamps, smooths the orientation trajectory, and applies the resulting correction through an undistort backend.

The stabilizer owns the complete processing chain: IMU acquisition, integration, interpolation, smoothing, and image correction. It is designed to process a continuous stream, so the same stabilizer instance should be reused across consecutive frames.

Pipeline Overview

The electronic stabilization pipeline is composed of five main stages:

IMU acquisition: reads raw IMU samples from the selected IMU backend.
Orientation integration: integrates IMU samples into an orientation timeline.
Interpolation: samples the integrated orientation at the current frame timestamp and neighboring timestamps.
Orientation smoothing: smooths the interpolated orientation window to remove unwanted camera motion.
Image compensation: applies the correction quaternion to the frame through the selected undistort backend.

The processing flow is:

Configure the electronic stabilization parameters.
Create an ElectronicStabilizer instance.
Start the stabilizer, or let Apply() start it lazily on the first frame.
Collect IMU samples automatically with the internal IMU thread, or manually with CollectSensorSample().
Set timestamps on the input frames using the same time base as the IMU data.
Call Apply() once per frame.
Check the returned RuntimeError.
Call Stop() when the stream finishes.

Parameters

The ElectronicStabilizerParams structure configures the complete electronic stabilization chain.

Parameter	Description	Default
`sensor`	IMU backend used to collect inertial samples.	`kBmi160`
`integrator_algorithm`	Algorithm used to integrate IMU samples into orientations.	`kSimpleComplementaryIntegrator`
`interpolator_algorithm`	Algorithm used to interpolate orientations at frame timestamps.	`kSlerp`
`undistort_algorithm`	Backend used to compensate each video frame.	`kFishEyeOpenCV`
`smoothing_algorithm`	Algorithm used to smooth interpolated orientation samples.	`kSphericalExponential`
`sensor_settings`	IMU runtime settings, including device selection.	New `SensorSettings`
`sensor_params`	IMU acquisition parameters, including sample rate and axis orientation.	New `SensorParams`
`integrator_settings`	Integrator-specific configuration.	New `IntegratorSettings`
`interpolator_settings`	Interpolator-specific configuration.	New `InterpolatorSettings`
`smoothing_params`	Smoothing-specific configuration. If null, compatible default smoothing parameters are created during configuration.	`nullptr`
`camera_matrix`	Camera intrinsic matrix used by the undistort backend.	Zero matrix
`distortion_coefficients`	Camera distortion coefficients used by the undistort backend.	Empty vector
`has_camera_matrix`	Enables applying `camera_matrix` and `distortion_coefficients` to the undistort backend.	`false`
`use_sensor_thread`	Enables the stabilizer-owned background IMU polling thread.	`true`
`fov_scale`	Field-of-view scale passed to the undistort backend.	`0.95`
`max_sensor_samples`	Maximum number of raw IMU samples kept in the pending queue.	`16`
`camera_timestamp_offset_us`	Offset added to IMU timestamps to align them with camera timestamps.	`0`
`frame_rate_hz`	Expected camera frame rate, used to estimate neighboring interpolation timestamps.	`30`
`interpolation_past_samples`	Number of previous frame orientations included in the smoothing window.	`1`
`interpolation_future_samples`	Number of future frame orientations included in the smoothing window.	`1`
`initial_orientation`	Orientation used to initialize the integrator and smoothing algorithm.	`(0.0, 0.70710678, 0.0, 0.70710678)`

Methods

The class provides the following public methods:

ElectronicStabilizer: constructs the stabilizer using optional pipeline parameters, runtime settings, and logger. If no parameters are provided, the default ElectronicStabilizerParams values are used. Processing backends are created lazily when Start() or Apply() is first called.
Apply: stabilizes one input frame and writes the corrected image to the output frame. The templated overload validates that the input and output frames use the expected allocator type. If the stabilizer has not been started, Apply() starts it automatically.
Start: configures the processing backends, starts the selected IMU backend, and starts the internal IMU polling thread when use_sensor_thread is enabled. Calling Start() more than once is allowed.
Stop: stops the IMU polling thread, stops the IMU backend, clears internal IMU and interpolation buffers, and releases processing backend instances.
Reset: stops the stabilizer and restores the default parameter set.
CollectSensorSample: collects one IMU sample when manual collection is enabled. This method is only available when use_sensor_thread is false.
UpdateParams: stops the active pipeline, replaces the current parameter set, and reconfigures the processing backends.
SetRuntimeSettings: updates the runtime settings used by the undistort backend.
GetRuntimeSettings: returns the active runtime settings from the undistort backend when available, or the pending runtime settings otherwise.
GetParams: returns a copy of the current electronic stabilization parameters.

IMU Collection Modes

The stabilizer supports two IMU collection modes.

With use_sensor_thread = true, the stabilizer starts an internal polling thread during Start(). This mode is useful when the application wants the stabilizer to own IMU acquisition.

With use_sensor_thread = false, the application is responsible for calling CollectSensorSample(). This mode is useful when the application already owns the IMU loop or needs tighter control over when samples are collected.

In both modes, collected IMU samples are stored in an internal pending queue. The queue is bounded by max_sensor_samples. During Apply(), pending samples are integrated and used to update the orientation timeline.

Timestamp Alignment

Frame timestamps are central to the electronic stabilization pipeline. The input frame timestamp is used to interpolate the integrated orientation timeline at the moment the frame was captured. IMU timestamps can be shifted with camera_timestamp_offset_us to align the IMU clock with the camera clock.

The interpolation window contains previous samples, the current frame sample, and future samples. The number of previous and future samples is controlled by interpolation_past_samples and interpolation_future_samples. The total interpolation window must contain at least three samples.

If the stabilizer does not have enough new IMU data for the requested frame, it reuses the previous correction and still applies the image compensation stage.

Camera Calibration

Camera calibration can be provided through camera_matrix and distortion_coefficients. Set has_camera_matrix to true to apply these values to the undistort backend. The camera matrices are pushed to the undistort backend on the first frame once the input frame dimensions are known.

Calibration is recommended for accurate image compensation, especially for fisheye lenses and wide field-of-view cameras.

Runtime Settings

Runtime settings are used by the selected undistort backend. If no runtime settings are provided, the stabilizer creates default settings according to the selected undistort algorithm. OpenCV, OpenCL, and CUDA undistort backends require compatible image memory and runtime settings.

The input and output frames must use the same allocator type. The templated Apply() overload can also validate that both images use the expected allocator family.

Basic Usage

The following example shows the basic structure for configuring and using the electronic stabilizer. The exact IMU settings, camera calibration, and image allocation depend on the target platform.

#include <memory>
#include <rvs/allocators/host.hpp>
#include <rvs/settings/opencv.hpp>
#include <rvs/smoothing/spherical-exponential.hpp>
#include <rvs/stabilizers/electronic.hpp>

auto params = std::make_shared<rvs::ElectronicStabilizerParams>();
params->sensor = rvs::Sensors::kBmi160;
params->integrator_algorithm =
    rvs::IntegratorAlgorithms::kSimpleComplementaryIntegrator;
params->interpolator_algorithm = rvs::InterpolatorAlgorithms::kSlerp;
params->undistort_algorithm = rvs::UndistortAlgorithms::kFishEyeOpenCV;
params->smoothing_algorithm =
    rvs::SmoothingAlgorithms::kSphericalExponential;

params->use_sensor_thread = true;
params->frame_rate_hz = 30;
params->fov_scale = 0.95;
params->max_sensor_samples = 16;
params->camera_timestamp_offset_us = 0;
params->interpolation_past_samples = 1;
params->interpolation_future_samples = 1;

params->smoothing_params =
    std::make_shared<rvs::SphericalExponentialParams>(
        0.2, params->initial_orientation);

params->camera_matrix = camera_matrix;
params->distortion_coefficients = distortion_coefficients;
params->has_camera_matrix = true;

auto runtime_settings = std::make_shared<rvs::OpenCVRuntimeSettings>();
rvs::ElectronicStabilizer stabilizer(params, runtime_settings);

rvs::RuntimeError ret = stabilizer.Start();
if (ret.IsError()) {
  /* Handle the start error. */
}

for (;;) {
  std::shared_ptr<rvs::IImage> input_frame = GetNextInputFrame();
  std::shared_ptr<rvs::IImage> output_frame = GetOutputFrame();

  input_frame->SetTimestamp(frame_timestamp_us);
  output_frame->SetTimestamp(frame_timestamp_us);

  ret = stabilizer.Apply<rvs::HostAllocator>(output_frame, input_frame);
  if (ret.IsError()) {
    /* Handle the frame error. */
    break;
  }

  DisplayOrStoreFrame(output_frame);
}

stabilizer.Stop();

Manual IMU Collection

When the application owns the IMU polling loop, disable the internal IMU thread and call CollectSensorSample() explicitly.

params->use_sensor_thread = false;

rvs::ElectronicStabilizer stabilizer(params, runtime_settings);
stabilizer.Start();

/* Usually called from the application IMU loop. */
std::shared_ptr<rvs::SensorPayload> sample_copy =
    std::make_shared<rvs::SensorPayload>();
stabilizer.CollectSensorSample(sample_copy);

/* Usually called from the video processing loop. */
input_frame->SetTimestamp(frame_timestamp_us);
output_frame->SetTimestamp(frame_timestamp_us);
stabilizer.Apply<rvs::HostAllocator>(output_frame, input_frame);

stabilizer.Stop();

Backend Selection

The electronic stabilizer is composed from several backend families:

IMU: collects raw inertial measurements from the selected IMU.
Integrator: converts raw IMU samples into an orientation timeline.
Interpolator: samples the orientation timeline at frame-related timestamps.
Smoothing: smooths the interpolated orientation window. Common choices include spherical exponential smoothing and fixed horizon smoothing.
Undistort: applies the final image compensation using the selected image backend.

The selected backends must be available in the current build and compatible with the runtime settings and image allocator used by the application.

Error Handling

Most methods return RuntimeError. Applications should check this return value after each call. Common error sources include null input or output frames, incompatible allocators, missing IMU settings, missing IMU parameters, unavailable backends, invalid interpolation window sizes, IMU start failures, or insufficient IMU data for the current frame.

Example Applications

The repository includes concept examples that show the lower-level stabilization stages used by electronic stabilization:

examples/concept/complete-offline-example.cpp
examples/concept/complete-online-example.cpp

These examples demonstrate IMU sample handling, integration, interpolation, smoothing, camera calibration, runtime settings, and image correction.

Reference Documentation

Reference Documentation: ElectronicStabilizer

❯

Share This Page

Send Email Using