Birds Eye View - RidgeRun's Birds Eye View Background

Follow us on:

Share This Page

Facebook Twitter LinkedIn WhatsApp Email Copy Link

Close

Send Email Using

Gmail (Web) Thunderbird (default mail app)

Close

⇦ How BEV Works/Overview

Home

How BEV Works/Birds Eye View and Libpanorama ⇨

Getting Started
Quickstart Guide Evaluating the Product How to Get the Code
How BEV Works
Overview Background Birds Eye View and Libpanorama
User Guide
Building and Installation Minimal Application
Calibration Guide
Understanding the Calibration Process Preparing the Calibration Images The Calibration Tool GUI Performing the Calibration
Calibration Guide Legacy
GStreamer
GstBEV Plugin Generic Pipelines Jetson Pipelines iMX8 Pipelines
Performance
PC Profiling NVIDIA Jetson Thor and Orin NXP iMX8
Contact Us

Birds Eye View (BEV) is a real-time surround-view solution that transforms multiple camera images into a single top-down view around a vehicle or robot. RidgeRun's Birds Eye View implementation uses inverse perspective mapping, homography, camera calibration, and fisheye rectification to project each camera onto a shared ground plane. This page explains the mathematical and computer vision concepts behind that transformation, along with the assumptions and performance considerations that matter in embedded BEV systems.

Perspective Transformation

When cameras are mounted on a vehicle with their field of views facing away from it, the captured images exhibit strong perspective distortion, as shown in Fig. 1(a). This perspective effect makes it difficult for drivers—and computer vision algorithms—to accurately judge distances, detect obstacles, or perform reliable image analysis. To address this, the camera views must be geometrically transformed into a top-down, birds eye view representation, as illustrated in Fig. 1(b). By applying perspective transformation (or inverse perspective mapping), the raw images are converted into a normalized birds eye view, shown in Fig. 1(c), which enables accurate perception, measurement, and scene understanding for both human operators and automated systems. [3]

Fig 1: Illustration of perspective transformation from a lateral camera view to a top-down elevated view, taken from [1]

To obtain the output birds eye view image the following projection matrix can be used. It maps the relationship between pixel (x,y) of the birds eye view image and pixel (u,v) from the input image.

To generate the output birds eye view image, a geometric transformation based on a projection matrix is applied. This matrix defines the mapping between each output pixel (x,y) in the bird’s-eye view and its corresponding pixel (u,v) in the original camera image. By using this projection relationship, the system maps the ground-plane region from the input image into a top-down perspective. The transformation is expressed as:

$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ w^{\prime }\end{array}\right]=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right]\left[\begin{array}{c}u\\ v\\ w\end{array}\right]wherex=\frac{x^{\prime }}{w^{\prime }}andy=\frac{y^{\prime }}{w^{\prime }}.$

This transformation is commonly referred to as Inverse Perspective Mapping (IPM) [4]. IPM takes a frontal-view image as input, applies a homography, and generates a top-down (bird’s-eye) view of the scene by mapping pixels onto a 2D ground-plane frame. In the region immediately surrounding a vehicle—where the road surface can be approximated as planar—IPM works effectively and produces geometrically meaningful top-down projections.

However, IPM introduces distortions for objects farther from the camera because the mapping is non-homogeneous. Vertical structures such as other vehicles, pedestrians, street signs, or curbs become stretched, skewed, or compressed, as illustrated in the left image of Figure 2. These artifacts limit the accuracy and usable range of applications that depend solely on basic IPM.

More advanced post-transformation techniques, such as Incremental Spatial Transformer [5], can significantly improve geometric consistency at longer distances, producing results similar to the right image in Fig. 2. For the purposes of this project, however, we use a standard IPM approach to illustrate the core concepts.

When IPM is applied without additional correction or refinement, the method relies on three key assumptions:

1. The camera remains in a fixed pose relative to the ground plane.
2. The road or ground surface is approximately planar.
3. The ground surface is free of obstacles or significant height variations.

Fig 2: Comparison between the top-down view generated with Inverse Perspective Mapping and Incremental Spatial Transformer, taken from [2]

To apply IPM correctly and to generate an accurate birds eye view projection, we must understand how a camera maps 3D points in the world onto 2D image coordinates. This relationship is defined by the camera model, which combines intrinsic parameters, extrinsic parameters, and the geometry of the ground plane. From this formulation, we can derive a homography matrix that transforms ground-plane pixels from the input image into their corresponding positions in the birds eye view frame. The following section describes this camera model and the homography used to compute the transformation.

Camera Model and Homography

A pinhole camera can be modeled as a geometric device that projects a 3D point in the world onto a 2D point in the image plane. This projection is described by the general camera equation [6]:

$[u,v,1{]}^t\simeq P_{3x4}\cdot [X,Y,Z,1{]}^t(1)$

Here, $P_{3x4}$ is the camera projection matrix, which maps a 3D world coordinate (X,Y,Z) into an image coordinate (u,v) in homogeneous form.

Projection Matrix Decomposition

The projection matrix can be decomposed into intrinsic and extrinsic parameters:

$P_{3x4}=K_{3x3}\cdot [R_{3x3}|t_{3x1}](2)$

Where:

1. $K_{3x3}$ is the intrinsic matrix, containing focal length and principal point parameters.
2. $R_{3x3}$ is a rotation matrix describing camera orientation.
3. $t_{3x1}$ is a translation vector describing camera position.
4. Together, $[R_{3x3}|t_{3x1}]$ expresses the camera pose relative to the world coordinate frame.

This decomposition is essential because Birds Eye View requires an accurate mapping between ground-plane points and image pixels.

Mapping Points on a Plane

Any point lying on the ground plane can be parameterized in homogeneous form as:

$[X,Y,Z,1{]}^t=\left[\begin{array}{c}x\cdot i+y\cdot j+d\\ 1\end{array}\right]=Q_{4x3}\cdot [x,y,1{]}^t(3)$

Where:

1. $(x,y)$ represents a point in ground-plane coordinates.
2. $Q$ is a 4x3 matrix that embeds the plane equation into homogeneous coordinates.

This formulation constrains all 3D points to lie on a planar surface—an assumption required by standard Inverse Perspective Mapping (IPM).

Deriving the Homography

Substituting (3) into the camera equation (1), we obtain:

$[u,v,1{]}^t\simeq P_{3x4}\cdot Q_{4x3}\cdot [x,y,1{]}^t=H_{3x3}\cdot [x,y,1{]}^t(4)$

This shows that the mapping between a point on the ground plane and its image projection is governed by a 3×3 homography matrix H.

Homography is the mathematical foundation that enables Birds Eye View systems to warp perspective images into a top-down representation.

Removing fisheye distortion

Wide-angle cameras are needed to obtain enough overlap between the images and generate the Birds Eye View perspective. Often, wide-angle cameras use fisheye lens, which produce images that need to be rectified to remove the distortion. Consider Fig. 3, this is a typical image captured using a camera with fisheye lens.

Fig 3: Image captured using a camera with fisheye lens

The image presented in Fig. 3 must be rectified before the Birds Eye View perspective transformation. Fig. 4 presents the rectified image, obtained after applying the methods presented in the remainder of this section.

Fig 4: Image obtained after applying undistortion rectification to an image captured with a fisheye camera

In order to remove the fisheye distortion, the camera must go through a calibration process. This section assumes that the output of the calibration process has been obtained and includes the camera matrix K and the distortion coefficients D:

• K: Output 3x3 floating-point camera matrix

$K=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$

• D: Output vector of distortion coefficients

$D=[k_1,k_2,k_3,k_4]$

The step by step process described in detail below can be summarized as follows:

1. Start with a 3D point.
2. Transform it into the camera coordinate frame.
3. Project it to normalized pinhole coordinates.
4. Apply fisheye distortion.
5. Map the point to pixel coordinates.

Let x, y and z be the coordinates of a 3D point P. The pinhole projection coordinates vector of P is [a; b] where

$a=\frac{x}{z},b=\frac{y}{z},r^2=a^2+b^2,\theta =atan(r)$

Then, the fisheye distortion is defined by:

${\theta }_d=\theta (1+k_1{\theta }^2+k_2{\theta }^4+k_3{\theta }^6+k_4{\theta }^8)$

Therefore, the distorted point coordinates are [x'; y'] where

$x^{\prime }=({\theta }_d/r)a,y^{\prime }=({\theta }_d/r)b$

Performance Considerations

The process described so far is based on state-of-the-art papers and research on how to apply IPM and other techniques to generate a top view image. The minimal steps always include the perspective mapping to get the IPM result, enlargement to adjust the measures between different cameras, and cropping to use only the required part of the image. These steps work great for a single image in an isolated environment to analyze and generate a single output frame. However, when the process needs to be repeated once for each frame at higher frame rates the system stress might be too high, and therefore the frame rate might be lower than expected.

To enhance the performance of the library we designed a simple method to wrap up all of the steps in a single warp perspective method. Usually, in Computer Vision, two images of the same planar surface in space can be related by a homography matrix. The method has many practical applications like image rectification, image registration, or computation of camera motion rotation, resize, and translation between two images. We will rely on the last application to speed up the process.

$H=\left[\begin{array}{ccc}{h}_{00}& h_{01}& h_{02}\\ h_{10}& h_{11}& h_{12}\\ h_{20}& h_{21}& h_{22}\end{array}\right]$

From an input image we need to map each $x_1,y_1$ source point to another $x_2,y_2$ destination point. The library uses a chessboard to collect inner corners and automatically adjust them by generating a perfect squared relationship and generate the output image as seen in the following figure.

The relationship between the points is defined by:

$\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]=H\cdot \left[\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_{00}& h_{01}& h_{02}\\ h_{10}& h_{11}& h_{12}\\ h_{20}& h_{21}& h_{22}\end{array}\right]\left[\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right]$

The main objective is to focus on how to adjust the homography matrix to absorb all the previous steps. With several geometric transformation the library also allows the user to fine-tune the final image. Matrix multiplication allows the quick and easy modification of the final homography, therefore the operation will be defined by $T_1$ (Move image to origin), $S_1$ (Scale image), $T_2$ (Move image back to original position), $M_1$ (Move image to fine adjust position), $A_1$ (Rotate image). The final homography ($H_f$) will be defined by:

$H_f=H\cdot T_1\cdot S_1\cdot T_2\cdot M_1\cdot A_1$

where $T_1$, $S_1$, $T_2$ , $M_1$, $A_1$ are:

$T_1=\left[\begin{array}{ccc}0& 1& -centerX\\ 0& 1& -centerY\\ 0& 0& 1\\ \end{array}\right]S_1=\left[\begin{array}{ccc}scaleX& 0& 0\\ 0& scaleY& 0\\ 0& 0& 1\\ \end{array}\right]T_2=\left[\begin{array}{ccc}0& 1& centerX\\ 0& 1& centerY\\ 0& 0& 1\\ \end{array}\right]M_1=\left[\begin{array}{ccc}1& 0& moveX\\ 0& 1& moveY\\ 0& 0& 1\\ \end{array}\right]A_1=\left[\begin{array}{ccc}cos(angle)& -sin(angle)& 0\\ sin(angle)& -cos(angle)& 0\\ 0& 0& 1\\ \end{array}\right]$

The input parameters for each matrix are:

• CenterX: Calculated X center between the inner corners of the chessboard
• CenterY: Calculated Y center between the inner corners of the chessboard
• ScaleX, ScaleY: User defined scale factor for the output image
• MoveX, MoveY: User defined amount of image movement adjustment
• Angle: User defined rotation angle

FAQ

Why is camera calibration required for BEV?

Camera calibration is required because the system needs accurate intrinsic parameters, distortion coefficients, and camera pose information to map image pixels correctly. Without calibration, the birds eye projection will be misaligned and geometric measurements will be unreliable.

What is inverse perspective mapping (IPM)?

Inverse perspective mapping is the geometric transformation that converts a perspective camera view into a top-down ground-plane view. It is effective when the scene near the vehicle can be approximated as planar.

What is a homography in a BEV system?

A homography is a 3×3 transformation matrix that relates points on a plane to their corresponding image coordinates. In a BEV pipeline, the homography is the mathematical foundation used to warp the camera image into the birds eye view.

Why do fisheye cameras need rectification?

Fisheye cameras provide a wide field of view, which helps capture enough overlap between cameras, but they also introduce strong lens distortion. Rectification removes that distortion before or during the BEV transformation so the projected ground plane is more accurate.

What assumptions does a standard BEV transformation make?

A standard IPM-based BEV pipeline assumes the camera pose is fixed, the ground is approximately planar, and the mapped surface does not contain large height variations. When these assumptions are violated, objects can appear stretched, compressed, or displaced in the top-down view.

Why do distant or tall objects look distorted in birds eye view?

Objects that rise above the ground plane, such as vehicles, pedestrians, curbs, or poles, do not satisfy the planar-scene assumption used by standard IPM. As a result, they can appear stretched or skewed after projection into the bird's-eye frame.