Birds Eye View - Real Time Birds Eye View Calibration Guide

Info

Check legacy documentation at Calibration Guide Legacy

The calibration tool allows you to generate a Birds Eye View (BEV) configuration file directly from your browser. This configuration (or calibration) can be then used with the Birds Eye View project.

LAUNCH THE CALIBRATION TOOL

Understanding the Calibration Process

The calibration process consists on generating a JSON file with the appropriate parameters to build a Birds Eye View scene from your camera setup. The following figure summarizes the process.

Diagram showing the Birds Eye View calibration process — Summary of the Birds Eye View calibration process

1. Prepare calibration images: The process starts by you preparing one image per camera, each containing a square pattern. The calibration images must be taken in the same physical configuration of the final setup. Refer to #Preparing the Calibration Images for more details.
2. Calibrate the scene: Use our online Birds Eye View Calibration Tool to calibrate the camera configuration. This can be done on any computer. It is not necessary to do this in the target hardware. Refer to #Using the Calibration Tool for more details.
3. Export the calibration file: The calibration tool exports a JSON file that you can dump into the target hardware.
4. Generate the Birds Eye View scene: In the target hardware, use the calibration file along with the BEV software to generate the aerial view from the cameras in real time.
5. Validate the resulting scene: If everything went okay, the result should be exactly the same as you calibrated in the calibration tool. It may be necessary to iterate the calibration process again to fix perspective artifacts.

Preparing the Calibration Images

Before any calibration takes place, you must prepare a set of calibration images. Consider the guidelines below in order to prepare these samples:

You must have a physical setup with your cameras in place. Take, for example, our testing structure:
Example of a test structure.
The resulting calibration will only work for this setup. If you move the orientation of a camera, you must recalibrate.
Make sure that the cameras have some overlap between them.
The cameras should cover the complete 360 view of the scene
Take a picture from each camera.
The orientation and rotation of the cameras do not matter, as it may be fixed during the calibration process
The picture must from a Rectilinear lens, Fisheye lenses must be rectified first. Check out our CUDA Undistort project if you are looking for solutions to convert from Fisheye to Rectilinear.
To get the best results, consider performing the lens distortion calibration for each camera individually
The picture must have a square pattern in it. Here are some guidelines regarding the pattern:
- It must be square, or contain a square in it.
- The bigger, the better.
- Chessboards, ArUco Markers, April Tags, are good square pattern examples.
- All the patters should be in the same surface, typically the floor.
- Must be placed in a flat surface and can't be curved, crumpled or have folding marks.
- The closer the pattern to the camera center, the better

Sample Calibration Images

The following set is a series of sample image you may use to play with the calibration tool. The square tiles may be used as the square pattern.

Camera 0
Camera 1
Camera 2
Camera 3
Camera 4
Camera 5

The resulting Birds Eye View is shown below. Each camera is is overlayed as a hint to aid you in the calibration process.

Resulting BEV with camera hints — Resulting BEV with camera position hints.

The Calibration Tool GUI

Once you have the calibration images available, use RidgeRun’s Calibration Tool to start the calibration process. This section guides you on the process on how to perform it.

LAUNCH THE CALIBRATION TOOL

Once you launch the tool, the Home Screen will appear, displaying the available calibration’s tools. For this wiki, we will focus on the BEV calibration tool. Simply click on it to proceed.

After selecting the BEV Calibration Tool, you will be greeted by its Home Screen, which includes:

A brief description of the BEV Calibration Tool.
A Start Calibrating button to get into the actual calibration tool.
A card linking to the full documentation for BEV and Libpanorama.

To continue, click Start Calibrating. This opens the Main Window of the BEV Calibration Tool, which is composed of four main sections:

Header: Tools header. Click on the RidgeRun's logo to return to the Home Screen.
Control Panel: Contains the primary controls for managing the calibration process, from loading the calibration images to exporting the calibration results.
Cameras Panel: Shows thumbnails of the loaded images and provides some image-specific controls that change based on the current calibration step.
Working Area: In this area, you will interact with the images and active calibration tools. Allows seamless switching between images without losing your progress.

Control Panel

The Control Panel provides essential tools for managing the calibration process. Below is a breakdown of its features:

Hide/Show: Toggle the visibility of this panel by clicking the hide/show button.
Save Project: Save your current project progress to continue later.
Open Project: Load existing project to resume work.
Add Source: Add the pictures required for calibration.
Export Calibration: Export the calibration results once the process is complete.
View Cameras: The tool has 2 different views. The first one is the Cameras View, click here to change to this view. Use it to change the perspective of the individual pictures.
View BEV: The second view is the BEV (Birds Eye View), click here to change to this view. Once you get the perspective of all the pictures, you can go to the BEV View to start generating your BEV image.
Settings: Open the Settings Panel (see Settings Panel section for more details)
Take a Tour: Launch an interactive tour to familiarize yourself with the GUI interface.
Help: Access RidgeRun’s BEV Calibrator documentation for additional guidance.
Contact Us: Reach out to RidgeRun for additional support.

Cameras Panel

The Cameras Panel allows you to manage the calibration images and provides a preview of all loaded images as thumbnails.

Add Images: Click this button to open a file selection dialog and select your calibration images. You can import multiple images simultaneously by holding Shift or Ctrl while selecting the files.

Thumbnail: Each thumbnail displays a preview of the picture, its name, and additional controls. You can click any of the thumbnails to set it as the active picture for editing.

Thumbnail Toolbar: Contains tools for managing the image. The available controls changes based on the current view. (See Thumbnail Toolbar section below for more details).

Thumbnail Title: Indicates the picture name. Clicking the title opens the Thumbnail Menu for additional controls (See Thumbnail Menu section below for more details).

Thumbnail Toolbar

The thumbnail toolbar contains controls specific to an image. It changes depending on the current view , check out below the toolbar appearance for Cameras View and BEV View.

Delete Picture: Delete the current image (available in both views).
Picture Reload: Undo any modification to the picture and start from scratch.
Picture Lock: Lock the picture and avoid any accidental editing. When enabled you can’t move, scale, rotate, or change the ROI.
Picture Filter: Use this button to show only this picture.
Picture Visibility: Show/Hide current picture.

Thumbnail Menu

The thumbnail drop down menu appears when you click on the thumbnail title, it contains controls to modify the image results. The controls change with the view, below are shown the appearance for Cameras View and BEV View.

Scale: Scale the transformed image results for this factor. Reflected until you click the Apply button in the Main Toolbar. (Cameras View)
X Offset: Normalized offset in the horizontal axis of the transformed image, can be positive or negative. This moves the result image left or right, changing the visible area of the transformed image, anything outside the visible area may not be available for BEV depending on the output resolution. Reflected until you click the Apply button in the Main Toolbar. (Cameras View)
Y Offset: Normalized offset in the vertical axis of the transformed image, can be positive or negative. This moves the resulting image up or down, changing the visible area of the transformed image; anything outside the visible area may not be available for BEV depending on the output resolution. Reflected until you click the Apply button in the Main Toolbar. (Cameras View)
Width: Adjust the width of the image in the Main View. Also reflects the current width when you modify it with the mouse. (BEV View)
Height: Adjust the height of the image in the Main View. Also reflects the current height when you modify it with the mouse. (BEV View)
Angle: Change the image rotation in degrees. The rotation is performed from the center of the image. Also reflects the current angle when you modify it with the mouse. (BEV View)
Camera Index: This is the index of the camera associated with the thumbnail. For example, if the cameras is located at /dev/video1, use 1 as the camera index.

Tip

Each thumbnail has an associated color, which makes identifying each picture in the BEV View easier. The highlighting and polygon color will match the color in the thumbnail of that image.

Working Area

Title Bar: Indicates the current view and the active picture name.
Main View: Main view is where you will edit your pictures and compose the BEV.
Results View: Whenever you apply the modifications to a picture, the result will be shown here.
Main Toolbar: Contains controls to edit your pictures, changes depending on the current view.
Zoom Bar: More controls to change the size and position of the pictures.
Position Bar: Shows the pointer position relative to the picture and relative to the view window.

Main Toolbar

Delete Tool: Deletes the square vertices points of the active image for Cameras View and removes the ROI in the active image for BEV View.
Polygon Tool:
1. Cameras View: Use the Polygon Tool to mark the position of the square pattern for each picture.
2. Bev View: Use the tool to draw a Region of interest (ROI) in each picture. Any areas outside of the ROI will be ignored. Shortcut Tip: You can hide/show the areas outside the ROI by pressing Shift + Click.
Scale Tool: Allows to scale and rotate the pictures.
Apply: Executes the image perspective transformation in the Cameras View and generates the BEV composition in the BEV View, the results will be shown in the Results View.

Zoom Bar

Zoom Controls: Zoom in and zoom out in any view. You can also write the zoom percentage manually.
Fit to Screen: Automatically adjust the view content to the view window visible area.
Fullscreen: Toggle to and out of fullscreen, when enabled the current view window will occupy the space of the full working area.

Position Bar

Pointer position relative to the picture top/left corner.
Pointer position relative to the view window.

Settings Panel

Opacity: Sets the opacity of the images in the BEV Main View, 100 for full opacity and 0 for full transparency.
Overlay: Define overlay image for the BEV Composition
Grid: Define a grid of vertical and horizontal lines overlaid over the Main and Results View.
Output Resolution: Define the output size of the BEV image. This changes the frame displayed in the BEV Main View.
Close Settings: Closes the Settings Panel.
Overlay Check: Show/Hide overlay image in the BEV Main View.
Overlay File: Shows current overlay image or “Overlay File” as a placeholder
Add Overlay: Selects the overlay image file, when hit it opens a file dialog window.
Grid Check: Enable/disable the grid.
Grid Columns: Define the number of columns of the grid. The amount of rows is going to be calculated automatically to make each grid cell a perfect square.
Output Width: BEV output image width.
Output Height: BEV output image height.

Performing the Calibration

The calibration is a highly iterative and time consuming process. In general, you can expect to follow this process:

Cameras Transform

The purpose of this stage is to convert each individual image into an aerial view. You do so by outlining the square calibration pattern in each image. You must be in the Cameras View.

Start by clicking the thumbnail of the image to which you desire to perform the transformation. The image will be displayed in the Main View.
Camera View, image loaded at Main View
Activate the Polygon Tool in the Main Toolbar.
Select the first corner of the pattern by clicking on it, this would correspond to the square top-left corner in the aerial view.
- Use the zoom-in to have a better view of the corner and perform a fine adjustment.
- You may drag the point you just placed to adjust its location.
Then locate and click to place the following corners, you must follow a clockwise order: top-right, bottom-right and finally bottom-left.
- Use the vertex point and the pattern contour to aid you in the placement.
- Remember you can use zoom for better precision and you can drag the points to adjust its location.
Once you are happy with your square selection click Apply in the Main Toolbar.
- The transformed image will be shown in the Result View.
- The thumbnail will show a green check indicating this image already has a transformation.
  At Cameras View, polygon selection on Main View and transformed image on Results View
Adjust any point as needed and click Apply, as many times as you need, until you can see a perfect square in the results View
- You can enable/adjust the grid in the Settings Menu to verify the square pattern in the Results View.

Note

You can start the process again by clicking on the Delete button in the Main Toolbar to delete the current polygon.

Adjust the scale and offset of the transformed image using the Toolbar Menu controls, Scale, X Offset and Y Offset. You must hit Apply to reflect the changes.
- Information discarded by the scale and offset may not be available in the BEV workspace (depending on the output resolution)
- Keeping too much information (zoom out) will make the image uncomfortable to manipulate in the BEV workspace.
- Scale and translation can be adjusted at any time.

Warning

If the original images have lens distortion, you might see curvature in straight lines even after performing the transformation

Build Birds Eye View

The purpose of this stage is to combine all of the perspective transformed images into a single BEV image. This process is performed in the BEV View. A rectangle frame indicates the valid output image area. Ensure all the images fit inside this frame, anything outside will be ignored.

Tip

Use the Fit button in the zoom toolbar to adjust the frame to the visible area.

Tip

Use the Opacity tool in the Settings Menu to fine-tune the overlap between the images

Click on View BEV in the Control Panel, to move the BEV View working space.
Make sure you have performed the transformation for all the images, each transformed image should appear in the Main View.
Choose your desired output image size and set it on the Settings Panel->Output Resolution. The size of the frame in the Main View will be adjusted accordingly.
Select the Scale Tool from the Main Toolbar.
Select the image you wish to work on by clicking on it in the Main View or by clicking on the Cameras Panel corresponding thumbnail, this image will be placed on top.
Resize the image, rotate it and place it at the desired position.
- You can use the Lock control in the Thumbnail Toolbar to avoid accidentally moving the image.
- You can always come back and manipulate this image again.
Bring the next image to the front of the BEV, adjust the position, rotation and size to find the best overlap between the two images.
- Use the Opacity control to quickly evaluate the quality of the overlap.
- Swap the front image and evaluate the quality of the overlap.
- If you are happy with the results you can lock the image in place.
Repeat step 6 for all the images.
- Don't hesitate revisiting previous images to adjust their configuration.
- You can always go back to the individual image in the Cameras View and adjust the transformation, scale and offset.
Once you have matched all the images you can click Apply to see the blended result. Actually you don’t need to wait until you have all the images, you can hit apply any time you want to see a partial BEV result. The resultant BEV will be shown at the Result View. At this point the resulting image may not be easy to view because you may have many overlaps that will be solved when you select the ROIs.

Warning

Do not expect a perfect match, the different perspectives of the cameras always will cause some disparities.

Region of Interest Selection

The purpose of this stage is to select the portion of each individual image you wish to keep and use for the image blending. Generally, big overlaps produce unnatural effects, while small overlaps in strategic zones produce smoother transitions.

Make sure you have built the BEV image already.
Click on the Polygon Tool button in the Main Toolbar.
Place the image you wish to work on at the front.
Click to place the first corner of the region of interest.
- Use the zoom-in to have a better view of the corner and perform a fine adjustment.
- You may drag the point you just placed to adjust its location.
Repeat step 4 for as many points as needed.
Once you have the polygon ready you can use Shift+Click inside the image to hide/show the region outside the ROI. Anything outside the ROI will be ignored in the blending.
Bring the next image to the top.
Create the ROI polygon for this new image.
- Reorder the top image to swap positions and evaluate the quality of the ROIs overlap.
- Focus on the overlap region between the images. Remember that there may be overlap between multiple images.
Repeat the process for all the images.
- Don't hesitate revisiting previous images to adjust their configuration.
- ROIs can be tricky, you can always experiment with different areas and positions.
Once you have matched all the images you can click Apply to see the blended result. Actually you don’t need to wait until you have all the images, you can hit apply any time you want to see a partial BEV result. The resultant BEV will be shown in the Result View.
BEV Calibration Tool ROI Selection
You can also add an overlay from the settings panel to hide the hole where the cameras are positioned.

Export the Calibration File

Once you have the BEV and ROIs selected you are finally ready to export the calibration file.

Note

Before exporting the calibration, make sure the Camera Index for each of the cameras is correct

Make sure you have built the BEV image already and the ROIs are properly selected.
Click on the Export Calibration in the Control Panel. This will download a JSON file calibration.json with your calibration ready to use with libpanorama.

If you plan to use the libpanorama example application you need to modify the exported JSON. You need to change:

The location in each camera pipeline.

  pipeline: "filesrc location=samples/bev_6_cameras/cam_0.jpg ! jpegparse ! jpegdec ! queue ! videoconvert ! video/x-raw,format=RGBx ! imagefreeze ! appsink name=appsink drop=false sync=true qos=false max-buffers=1 max-lateness=-1"

The location, size and position of the overlay image.

  overlay: {
    pipeline: "filesrc location=samples/bev_6_cameras/top_car.png ! pngparse ! pngdec ! videoconvert ! videoscale ! video/x-raw,format=RGBx,width=600,height=550 ! appsink name=appsink"
    x_pos: 240
    y_pos: 650
  }

You may remove the overlay portion of the JSON if you do not wish to render an image on top of the BEV.
You may modify the camera pipelines to use the actual cameras or videos

Project File

Save the Project: You can save your project to continue later. Click on Save Project in the Control Panel to download a file bev_project.bev with the current state of your calibration.
Load the Project: When you want to reload the project, just launch the BEV calibration tool and click Open Project to open a file selection dialog and select your bev_project.bev. This will load the calibration state that you saved.

Test in the Target Hardware

You are finally ready to test the calibration result in the target hardware.

Make sure you have exported a valid calibration file.
The easiest way to test is by using the birds_eye_view example provided in the project.
At the root of the project, execute the following command:

  builddir/examples/birds_eye_view -p /tmp/calibration.json

This assumes the build directory was set to builddir.
This assumes the calibration file was saved to /tmp/calibration.json.
If you followed the build instructions, it should work out of the box

Hot Keys

Ctrl + Scroll: Zoom in/Out
Ctrl + Drag: Move whole view
Scroll: Vertical Scroll
Shift + scroll: Horizontal scroll
Shift + click: In BEV View, show/hide zone outside of polygon. You must click in any place inside the image you want to hide the region.

Home

❯