Birds Eye View

The calibration tool allows you to generate a Birds Eye View (BEV) configuration file directly from your browser. This configuration (or calibration) can be then used with the Birds Eye View project.

LAUNCH THE CALIBRATION TOOL

Understanding the Calibration Process

The calibration process consists on generating a JSON file with the appropriate parameters to build a Birds Eye View scene from your camera setup. The following figure summarizes the process.

Diagram showing the Birds Eye View calibration process — Summary of the Birds Eye View calibration process

1. Prepare calibration images: The process starts by you preparing one image per camera, each containing a square pattern. The calibration images must be taken in the same physical configuration of the final setup. Refer to #Preparing the Calibration Images for more details.
2. Calibrate the scene: Use our online Birds Eye View Calibration Tool to calibrate the camera configuration. This can be done on any computer. It is not necessary to do this in the target hardware. Refer to #Using the Calibration Tool for more details.
3. Export the calibration file: The calibration tool exports a JSON file that you can dump into the target hardware.
4. Generate the Birds Eye View scene: In the target hardware, use the calibration file along with the BEV software to generate the aerial view from the cameras in real time.
5. Validate the resulting scene: If everything went okay, the result should be exactly the same as you calibrated in the calibration tool. It may be necessary to iterate the calibration process again to fix perspective artifacts.

Preparing the Calibration Images

Before any calibration takes place, you must prepare a set of calibration images. Consider the guidelines below in order to prepare these samples:

You must have a physical setup with your cameras in place. Take, for example, our testing structure:
Example of a test structure.
The resulting calibration will only work for this setup. If you move the orientation of a camera, you must recalibrate.
Make sure that the cameras have some overlap between them.
The cameras should cover the complete 360 view of the scene
Take a picture from each camera.
The orientation and rotation of the cameras do not matter, as it may be fixed during the calibration process
The picture must from a Rectilinear lens, Fisheye lenses must be rectified first. Check out our CUDA Undistort project if you are looking for solutions to convert from Fisheye to Rectilinear.
To get the best results, consider performing the lens distortion calibration for each camera individually
The picture must have a square pattern in it. Here are some guidelines regarding the pattern:
- It must be square, or contain a square in it.
- The bigger, the better.
- Chessboards, ArUco Markers, April Tags, are good square pattern examples.
- All the patters should be in the same surface, typically the floor.
- Must be placed in a flat surface and can't be curved, crumpled or have folding marks.
- The closer the pattern to the camera center, the better

Sample Calibration Images

The following set is a series of sample image you may use to play with the calibration tool. The square tiles may be used as the square pattern.

Camera 0
Camera 1
Camera 2
Camera 3
Camera 4
Camera 5

The resulting Birds Eye View is shown below. Each camera is is overlayed as a hint to aid you in the calibration process.

Resulting BEV with camera hints — Resulting BEV with camera position hints.

The Calibration Tool GUI

Once you have the calibration images available, head up to https://bev.ridgerun.com to start the calibration process. This section guides you on the process on how to perform it.

LAUNCH THE CALIBRATION TOOL

Once you launch the tool, you're met with the Main Window. This window is composed by three sections, which you can resize at will by dragging the vertical separators.

Image Bar: Here you'll find the thumbnails of the different image you have loaded into the system, as well as a thumbnail of the Birds Eye View canvas. You may click through the different thumbnails to change the currently active image. The active image will be highlighted by a subtle shadow beneath it.
Working Area: In this area you will interact with the images and the different calibration tools currently active. You may switch between images and your current progress won't be lost.
Control Area: Here you'll find instructions and helper accompanying components for the currently active tool. In is contextual, meaning that it will change according to the currently active tool.

The next sections delve into more details about these sections.

Image Bar

The image bar allows you to import new images, select the currently active one, reorder them and control certain behavior.

1. Birds Eye View Thumbnail: Clicking this thumbnail will activate the Birds Eye View working space. In there, you can combine individual images into a single, composite view.
2. Add Images: As the name implies, clicking this button will open a file selection dialog that will allow you to import your calibration images. You can import several images at once by holding shift or ctrl and selecting multiple files.
3. Image Thumbnail: Clicking this thumbnail will activate the Inverse Perspective Transform working space. In there, you will work with the square pattern to transform the image into an aerial view.
4. Reordering Controls: Click on the up or down arrows to bring an image to the front or back in the Birds Eye View working space.
5. Solo: Clicking this icon will immediately hide all the other images. You will need to manually enable the visibility of each one of them by clicking the Visibility control.
6. Lock: Clicking this icon will lock the image in place in the Birds Eye View working space so you don't accidentally move it. Clicking it again will toggle this behavior.
7. Visibility: Clicking this icon will hide the image. Clicking it again will toggle this behavior.
8. Delete: Remove an image from the project. All its configuration will be lost and cannot be undone.

Working Area and Control Bar

The working area is contextual. This means that its content will vary according to the current Image Bar selection. Currently, there are two working spaces:

Birds Eye View: activated when you click on the Birds Eye View thumbnail.
Inverse Perspective Transform: activated when you click on any of the individual image thumbnails. Each image has its own Inverse Perspective Transform view.

This view will be activated when you click on the Birds Eye View thumbnail in the Image Bar. It holds all the individual images for you to move around, scale and rotate, select their regions of interest, and finally export the calibration file. At the top of the working space, you'll see a small toolbar that allows you to select the currently active tool, as described below.

Calibration tool working space — Calibration tool Birds Eye View working space

1. Canvas: The canvas is where you'll see all the images combined together and, potentially, interact with them. The behavior of the canvas changes according to the selected tool. The gray area is what is going to be visible in the final Birds Eye View scene. Everything outside of it will be cropped. The z-ordering of the images in the canvas (which image is on top and in the back) is controlled by the order of the images in the Image Bar. The topmost image in the bar will be the topmost image on the canvas. You can hide images momentarily by using the Solo and Visibility controls.
2. Free Image Control: When this tool is active, you can click on images, drag them around, scale them and rotate them. You may lock the image using the Lock control to inhibit this behaviour.
3. Region of Interest Selection: When this tool is active, you can no longer interact with the images. Instead, you are prompted to define a region of interest (ROI) for the active image. The ROI is defined by placing a series of points that form a polygon. The active image is the topmost image, and each image must have it's own ROI.
4. Calibration Export: With this control, you can view, instead of the canvas, the generated JSON with the calibration parameters. You can copy it and use it with the Birds Eye View project.
5. Control Bar: In the bar, you will see extra information about the currently active tool. By switching between 'Free Image Control', 'Region of Interest Selection', or 'Calibration Export', the information here will change. Additionally, there are some helper controls that will help you in the application of the tool.

Inverse Perspective Transform

This view will be activated when you click on an individual image thumbnail. Each image will have its own view. As its name suggests, this working space allows you to perform the Inverse Perspective Transform (IPT) to each image. The transformed image will be immediately reflected in the Birds Eye View workspace.

1. Canvas: The canvas is where you'll identify the square pattern. The pattern is identified by placing markers in the four corners of the pattern. The image will be automatically warped once 4 points are placed. Points can be adjusted afterwards.
2. Transformed Image: In this section you'll se the transformed image. It will be automatically updated when the pattern is identified.
3. Zoom Control: This control allows you to zoom in or out the transformed image. Zooming in will lose the information outside of the visible area, but it will be easier to handle in the Birds Eye View workspace.
4. Precision Visualization: A helper tool that allows you to select, with finer detail, the calibration pattern corner.

Performing the Calibration

The calibration is a highly iterative and time consuming process. In general, you can expect to follow this process:

Inverse Perspective Transform

The purpose of this stage is to convert each individual image into the aerial view. You do so by outlining the square calibration pattern in each image.

Start by clicking the thumbnail of the image to which you desire to perform the IPT. The image will be displayed in the working space.
Select a corner of the pattern by clicking (and dragging) on it.
- Use the precision visualizer to click at a pixel-level precision.
- You may drag the point you just placed to adjust its location.
Click and hold to start the placement of the second point.
- Use the vertex and the pattern contour to aid you in the placement.
- Again, use the precision visualizer.
- You can, again, drag the new point to adjust its location.
Repeat the last step for the 3rd and 4th corners.
- The square will close automatically.
- The warped image will be reflected immediately below.
- The warped image will be reflected in the BEV thumbnail and workspace.
Adjust any point as needed.
Click on a new location to start the process again
Adjust the zoom level as needed
- Information discarded by the zoom will not be available in the BEV workspace.
- Keeping too much information (zoom out) will make the image uncomfortable to manipulate in the BEV workspace.
- Zoom can be adjusted at any time.

Build Birds Eye View

The purpose of this stage is to combine all of the IPT images into a single BEV image. Note that no blending will be made. Use the Visibility and Reordering tools to help you find the best overlap.

Make sure you have performed the IPT for all the images.
Click on the Birds Eye View thumbnail.
Choose your desired output image size and set it on the control bar
Use the Reorder controls to place the image you wish to work on at the top of the image bar
Resize the image, rotate it and place it at the desired position.
- You can use the Lock control to avoid further accidentally moving the image.
- You can always come back and manipulate this image again.
Place the next image at the top of the image bar to bring it to the front of the BEV.
Adjust the position, rotation and size to find the best overlap between the two images.
- Use the Visibility controls to quickly evaluate the quality of the overlap.
- Use the Reorder controls to swap positions and evaluate the quality of the overlap.
- If you are happy with the results you can lock the image in place.
Repeat the process for all the images.
- Don't hesitate revisiting previous images to adjust their configuration.
- You can always go back to the individual image and adjust the zoom level.

Region of Interest Selection

The purpose of this stage is to select the portion of each individual image you wish to keep and use for the image blending. Generally, big overlaps produce unnatural effects, while small overlaps in strategic zones produce smoother transitions.

Make sure you have built the BEV image already.
Click on the ROI selection button in the toolbar.
Use the Reorder controls to place the image you wish to work on at the top of the image bar
Click (and drag) the first corner of the region of interest.
- Use the precision visualizer to help you hit the right location.
- Once dropped, you can move the point to adjust its location.
Repeat the process for as many points as needed.
Close the polygon by dragging the last point on top of the first one. The outside of the polygon will color up to indicate that the polygon is closed.
Use the Reorder controls again to bring the next image to the top.
Create the ROI polygon for this new image.
- Use the Reorder controls to swap positions and evaluate the quality of the ROIs overlap.
- Focus on the overlap region between the images. Remember that there may be overlap between multiple images!
Repeat the process for all the images.
- Don't hesitate revisiting previous images to adjust their configuration.

Export the Calibration File

Once you have the BEV and ROIs selected you are finally ready to export the calibration file.

Make sure you have built the BEV image already and the ROIs are properly selected.
Click on the JSON export button in the toolbar.
Use the Copy icons within the json to copy valid JSON content.
Paste the contents in a file of your choice. The name is not important.

The exported JSON defaults to the sample images. Make sure to modify it for your use case:

The location in each camera pipeline.

  pipeline: "filesrc location=samples/bev_6_cameras/cam_0.jpg ! jpegparse ! jpegdec ! queue ! videoconvert ! video/x-raw,format=RGBx ! imagefreeze ! appsink name=appsink drop=false sync=true qos=false max-buffers=1 max-lateness=-1"

The location, size and position of the overlay image

  overlay: {
    pipeline: "filesrc location=samples/bev_6_cameras/top_car.png ! pngparse ! pngdec ! videoconvert ! videoscale ! video/x-raw,format=RGBx,width=600,height=550 ! appsink name=appsink"
    x_pos: 240
    y_pos: 650
  }

You may remove the overlay portion of the JSON if you do not wish to render an image on top of the BEV.
You may modify the camera pipelines to use the actual cameras or videos

Test in the Target Hardware

You are finally ready to test the calibration result in the target hardware.

Make sure you have exported a valid calibration file.
The easiest way to test is by using the birds_eye_view example provided in the project.
At the root of the project, execute the following command:

  builddir/examples/birds_eye_view -p /tmp/calibration.json

This assumes the build directory was set to builddir.
This assumes the calibration file was saved to /tmp/calibration.json.
If you followed the build instructions, it should work out of the box

Home

❯