View source for Birds Eye View/Getting Started/Startup Guide

<noinclude>
{{Birds Eye View/Head|next=Getting_Started/Minimal_Application|previous=Getting_Started|keywords=|title=Bird's Eye View Startup Guide}}
</noinclude>

<seo 
title="Birds Eye View - Startup Guide | RidgeRun" titlemode="replace" 
metakeywords="bev, birds-eye-view, birds eye view, cuda, stiching, stitcher, image processing, computer vision, jetson, nvidia" description="This wiki will show you a summary, prerequisites, and calibration script for starting RidgeRun's Birds Eye View software.">
</seo>


__TOC__

== Prerequisites ==

* Four fisheye cameras ready to use
* Cameras mounted in a car, boat, device, table, etc, in final distribution that will be used
* Completion of [[Birds_Eye_View/Getting_the_Code/Building_and_Installation_Guide | Installation Guide]]

== Summary ==

#'''Cameras layout setup:''' create and put the cameras in correct position for desired application like a car, boat or any other device.
#'''Cameras distortion parameters calibration:''' get the K/D distortion parameters for each camera to correctly remove fisheye from image.
#'''Top view image calibration:''' using a chessboard calibration pattern the birds eye view calibration process is fast, with easy fine-tuning capabilities.
#'''Run application:''' if the cameras layout does not change, previous steps are only made once, after that, you will be able to run your applications with same configuration.

[[File:CalibrationProcessSummary.png|900px|frameless|center]]

== Auto Calibration Script ==

The RidgeRun's Bird's Eye View library provides a calibration script that will help in finding camera distortion parameters. Navigate to the following directory:

<syntaxhighlight lang="bash">
cd birds-eye-view/prototype
</syntaxhighlight>

To check the python script allowed parameters the following command can be used:

<syntaxhighlight lang="bash">
python3 autoCalibration.py -h
</syntaxhighlight>

The options are:

<center>
'''Table 1. Calibration script options'''
<html>
<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-u1yq{background-color:#c0c0c0;font-weight:bold;text-align:center;vertical-align:top}
.tg .tg-0lax{text-align:left;vertical-align:top}
</style>
<table class="tg">
<thead>
  <tr>
    <th class="tg-u1yq">Parameter</th>
    <th class="tg-u1yq">Short Parameter</th>
    <th class="tg-u1yq">Description</th>
    <th class="tg-u1yq">Arguments</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td class="tg-0lax">--help</td>
    <td class="tg-0lax">-h</td>
    <td class="tg-0lax">Shows the help message and exit</td>
    <td class="tg-0lax">-</td>
  </tr>
  <tr>
    <td class="tg-0lax">--load</td>
    <td class="tg-0lax">-l</td>
    <td class="tg-0lax">Load current settings file and show preview</td>
    <td class="tg-0lax">-</td>
  </tr>
  <tr>
    <td class="tg-0lax">--debug</td>
    <td class="tg-0lax">-d</td>
    <td class="tg-0lax">Show debug FPS information for preview</td>
    <td class="tg-0lax">-</td>
  </tr>
  <tr>
    <td class="tg-0lax">--remove</td>
    <td class="tg-0lax">-r</td>
    <td class="tg-0lax">Allow to remove calibration images from filesystem</td>
    <td class="tg-0lax">-</td>
  </tr>
  <tr>
    <td class="tg-0lax">--skip</td>
    <td class="tg-0lax">-s</td>
    <td class="tg-0lax">Skip saving fisheye images and calibrate with current<br>images that are on the filesystem</td>
    <td class="tg-0lax">-</td>
  </tr>
  <tr>
    <td class="tg-0lax">--imagesPath</td>
    <td class="tg-0lax">-p</td>
    <td class="tg-0lax">Path to the directory where the calibration images<br>will be saved</td>
    <td class="tg-0lax">-</td>
  </tr>
  <tr>
    <td class="tg-0lax">--json</td>
    <td class="tg-0lax">-j</td>
    <td class="tg-0lax">Output json file to save the calibration values to</td>
    <td class="tg-0lax">Path with file name to save calibration<br>settings in JSON format.</td>
  </tr>
</tbody>
</table>
</html>
</center>

== A. Cameras Layout Setup ==

Put the cameras in the final setup where they will be used, for example, a Car, Boat, other devices. In this wiki an indoor setup with a small table as camera holder will be used.
Label each camera and make sure to know which Linux device is bound for each camera.

<center>
'''Table 2. Cameras device mapping'''
<html>
<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-u1yq{background-color:#c0c0c0;font-weight:bold;text-align:center;vertical-align:top}
.tg .tg-0lax{text-align:left;vertical-align:top}
</style>
<table class="tg">
<thead>
  <tr>
    <th class="tg-u1yq">Camera</th>
    <th class="tg-u1yq">Device</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td class="tg-0lax">Front</td>
    <td class="tg-0lax">/dev/video2</td>
  </tr>
  <tr>
    <td class="tg-0lax">Left</td>
    <td class="tg-0lax">/dev/video4</td>
  </tr>
  <tr>
    <td class="tg-0lax">Right</td>
    <td class="tg-0lax">/dev/video6</td>
  </tr>
  <tr>
    <td class="tg-0lax">Back</td>
    <td class="tg-0lax">/dev/video8</td>
  </tr>
</tbody>
</table>
</html>
</center>

Add the information to the autoCalibrationSettings file:

<syntaxhighlight lang="bash">
vim birds-eye-view/prototype/autoCalibrationSettings.json
</syntaxhighlight>

Add the information of the resolution size for the cameras, input framerate, and output size of frames

<syntaxhighlight lang="bash">
...
    "camerasWidth": 1920,
    "camerasHeight": 1080,
    "framerate": 30,
    "outputWidth": 600,
    "outputHeight": 300,
...
</syntaxhighlight>

There is a node inside the "views" node in the JSON file for each camera where you will need to add the device ID value. For example:

<syntaxhighlight lang="bash">
...
"views": [
        {
            "tag": "FrontCamera",
            "deviceID": 2,
...
</syntaxhighlight>

[[File:Bev start 1.png|500px|frameless|center]]


== B. Cameras Distortion Parameters Calibration ==

In this step, the user has two options:

==='''OPTION 1: Camera parameters are already available'''===

The floating-point camera matrix, which is defined by:

<math>
\begin{bmatrix}
f_x & 0 & c_x\\
0 & f_y & c_y\\
0 & 0 & 1
\end{bmatrix}
</math>

And the vector of distortion coefficients defined by: 

<math>
\begin{bmatrix}
k_1 & k_2 & k_3 & k_4\\
\end{bmatrix}
</math>

The information for each camera can be added to same node in the autoCalibrationSettings.json file, for example:

<syntaxhighlight lang="bash">
...
"views": [
        {
            "tag": "FrontCamera",
            "deviceID": 2,
            "balance": 0,
            "camK": [[533,0.0,954],[1.0,530,541],[0.0,0.0,1.0]],
            "camD": [[-0.07808330315826352],[0.06710634719219645],[-0.048871815691283065],[0.011097077754507887]],
...
</syntaxhighlight>

Continue with next step [[Birds_Eye_View/Getting_Started/Startup_Guide#C._Top_View_Image_Calibration | 	C. Top View Image Calibration]]

==='''OPTION 2: Camera parameters need to be automatically obtained'''===

If the camera parameters are not available, the auto-calibration script can get several calibration images and compute the resulting parameters. A chessboard pattern is needed, several are available through the internet for example [https://markhedleyjones.com/projects/calibration-checkerboard-collection here] the user can find several chessboard patterns for different printing sizes. Once a chessboard pattern has been selected, add the '''inner''' corner width and height number to the autoCalibrationSettings.json

<syntaxhighlight lang="bash">
...
    "chessboardInnerCornersWidth": 7,
    "chessboardInnerCornersHeight": 4,
...
</syntaxhighlight>


Run the script as follows:

<syntaxhighlight lang="bash">
python3 autoCalibration.py --fisheye
</syntaxhighlight>

The script will start capturing images when it detects a valid chessboard pattern every two seconds. It will save the images at: "birds-eye-view/prototype/imgs/calibration", within a directory with the device ID for each camera. 

After taking enough valid images, '''press 'q' on your keyboard''' to continue with the next camera.

'''Guidelines:'''
* Make sure the chessboard pattern inner corner number matches the configured in the autoCalibrationSettings.json
* Take around '''30-40''' pictures for each camera.
* Use correct valid images for calibration:

The script will show a green border over the image when a chessboard pattern is identified, however, that might not be a good calibration image. Please follow these example images, and try to get many different positions to get a better calibration process.
[[File:ExamplesCalibImages.png|900px|frameless|center]]


* Take some images perpendicular to the camera with the chessboard covering the camera field of view, for example:

[[File:CalibAnimation.gif|center|frame]]

==C. Top View Image Calibration==

The script will automatically continue to this step if the user is calibrating from the fisheye process. Otherwise, start the script with:

<syntaxhighlight lang="bash">
python3 autoCalibration.py
</syntaxhighlight>

NOTE: Use a high size and resolution chessboard, for example, for the previous fisheye step a calibration size of '''A4 paper''' was used, for this step '''A1 paper''' will be used. <br>

'''1. Place the chessboard perpendicular to the camera'''

[[File:ValidChessboardIPM.png|900px|frameless|center]]

'''2. If the chessboard is valid, press Enter'''

The script will draw a green border if the chessboard is correctly found. To continue to the next step press '''Enter''' key.

'''3. Review the preview and press Enter'''

A small preview window with the resulting transformation will be shown. Press '''Enter''' key to continue.

[[File:ValidChessboardIPM2.png|400px|frameless|center]]

'''4. Repeat the process for each camera'''

The script will automatically load the next camera according to autoCalibrationSettings.

'''5. Fine-tune the BEV image'''

Once all cameras have been configured, the settings can be fine-tuned with controls. For example, you can configure different system layouts if you want to use stitching or not.

[[File:StitchingReady.png|800px|frameless|center]]

Controls can be used to adjust the system as needed:

[[File:CalibIPMAnimation.gif|center|frame]]

Refer to the following table for controls:

<center>
'''Table 3. Fine-tuning Script Controls'''
<html>
<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  overflow:hidden;padding:10px 5px;word-break:normal;}
.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;
  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}
.tg .tg-kftd{background-color:#efefef;text-align:left;vertical-align:top}
.tg .tg-0lax{text-align:left;vertical-align:top}
</style>
<table class="tg">
<thead>
  <tr>
    <th class="tg-kftd">Key</th>
    <th class="tg-kftd">Control</th>
    <th class="tg-kftd">Description</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td class="tg-0lax">Right-click</td>
    <td class="tg-0lax">Edit speed</td>
    <td class="tg-0lax">Multiplies the factors by 3 to speed up<br>modification process</td>
  </tr>
  <tr>
    <td class="tg-0lax">H</td>
    <td class="tg-0lax">Homography</td>
    <td class="tg-0lax">Change to homograpy controls (blue controls)</td>
  </tr>
  <tr>
    <td class="tg-0lax">P</td>
    <td class="tg-0lax">Position</td>
    <td class="tg-0lax">Change to position controls (green controls)</td>
  </tr>
  <tr>
    <td class="tg-0lax">Left Arrow</td>
    <td class="tg-0lax">- FactorX</td>
    <td class="tg-0lax">Moves image homography points to the left </td>
  </tr>
  <tr>
    <td class="tg-0lax">Right <span style="font-weight:400;font-style:normal">Arrow</span></td>
    <td class="tg-0lax">+ FactorX</td>
    <td class="tg-0lax"><span style="font-weight:400;font-style:normal">Moves image homography points to the right</span></td>
  </tr>
  <tr>
    <td class="tg-0lax">Up <span style="font-weight:400;font-style:normal">Arrow</span></td>
    <td class="tg-0lax">+ FactorY</td>
    <td class="tg-0lax"><span style="font-weight:400;font-style:normal">Moves up image homography points</span></td>
  </tr>
  <tr>
    <td class="tg-0lax">Down <span style="font-weight:400;font-style:normal">Arrow</span></td>
    <td class="tg-0lax">- FactorY</td>
    <td class="tg-0lax"><span style="font-weight:400;font-style:normal">Moves down image homography points</span></td>
  </tr>
  <tr>
    <td class="tg-0lax">Q</td>
    <td class="tg-0lax">- Scale</td>
    <td class="tg-0lax">Zooms out the image</td>
  </tr>
  <tr>
    <td class="tg-0lax">W</td>
    <td class="tg-0lax">+ Scale</td>
    <td class="tg-0lax">Zooms in the image</td>
  </tr>
  <tr>
    <td class="tg-0lax">A</td>
    <td class="tg-0lax">- Angle</td>
    <td class="tg-0lax">Rotates the image counter clock-wise</td>
  </tr>
  <tr>
    <td class="tg-0lax">S</td>
    <td class="tg-0lax">+ Angles</td>
    <td class="tg-0lax">Rotates the image clock-wise</td>
  </tr>
  <tr>
    <td class="tg-0lax">Z</td>
    <td class="tg-0lax">- IPM</td>
    <td class="tg-0lax">Reduces IPM view angle</td>
  </tr>
  <tr>
    <td class="tg-0lax">X</td>
    <td class="tg-0lax">+ IPM</td>
    <td class="tg-0lax">Rises IPM view angle</td>
  </tr>
</tbody>
</table>
</html>
</center>


Once the image has been calibrated as desired, press '''"Enter"''' key to save the autoCalibrationSettings.json file. If later the calibration needs to be modified but without placing the chessboard for each camera and without fisheye calibration, launch the script as follows:

<syntaxhighlight lang="bash">
python3 autoCalibration.py --load
</syntaxhighlight>

==D. Run the application==

Modify the '''birds-eye-view/src/examples/bevFromCameras.cpp''' according to your system setup.


'''1. Select undistort depending on your compatible platform'''

*Desktop PC
<syntaxhighlight lang="cpp">
 auto camFront = factory->makeCamera(CAM_WIDTH,CAM_HEIGHT,CAM_FRAMERATE,"jpegparse",
                                        "jpegdec", "",
                                        "../prototype/autoCalibrationSettings.json",err);
</syntaxhighlight>

*NVIDIA Jetson Boards
In Jetson systems, the library uses a gst-cuda compatible plugin to perform fisheye undistortion.

<syntaxhighlight lang="cpp">
 auto camFront = factory->makeCamera(CAM_WIDTH,CAM_HEIGHT,CAM_FRAMERATE,"jpegparse",
                                        "jpegdec", "cudaundistort",
                                        "../prototype/autoCalibrationSettings.json",err);
</syntaxhighlight>

'''2. Select a valid device ID for each camera'''

<syntaxhighlight lang="cpp">
//Init front camera (Make sure device matches camera position)
err = camFront->load("2");
if(err.isError()) {
    std::cerr << err.getDescription() << std::endl;
    exit(1);
}
</syntaxhighlight>


'''3. Compile and run the bevFromCameras Example'''
<syntaxhighlight lang="cpp">
cd birds-eye-view
mkdir build && cd build
meson -Dopencv-cuda=enabled ..
ninja
</syntaxhighlight>

<syntaxhighlight lang="cpp">
./src/examples/bevFromCameras
</syntaxhighlight>

A window with a live feed from cameras with the autoCalibrationSettings will be shown

[[File:ResultBEV.png|400px|frameless|center]]

<noinclude>
{{Birds Eye View/Foot|Getting_Started|Getting_Started/Minimal_Application}}
</noinclude>