RidgeRun Developer Wiki

Birds Eye View/Calibration Guide: Difference between revisions

Page Discussion

Birds Eye View/Calibration Guide (view source)

Revision as of 03:47, 16 January 2025

2,289 bytes added ,  16 January
→‎Working Area
Visual
Line 160: Line 160:


=== Working Area ===
=== Working Area ===
[[File:BEV Calibration Tool Working Area.svg|alt=BEV Calibration Tool Working Area. Includes indicators for title bar, main view window, results view window, main toolbar, zoom bar and position bar.|center|thumb|900x900px|BEV Calibration Tool Working Area]]
# '''Title Bar:''' Indicates the current view and the active picture name.
# '''Title Bar:''' Indicates the current view and the active picture name.
# '''Main View:''' Main view is where you will edit your pictures and compose the BEV.
# '''Main View:''' Main view is where you will edit your pictures and compose the BEV.
Line 168: Line 169:


==== Main Toolbar ====
==== Main Toolbar ====
[[File:BEV Calibration Tool MainToolbar.svg|alt=BEV Calibration Tool Main Toolbar for cameras and BEV views|center|thumb|600x600px|BEV Calibration Tool Main Toolbar]]
# '''Delete Tool:''' Deletes the square vertices points of the active image for Cameras View and removes the ROI in the active image for BEV View.
# '''Delete Tool:''' Deletes the square vertices points of the active image for Cameras View and removes the ROI in the active image for BEV View.
# '''Polygon Tool:'''  
# '''Polygon Tool:'''  
Line 176: Line 178:


==== Zoom Bar ====
==== Zoom Bar ====
[[File:BEV Calibration Tool Zoom Bar.svg|alt=BEV Calibration Tool Zoom Bar. Includes zoom controls, fit to screen and full screen buttons|center|thumb|500x500px|BEV Calibration Tool Zoom Bar]]
# '''Zoom Controls:''' Zoom in and zoom out in any view. You can also write the zoom percentage manually.
# '''Zoom Controls:''' Zoom in and zoom out in any view. You can also write the zoom percentage manually.
# '''Fit to Screen:''' Automatically adjust the view content to the view window visible area.
# '''Fit to Screen:''' Automatically adjust the view content to the view window visible area.
Line 181: Line 184:


==== Position Bar ====
==== Position Bar ====
[[File:BEV Calibration Tool Position Bar.svg|alt=BEV Calibration Tool Position Bar with indicators of the position relative to window and position relative to image|center|thumb|500x500px|BEV Calibration Tool Position Bar]]
# Pointer position relative to the picture top/left corner.
# Pointer position relative to the picture top/left corner.
# Pointer position relative to the view window.
# Pointer position relative to the view window.


=== Settings Panel ===
=== Settings Panel ===
[[File:BEV Calibration Tool Settings Panel.svg|alt=BEV Calibration Tool Settings Panel|thumb|BEV Calibration Tool Settings Panel]]
# '''Opacity:''' Sets the opacity of the images in the BEV Main View, 100 for full opacity and 0 for full transparency.
# '''Opacity:''' Sets the opacity of the images in the BEV Main View, 100 for full opacity and 0 for full transparency.
# '''Overlay:''' Define overlay image for the BEV Composition
# '''Overlay:''' Define overlay image for the BEV Composition
Line 195: Line 200:
# '''Grid Width:''' Number of divisions along the horizontal axis.
# '''Grid Width:''' Number of divisions along the horizontal axis.
# '''Grid Height:''' Number of divisions along the vertical axis.
# '''Grid Height:''' Number of divisions along the vertical axis.
# '''Output Width: ''' Bev output image width.
# '''Output Width: ''' BEV output image width.
# '''Output Height:''' Bev output image height.
# '''Output Height:''' BEV output image height.
:
:


== Performing the Calibration ==
== Performing the Calibration ==
Line 205: Line 216:
[[File:bev-tool-iterative-process.png|thumbnail|center|640px|alt=Iterative calibration process|Iterative calibration process]]
[[File:bev-tool-iterative-process.png|thumbnail|center|640px|alt=Iterative calibration process|Iterative calibration process]]


=== Inverse Perspective Transform ===


The purpose of this stage is to convert each individual image into the aerial view. You do so by outlining the square calibration pattern in each image.
=== Cameras Transform ===
The purpose of this stage is to convert each individual image into an aerial view. You do so by outlining the square calibration pattern in each image. You must be in the '''Cameras View'''.


<ol>
# Start by clicking the thumbnail of the image to which you desire to perform the transformation. The image will be displayed in the '''Main View'''.
<li>Start by clicking the thumbnail of the image to which you desire to perform the IPT. The image will be displayed in the working space.</li>
# Activate the '''Polygon Tool''' in the Main Toolbar.
<li>Select a corner of the pattern by clicking (and dragging) on it.
# Select the first corner of the pattern by clicking on it, this would correspond to the square top-left corner in the aerial view.
<ul>
#* Use the zoom-in to have a better view of the corner and perform a fine adjustment.
<li>Use the precision visualizer to click at a pixel-level precision.</li>
#* You may drag the point you just placed to adjust its location.
<li>You may drag the point you just placed to adjust its location.
# Then locate and click to place the following corners, you must follow a clockwise order: top-right, bottom-right and finally bottom-left.
</ul>
#* Use the vertex point and the pattern contour to aid you in the placement.
</li>
#* Remember you can use zoom for better precision and you can drag the points to adjust its location.
<li>Click and hold to start the placement of the second point.
# Once you are happy with your square selection click '''Apply''' in the Main Toolbar.
<ul>
#* The transformed image will be shown in the '''Result View.'''
<li>Use the vertex and the pattern contour to aid you in the placement.</li>
#* The thumbnail will show a green check indicating this image already has a transformation.
<li>Again, use the precision visualizer.</li>
# Adjust any point as needed and click Apply, as many times as you need, until you can see a perfect square in the results View.  
<li>You can, again, drag the new point to adjust its location.</li>
#* You can enable/adjust the grid in the Settings Menu to verify the square pattern in the Results View.
</ul>
</li>
<li>Repeat the last step for the 3rd and 4th corners.
<ul>
<li>The square will close automatically.</li>
<li>The warped image will be reflected immediately below.</li>
<li>The warped image will be reflected in the BEV thumbnail and workspace.</li>
</ul>
</li>
<li>Adjust any point as needed.</li>
<li>Click on a new location to start the process again</li>
<li>Adjust the zoom level as needed
<ul>
<li>Information discarded by the zoom will not be available in the BEV workspace.</li>
<li>Keeping too much information (zoom out) will make the image uncomfortable to manipulate in the BEV workspace.</li>
<li>Zoom can be adjusted at any time.</li>
</ul>
</li>
</ol>


NOTE: You can start the process again by clicking on the Delete button in the Main Toolbar to delete the current polygon.{{Message|type=info}}


{{ambox|type=content|text=Note that if the original images have lens distortion, you might see curvature in straight lines even after performing the IPT}}
# Adjust the scale and offset of the transformed image using the Toolbar Menu controls, '''Scale, X Offset and Y Offset.''' You must hit '''Apply''' to reflect the changes.
#* Information discarded by the scale  and offset may not be available in the BEV workspace (depending on the output resolution)
#* Keeping too much information (zoom out) will make the image uncomfortable to manipulate in the BEV workspace.
#* Scale and translation can be adjusted at any time.


NOTE: if the original images have lens distortion, you might see curvature in straight lines even after performing the transformation


[[File:bev-ipt.gif|thumbnail|center|640px|alt=Inverse Perspective Transform|Inverse Perspective Transform]]
== Build Birds Eye View ==
The purpose of this stage is to combine all of the perspective transformed images into a single BEV image. This process is performed in the BEV View.  A rectangle frame indicates the valid output image area. Ensure all the  images fit inside this frame, anything outside will be ignored.  


=== Build Birds Eye View ===
'''Tips:'''


The purpose of this stage is to combine all of the IPT images into a single BEV image. Note that no blending will be made. Use the '''Visibility''' and '''Reordering''' tools to help you find the best overlap.
* Use the '''Fit''' button in the zoom toolbar to adjust the frame to the visible area.
* Use the '''Opacity''' tool in the Settings Menu to fine-tune the overlap between the images


<ol>
'''Steps:'''
<li>Make sure you have performed the IPT for all the images.</li>
<li>Click on the Birds Eye View thumbnail.</li>
<li>Choose your desired output image size and set it on the control bar</li>
<li>Use the '''Reorder''' controls to place the image you wish to work on at the top of the image bar</li>
<li>Resize the image, rotate it and place it at the desired position.
<ul>
<li>You can use the '''Lock''' control to avoid further accidentally moving the image.</li>
<li>You can always come back and manipulate this image again.
</ul>
</li>
<li>Place the next image at the top of the image bar to bring it to the front of the BEV.</li>
<li>Adjust the position, rotation and size to find the best overlap between the two images.
<ul>
<li>Use the '''Visibility''' controls to quickly evaluate the quality of the overlap.</li>
<li>Use the '''Reorder''' controls to swap positions and evaluate the quality of the overlap.</li>
<li>If you are happy with the results you can lock the image in place.</li>
</ul>
</li>
<li>Repeat the process for all the images.
<ul>
<li>Don't hesitate revisiting previous images to adjust their configuration.</li>
<li>You can always go back to the individual image and adjust the zoom level.</li>
</ul>
</li>
</li>Sometimes selection can be tricky. If you experience problems, click on the empty canvas and retry selecting the image</li>
</ol>


# Click on View BEV in the Control Panel, to move the BEV View working space.
# Make sure you have performed the transformation for all the images, each transformed image should appear in the Main View.
# Choose your desired output image size and set it on the Settings Panel->Output Resolution. The size of the frame in the Main View will be adjusted accordingly.
# Select the Scale Tool from the Main Toolbar.
# Select the image you wish to work on by clicking on it in the Main View or by clicking on the Cameras Panel corresponding thumbnail, this image will be placed on top.
# Resize the image, rotate it and place it at the desired position.
#* You can use the '''Lock''' control in the Thumbnail Toolbar to avoid accidentally moving the image.
#* You can always come back and manipulate this image again.
# Bring the next image to the front of the BEV, adjust the position, rotation and size to find the best overlap between the two images.
#* Use the '''Opacity''' control to quickly evaluate the quality of the overlap.
#* Swap the front image and evaluate the quality of the overlap.
#* If you are happy with the results you can lock the image in place.
# Repeat step 6 for all the images.
#* Don't hesitate revisiting previous images to adjust their configuration.
#* You can always go back to the individual image in the Cameras View and adjust the transformation, scale and offset.
# Once you have matched all the images you can click '''Apply''' to see the blended result. Actually you don’t need to wait until you have all the images, you can hit apply any time you want to see a partial BEV result. The resultant BEV will be shown at the '''Result View.''' At this point the resulting image may not be easy to view because you may have many overlaps that will be solved when you select the ROIs.


{{ambox|type=content|text=Don't expect a perfect match! The different perspectives of the cameras will always cause disparity defects.}}
NOTE: Do not expect a perfect match, the different perspectives of the cameras always will cause some disparities.  


== Region of Interest Selection ==
The purpose of this stage is to select the portion of each individual image you wish to keep and use for the image blending. Generally, big overlaps produce unnatural effects, while small overlaps in strategic zones produce smoother transitions.


[[File:bev-bev.gif|thumbnail|center|640px|alt=Building the Birds Eye View|Building the Birds Eye View]]
# Make sure you have built the BEV image already.
# Click on the '''Polygon Tool''' button in the Main Toolbar.
# Place the image you wish to work on at the front.
# Click to place the first corner of the region of interest.
#* Use the zoom-in to have a better view of the corner and perform a fine adjustment.
#* You may drag the point you just placed to adjust its location.
# Repeat step 4 for as many points as needed.
# Once you have the polygon ready you can use Shift+Click inside the image to hide/show the region outside the ROI. Anything outside the ROI will be ignored in the blending.
# Bring the next image to the top.
# Create the ROI polygon for this new image.
#* Reorder the top image to swap positions and evaluate the quality of the ROIs overlap.
#* Focus on the overlap region between the images. Remember that there may be overlap between multiple images.
# Repeat the process for all the images.
#* Don't hesitate revisiting previous images to adjust their configuration.
#* ROIs can be tricky, you can always experiment with different areas and positions.
# Once you have matched all the images you can click '''Apply''' to see the blended result. Actually you don’t need to wait until you have all the images, you can hit apply any time you want to see a partial BEV result. The resultant BEV will be shown in the '''Result View.'''


=== Region of Interest Selection ===
# You can also add an overlay from the settings panel to hide the hole where the cameras are positioned.


The purpose of this stage is to select the portion of each individual image you wish to keep and use for the image blending. Generally, big overlaps produce unnatural effects, while small overlaps in strategic zones produce smoother transitions.
== Export the Calibration File ==
Once you have the BEV and ROIs selected you are finally ready to export the calibration file.


<ol>
# Make sure you have built the BEV image already and the ROIs are properly selected.
<li>Make sure you have built the BEV image already.</li>
# Click on the '''Export Calibration'''  in the Control Panel. This will download a JSON file calibration.json with your calibration ready to use with libpanorama.
<li>Click on the ROI selection button in the toolbar.</li>
# If you plan to use the libpanorama example application you need to modify the exported JSON. You need to change:
<li>Use the '''Reorder''' controls to place the image you wish to work on at the top of the image bar</li>
#* The '''location''' in each camera pipeline.  
<li>Click (and drag) the first corner of the region of interest.
<ul>
<li>Use the precision visualizer to help you hit the right location.</li>
<li>Once dropped, you can move the point to adjust its location.
</ul>
</li>
<li>Repeat the process for as many points as needed.</li>
<li>Close the polygon by dragging the last point on top of the first one. The outside of the polygon will color up to indicate that the polygon is closed.</li>
<li>Use the '''Reorder''' controls again to bring the next image to the top.
<li>Create the ROI polygon for this new image.
<ul>
<li>Use the '''Reorder''' controls to swap positions and evaluate the quality of the ROIs overlap.</li>
<li>Focus on the overlap region between the images. Remember that there may be overlap between multiple images!</li>
</ul>
</li>
<li>Repeat the process for all the images.
<ul>
<li>Don't hesitate revisiting previous images to adjust their configuration.</li>
</ul>
</li>
</ol>


pipeline: "filesrc '''location=samples/bev_6_cameras/cam_0.jpg''' ! jpegparse ! jpegdec ! queue ! videoconvert ! video/x-raw,format=RGBx ! imagefreeze ! appsink name=appsink drop=false sync=true qos=false max-buffers=1 max-lateness=-1"


{{ambox|type=content|text=ROIs can be tricky, experiment with different areas and positions.}}
* The '''location''', '''size''' and '''position''' of the overlay image


overlay: {


[[File:bev-roi.gif|thumbnail|center|640px|alt=Selecting the regions of interest|Selecting the regions of interest]]
   pipeline: "filesrc '''location=samples/bev_6_cameras/top_car.png''' ! pngparse ! pngdec ! videoconvert ! videoscale ! video/x-raw,format=RGBx,width=600,height=550 ! appsink name=appsink"


=== Export the Calibration File ===
   x_pos: '''240'''
 
Once you have the BEV and ROIs selected you are finally ready to export the calibration file.


<ol>
   y_pos: '''650'''
<li>Make sure you have built the BEV image already and the ROIs are properly selected.</li>
<li>Click on the JSON export button in the toolbar.</li>
<li>Use the '''Copy''' icons within the json to copy valid JSON content.</li>
<li>Paste the contents in a file of your choice. The name is not important.</li>
<li>The exported JSON defaults to the sample images. Make sure to modify it for your use case:
<ul>
<li>The '''location''' in each camera pipeline.</li>
<html>
<pre style="font-family:Consolas,Courier New,monospace;">
  pipeline: "filesrc <b>location=samples/bev_6_cameras/cam_0.jpg</b> ! jpegparse ! jpegdec ! queue ! videoconvert ! video/x-raw,format=RGBx ! imagefreeze ! appsink name=appsink drop=false sync=true qos=false max-buffers=1 max-lateness=-1"
</pre>
</html>
<li>The '''location''', '''size''' and '''position''' of the overlay image</li>
<html>
<pre style="font-family:Consolas,Courier New,monospace;">
  overlay: {
    pipeline: "filesrc <b>location=samples/bev_6_cameras/top_car.png</b> ! pngparse ! pngdec ! videoconvert ! videoscale ! video/x-raw,format=RGBx,width=600,height=550 ! appsink name=appsink"
    x_pos: <b>240</b>
    y_pos: <b>650</b>
  }
</pre>
</html>
</ul>
</li>
<li>You may remove the overlay portion of the JSON if you do not wish to render an image on top of the BEV.</li>
<li>You may modify the camera pipelines to use the actual cameras or videos</li>
</ol>


 }


{{ambox|type=content|text=Don't close the browser window just yet! You might need to iterate refine the BEV and there's no way to save your progress}}
* You may remove the overlay portion of the JSON if you do not wish to render an image on top of the BEV.
* You may modify the camera pipelines to use the actual cameras or videos


== Project File ==


{{ambox|type=content|text=Don't copy using CTRL+C, copy using the icons within the JSON.}}
* '''Save the Project:''' You can save your project to continue later. Click on Save Project in the Control Panel to download a file bev_project.bev with the current state of your calibration.
* '''Load the Project:''' When you want to reload the project, just launch the BEV calibration tool and click Open Project to open a file selection dialog and select your bev_project.bev. This will load the calibration state that you saved.


= Hot Keys =


[[File:bev-export.gif|thumbnail|center|640px|alt=Exporting the calibration file|Exporting the calibration file]]
# '''Ctrl + Scroll:''' Zoom in/Out
# '''Ctrl + Drag:''' Move whole view
# '''Scroll:''' Vertical Scroll
# '''Shift + scroll:''' Horizontal scroll
# '''Shift + click:''' In BEV View, show/hide zone outside of polygon. You must click in any place inside the image you want to hide the region.


=== Test in the Target Hardware ===
=== Test in the Target Hardware ===
Mmontero
901

edits

Retrieved from "https://developer.ridgerun.com/wiki/index.php/Special:MobileDiff/63857"