Full Body Pose Estimation for Sports Analysis - Camera Calibration
 | WORK IN PROGRESS. Please Contact RidgeRun OR email to support@ridgerun.com if you have any questions. |
This wiki summarizes the documentation of the camera calibration library. It contains information such as system design, scalability, and usage.
Introduction
Camera calibration is a key task in computer vision applications. It provides a way to relate a camera frame with what it's being actually captured in the real world. A camera is considered calibrated when its calibration parameters are known. Therefore, the camera calibration process is a procedure that allows the determination of the camera's parameters.
Camera calibration parameters are separated into three groups. First, the extrinsic calibration describes the position and orientation of the camera in relation to the world's coordinate system. They are defined by a rotation and a translation, which are commonly expressed as a rotation matrix R and a translation vector t. Then, the distortion coefficients account for the camera lens distortion. And finally, the intrinsic parameters model the the behavior of the internal geometry and optical characteristics of the camera, such as focal length (fx, fy), principal point offset (x0, y0) and axis skew (s). They are unique to the camera, and they are represented by the intrinsic calibration matrix K.
The camera calibration library consists of a python package that provides an interface for the use of different camera calibration methods. It was developed along with the Full Body Pose Estimation System, however, it is a standalone module as shown in the next image.
Library Features
The library provides two approaches for computing camera calibration parameters. Both are based on classic computer vision techniques and require the use of a calibration object.
If you don't know what a calibration object is, or why is it required by classic calibration methods to calibrate your camera, don't worry, it is a relatively simple concept. By using an object with known geometry and dimensions, it is possible to establish a relation between known points in the object (object points) and their corresponding points in the camera frame (image points). Such an object would be referred to as the calibration object since it would be used to perform the calibration process. Consequently, it is not only required to be able to identify the object on the image but also to match the object points with their corresponding image points.
Chessboard calibration
The OpenCV library provides a series of functions that facilitate the process required to compute the camera intrinsic and extrinsic calibration parameters.
As already mentioned, this approach is based on a classic method that requires the use of a calibration object. In this case, the said object corresponds to a chessboard with known dimensions. Ideally, the chessboard calibration object must be built on sturdy material and with the highest tolerances at hand to improve the calibration quality. Also, it is required for the chessboard to be asymmetrical, otherwise, the resulting calibration parameters will be wrong.
You can read more about this calibration method in the Chessboard Calibrator Wiki.
AprilTag calibration
When calibrating a multi-camera system, it is required for every camera to capture the same calibration object in order to have enough data to compute the transformation between the cameras coordinate systems. Sometimes, depending on the camera angles, it could be difficult to capture a chessboard like a calibration object.
AprilTag, is a visual fiducial system popular in robotics research. The library provides the capability of identifying predefined tags on a given image. This makes easier the construction of a three-dimensional calibration object since each of the object faces can be associated with a different tag.
Same as with other calibration methods, its accuracy depends on the quality of the calibration object. However, if the application is not critical, a simple cardboard box with the tags printed on the sides is enough.
You can read more about this calibration method in the AprilTag Calibrator Wiki.
How to use the calibration library
The camera calibration library offers an interface to its camera calibrators by implementing the facade design pattern. This structural pattern hides the system complexities from the library user. To obtain further information on this design pattern, you can check the information on this tutorial.
To perform intrinsic and extrinsic camera calibration, it is necessary to get the required calibrator objects instantiated. As mentioned in the #Library Features section, there are currently two different calibration methods available. Hence, you can choose between the chessboard and the apriltag calibrators. However, if you want to check which calibrator options you have available in your system, you can do it as follows.
import camera_calibration.calibrationfacade
available_calibrators = calibratorfacade.get_available_calibrators()
print('Available calibrators: %s' % available_calibrators)
Now, it is not required for you to use the same calibrator for both calibrations. For instance, in the following code snippet, we create a chessboard intrinsic calibrator and an apriltag extrinsic calibrator.
import camera_calibration.calibrationfacade
available_calibrators = calibratorfacade.get_available_calibrators()
print('Available calibrators: %s' % available_calibrators)
intrinsic_calibrator = calibrationfacade.get_intrinsic_calibrator('chessboard')
extrinsic_calibrator = calibrationfacade.get_extrinsic_calibrator('apriltag')
Finally, all that is left is to perform the actual calibration. Both, the intrinsic and the extrinsic calibrators provide a calibrate() function that takes a list of calibration pictures in RGB format. However, the extrinsic calibrator also requires the intrinsic calibration matrix and the distortion coefficients of the camera. Take as an example the following piece of code, which corresponds to the complete sample code we have been working on in this section.
import camera_calibration.calibrationfacade
available_calibrators = calibratorfacade.get_available_calibrators()
print('Available calibrators: %s' % available_calibrators)
intrinsic_calibrator = calibrationfacade.get_intrinsic_calibrator('chessboard')
extrinsic_calibrator = calibrationfacade.get_extrinsic_calibrator('apriltag')
# Calibration images with chessboard like calibration object
chessboard_images = []
# Calibration images with apriltag calibration object
apriltag_images = []
# Perform intrinsic calibration
# K = intrinsic calibration matrix
# dist = distortion coefficients
# err = re-projection error
K, dist, err = intrinsic_calibrator.calibrate(chessboard_images)
#Perform extrinsic calibration
# r = rotation vector
# t = translation vector
# err = re-projection error
r, t, err = extrinsic_calibrator.calibrate(apriltag_images, K, dist)
System design
Library base architecture
The camera calibration library was modeled based on the object-oriented programming paradigm. Its two main objects are the intrinsic and extrinsic calibrators. Therefore, the whole calibration system was designed around these two classes.
The design is based on the abstract factory design pattern. This creational pattern provides interfaces for both the concrete calibrators and their factories. Factories allow the creation of new calibrator instances and since this pattern is based on the concept of a super factory or factory of factories, concrete factories can be instantiated using a factory creator, which then can be used to instantiate new concrete calibrators. For more information on this pattern, check this tutorial.
In this case, the factory creator class corresponds to the calibration library facade, as shown in the next image. Please note that the diagram shows the classes colored according to their abstraction level. For instance, the library facade corresponds to the higher level of abstraction you can use to interact with the library and is colored green, interfaces and base classes are painted blue and orange respectively, and the concrete classes are color red.
Library modules
The camera calibration library is composed of several python modules, as shown in the next image. The colors of the diagram represent the level of abstraction, the more abstraction. For instance, the calibrationfacade offers the simplest interface to interact with the library and is colored green. Whereas concrete modules like chessboardcalibrator is represented by the color red.
The orange classes that are part of two of the modules in the Camera calibration library module diagram are base classes. Which are different from the blue colored classes that correspond to interfaces. On one hand, the base classes provide methods or properties to be extended by other classes, for instance, the BaseIntrinsicCalibrator and BaseExtrinsicCalibrator classes offer OpenCV based methods to compute calibration parameters from sets of object and image points. On the other hand, the Calibrator and CalibratorFactory interfaces enforce a determined structure on concrete calibrator and factory classes that implement them.
Red colored modules provide specific functionalities that are unique to them. For example, the chessboardcalibrator offers the ability to compute camera calibration parameters from images with a chessboard like calibration object. Similarly, the apriltagcalibrator provides the possibility to calibrate a camera using a three-dimensional calibration object build with tags. Both of these concrete modules extend the base calibrator classes since they use the same mathematical functions provided by OpenCV to compute the parameters. At the same time, base calibrators implement the calibrator interface. Therefore, concrete calibrators can be treated the same, no matter what special calibration characteristic they have, by using polymorphism.
Classes, attributes and methods
If you would like to know in-depth the library implementation, please check its full class diagram in the next image. The diagram follows a color scheme based on the level of abstraction. The CameraCalibrationFacade, which offers the simplest method of interaction with the library is shown in green color. Then all interface classes, such as CameraCalibratorFactory and Calibrator are shown in color blue. Similarly, all base classes are identified by an orange color. And finally, all concrete classes, such as ChessboardIntrinsicCalibrator are colored red.
Also, note that some classes have the same name, such as Factory or Calibration, this is possible because they belong to separate modules. In fact, the use of the same name for the Factory class at the calibrator module level is important, because it is the name that the calibrationfacade module looks for, to instantiate new calibrators.
Scalability
The library was designed around the idea that new calibration methods could be included. That is the reason why the creation of new camera calibrator classes must follow the library defined interfaces. The user must not only implement the custom calibrator class, but also its concrete factory.
How to add a new calibrator to the library
Step 1. Create a new python file inside the camera_calibration directory
As an example, let's say our new calibrator module will be called newcalibrator.py. In that case, you can proceed to open the python file with your editor of choice.
cd rrpose/camera_calibration/ editor newcalibrator.py
Step 2. Add the calibrator classes to the calibrator module
There are two possible calibrators to implement. The intrinsic camera calibrator and the extrinsic camera calibrator. These are used to compute the intrinsic and extrinsic camera parameters respectively. Each calibrator module might include either one or both calibrator classes. Now, said classes must inherit from the IntrinsicCalibrator or ExtrinsicCalibrator base classes provided in calibratorinterface.py, depending on what is required.
Of course, it is also possible for the calibrator classes to inherit from other already implemented concrete calibrators, as they are already based on the base calibrator classes and might have other reusable code. For instance, the AprilTag calibrator extends this calibrator.
If your new calibrator requires some sort of configuration parameters to be provided when being initialized, then you can just specify them as arguments in the constructor. However, we heavily encourage the use of a calibration class, which can hold all your calibration parameters in a much cleaner way. For instance, in the following code example, you will find how to create your own intrinsic and extrinsic calibrator classes with their configuration class.
from .calibratorinterface import BaseConfigurationClass
from .calibratorinterface import Calibrator
class Configuration(BaseConfigurationClass):
'''New-calibrator configuration class.
Attributes:
obj_points (None): Some required configuration
parameter. Defaults to None.
'''
def __init__(self):
self.configuration_parameter = None
class NewIntrinsicCalibrator(Calibrator):
"""New concrete intrinsic calibrator.
This calibrator...
Args:
configuration (Configuration): Configuration object.
Attributes:
configuration (Configuration): Configuration object.
"""
def __init__(self, configuration):
self.configuration = configuration
def calibrate(self, images):
"""Compute intrinsic calibration parameters.
This function computes the intrinsic calibration matrix
and distortion coefficients of a camera. It also
computes the re-projection error resulting from the use
of the calculated calibration parameters.
It is important to maintain the function return values.
This way the calibrator can be compatible with other
software that already uses the library.
Args:
images (list): Image list captured with the same camera.
Returns:
mtx (np.array): Camera's intrinsic calibration matrix.
dist (np.array): Camera's distortion coefficients.
err (float): Re-projection error.
"""
mtx = np.zeros(1)
dist = np.zeros(1)
err = float('inf')
return [mtx, dist, err]
class NewExtrinsicCalibrator(Calibrator):
"""New concrete extrinsic calibrator.
This calibrator...
Args:
configuration (Configuration): Configuration object.
Attributes:
configuration (Configuration): Configuration object.
"""
def __init__(self, configuration):
self.configuration = configuration
def calibrate(self, images):
"""Compute extrinsic calibration parameters.
This function computes the rotation and translation
of the camera, relative to the world coordinate system.
It also computes the re-projection error resulting from
the use of the calculated calibration parameters.
It is important to maintain the function return values.
This way the calibrator can be compatible with other
software that already uses the library.
Args:
images (list): Image list captured with the same camera.
Returns:
rotation_vector (list): Estimated rotation vector.
translation_vector (list): Estimated translation vector.
"""
rvec = np.zeros(1)
tvec = np.zeros(1)
err = float('inf')
return [rvec, tvec, err]
Step 3. Add the Factory class to the calibrator module
This class must implement the CalibratorFactory interface. And, it is very important to keep the name Factory for this class in every calibrator module so that it can be managed by the calibration library facade.
from .calibratorinterface import BaseConfigurationClass
from .calibratorinterface import Calibrator
from .calibratorinterface import CalibratorFactory
class Configuration(BaseConfigurationClass):
'''New-calibrator configuration class.
Attributes:
obj_points (None): Some required configuration
parameter. Defaults to None.
'''
def __init__(self):
self.configuration_parameter = None
class NewIntrinsicCalibrator(Calibrator):
"""New concrete intrinsic calibrator.
This calibrator...
Args:
configuration (Configuration): Configuration object.
Attributes:
configuration (Configuration): Configuration object.
"""
def __init__(self, configuration):
self.configuration = configuration
def calibrate(self, images):
"""Compute intrinsic calibration parameters.
This function computes the intrinsic calibration matrix
and distortion coefficients of a camera. It also
computes the re-projection error resulting from the use
of the calculated calibration parameters.
It is important to maintain the function return values.
This way the calibrator can be compatible with other
software that already uses the library.
Args:
images (list): Image list captured with the same camera.
Returns:
mtx (np.array): Camera's intrinsic calibration matrix.
dist (np.array): Camera's distortion coefficients.
err (float): Re-projection error.
"""
mtx = np.zeros(1)
dist = np.zeros(1)
err = float('inf')
return [mtx, dist, err]
class NewExtrinsicCalibrator(Calibrator):
"""New concrete extrinsic calibrator.
This calibrator...
Args:
configuration (Configuration): Configuration object.
Attributes:
configuration (Configuration): Configuration object.
"""
def __init__(self, configuration):
self.configuration = configuration
def calibrate(self, images):
"""Compute extrinsic calibration parameters.
This function computes the rotation and translation
of the camera, relative to the world coordinate system.
It also computes the re-projection error resulting from
the use of the calculated calibration parameters.
It is important to maintain the function return values.
This way the calibrator can be compatible with other
software that already uses the library.
Args:
images (list): Image list captured with the same camera.
Returns:
rotation_vector (list): Estimated rotation vector.
translation_vector (list): Estimated translation vector.
"""
rvec = np.zeros(1)
tvec = np.zeros(1)
err = float('inf')
return [rvec, tvec, err]
class Factory(CalibratorFactory):
"""Calibrator concrete factory for new_calibrator.
This factory allows the creation of intrinsic and extrinsic
calibrator instances.
"""
def get_intrinsic_calibrator(self):
"""Create instance of NewIntrinsicCalibrator.
Args:
configuration_object (Configuration): Configuration class instance
with all newcalibrator configuration parameters for the
camera calibration process.
Returns:
NewIntrinsicCalibrator instance.
"""
if configuration_object is not None:
return NewIntrinsicCalibrator(configuration_object)
return NewIntrinsicCalibrator(Configuration())
def get_extrinsic_calibrator(self, configuration_object=None):
"""Create instance of NewExtrinsicCalibrator.
Args:
configuration_object (Configuration): Configuration class instance
with all newcalibrator configuration parameters for the
camera calibration process.
Returns:
NewExtrinsicCalibrator instance.
"""
if configuration_object is not None:
return NewExtrinsicCalibrator(configuration_object)
return NewExtrinsicCalibrator(Configuration())
Step 4. Add the new calibrator module to the library facade
To add the new calibrator module to the library facade, is as simple as adding a new item to the CALIBRATOR_MODULES dictionary. Make sure to follow the <module_name>: <alias> item structure, where the module_name corresponds to the actual name of the python file, and the alias corresponds to the simpler name the user will use to request the calibrator. For example, if the new calibrator module name is newcalibrator.py but you want the user to call it ncalib, then add newcalibrator: ncalib to the CALIBRATOR_MODULES dictionary.
First, open calibrationfacade.py for editing with your editor of choice:
editor calibrationfacade.py
Then, add newcalibrator to the facade dictionary.
CALIBRATOR_MODULES = {'chessboardcalibrator': 'chessboard',
'apriltagcalibrator': 'apriltag',
'newcalibrator': 'ncalib'}
.
.
.