Full Body Pose Estimation for Sports Analysis - Camera Abstraction
 | WORK IN PROGRESS. Please Contact RidgeRun OR email to support@ridgerun.com if you have any questions. |
This wiki summarizes the documentation of the camera abstraction library. It contains information such as system design, scalability, and usage.
Introduction
The camera abstraction library consists of a python package that provides the ability to interact with different video sources as they were all the same. It exposes the cameras through an interface that makes them appear as simple cameras. It is part of the Full Body Pose Estimation for Sports Analysis project, however, it is independent of the pose estimation system, as shown in the following diagram.
Library Features
As mentioned in the #Introduction, this library is meant to abstract different video sources into the camera like objects. This allows for the interaction with different video sources as they were all the same, which makes the use of input video data highly modular.
The library offers two different types of cameras based on how is the data being acquired. The OpenCV Camera abstracts the use of cameras through the OpenCV library. Similarly, the GStreamer RTSP Dual Camera abstracts two incoming RTSP video streams as if they were cameras.
OpenCV Camera
This type of camera abstracts video input managed with the OpenCV library. Its initialization does not require any special argument. If you would like to learn how to create instances of this object using the library, please check the #How to use the abstraction library section. Similarly, if you require to know more in-depth information on the module's implementation, we encourage you to take a look at the library's full class diagram in the #Classes, attribute, and methods section.
GStreamer RTSP Dual Camera
This feature allows the abstraction of two GStreamer RTSP streams as if they were two different cameras, part of a multi-camera system. The abstraction provides the ability to interact with each one of the streams as an individual camera or with the dual-camera system as a whole. By abstracting this video source, other software that already uses the camera abstraction layer can interact with it using the same interface, which means no change in the software is needed.
To initialize a GstRTSPDualCam object, it is required to provide a few arguments. Since the video source corresponds to two GST RTSP streams, it is necessary to provide the ip_address, port, mapping0 and mapping1 values required to build the GStreamer pipeline. If you require more information on the implementation of this module, please check its class description shown in the library full class diagram found in section #Classes, attributes, and methods
How to use the abstraction library
The camera abstraction library offers an interface to its camera classes by implementing the facade design pattern. This structural pattern hides the system complexities from the library user. To obtain further information on this design pattern, you can check the information on this tutorial.
Then, to start using the camera abstraction library, you only need to import the library's facade. This module will take care of checking all the imports for concrete camera dependencies. If the library facade is not able to import any one of the camera modules, then it will let you know by showing a warning message. In order to know what camera modules are available in your system, you can check as follows.
import camera_abstraction_layer.abstractionlayerfacade as CameraFactory
available_cameras = CameraFactory.get_available_cameras()
print('Available cameras: %s' % available_cameras)
Now, let's say we want an instance of the gstrtspdualcam. In that case we would require to get a camera factory of the required type. Then we can request our camera by providing the get_camera function with all camera specific arguments. In this case, the camera_id, ip_address, port, mapping0 and mapping1 are specific to the gstrtspdualcam. Also, note that the arguments are supplied with their names, this is because each camera initialization function has its own specific arguments defined, however, the camera creation functions take them as **kwargs in order to simplify the code at a higher level of abstraction.
import camera_abstraction_layer.abstractionlayerfacade as CameraFactory
available_cameras = CameraFactory.get_available_cameras()
print('Available cameras: %s' % available_cameras)
camera_type = 'gstrtspdualcam'
camera_factory = CameraFactory.get_camera_factory(camera_type)
camera = camera_factory.get_camera(0, ip_address='10.42.0.206',
port='5000', mapping0='stream0',
mapping1='stream1')
Finally, you can check the camera availability and capture an image with a couple of function calls.
import camera_abstraction_layer.abstractionlayerfacade as CameraFactory
available_cameras = CameraFactory.get_available_cameras()
print('Available cameras: %s' % available_cameras)
camera_type = 'gstrtspdualcam'
camera_factory = CameraFactory.get_camera_factory(camera_type)
camera = camera_factory.get_camera(0, ip_address='10.42.0.206',
port='5000', mapping0='stream0',
mapping1='stream1')
if camera.available():
image = camera.capture()
System design
Library base architecture
The camera abstraction library was modeled using the object-oriented programming paradigm. The whole design revolves around its main object, which corresponds to the camera itself.
The design is based on the abstract factory design pattern. This creational pattern provides interfaces for both the concrete cameras and their factories. Interfaces enforce a determined structure on the classes that implement them, for instance, all concrete cameras have their own unique characteristics, however, they have the same structure enforced by the abstract camera. Factories allow the creation of new calibrator instances and since this pattern is based on the concept of a super factory or factory of factories, concrete factories can be instantiated using a factory creator, which then can be used to instantiate new concrete cameras. For more information on this pattern, check this tutorial.
In this case, the factory creator class corresponds to the abstraction library facade, as shown in the image on the right. The diagram shown in the right side image corresponds to the library base design class diagram. Please note that the diagram shows the classes colored according to their abstraction level, where the warmer the color the lower the level of abstraction. For instance, the library facade corresponds to the higher level of abstraction you can use to interact with the library and is colored green, whereas the concrete camera classes are color red.
Library modules
The camera abstraction package contains several python modules, which are shown in the camera abstraction library module diagram. Same as with the base design class diagram on the right, the module diagram uses a color scheme based on the abstraction level of the classes. Hence, the abstraction layer facade, which corresponds to the highest level of abstraction you can use to interact with the library is colored green. Similarly, interfaces such as CameraFactory have colored blue and base classes like Camera are colored orange. Finally, modules that contain concrete classes are painted red. These concrete classes offer unique characteristics that no other class shares. For instance, the gstrtspdualcam module offers the ability to interact with two GStreamer RTSP incoming streams as if they were cameras.
Classes, attributes and methods
For a more in-depth description of the library implementation, please check the camera abstraction library full class diagram. The diagram uses the same color scheme as the base design class diagram. At the top, in green color, you can see the library facade, which provides the simplest interface to interact with the library functionalities. Then, in blue and orange colors, the interface and base classes respectively. Finally, every concrete class is shown in red color.
Please note that some classes share the exact same name. Classes like Factory can be named the same as long as they are part of different modules. In fact, it is very important for every module with a concrete camera implementation to have a class named Factory. This will allows the module to be instantiated by the library facade, which abstracts the camera creation and module availability from the user. Therefore, the user would not have to manage library imports specific to each one of the concrete cameras. The library facade tries to import every module and if the user development system does not meet any of the dependencies, the library would let him know with a warning.
Concrete camera classes offer special functionalities that are unique to them. However, despite their implementation and characteristics, concrete camera classes can be used as they were all the same by using polymorphism. This is possible because every concrete camera extends the Camera base class, which not only provides reusable code across all concrete cameras but also ensures they have the same interaction interface.
Scalability
The library was designed around the idea that new cameras could be included. That is the reason why the creation of new camera classes must follow the library defined interfaces. The user must not only implement the custom camera class, but also its concrete factory.
How to add a new camera to the library
Step 1. Create a new python file inside the camera_abstraction_layer directory
As an example, let's say our new camera module will be called newcamera.py. In that case, you can proceed to open the python file with your editor of choice.
cd rrpose/camera_abstraction_layer/ editor newcamera.py
Step 2. Add the camera class to the python module
Every camera class must extend from the Camera class in camera base classes. This way all camera objects can be treated the same.
Of course, it is also possible for a camera class to inherit from other already implemented concrete cameras, as they are already based on the base camera class and might have other reusable code.
from .camerabaseclasses import Camera
class NewCamera(Camera):
"""New concrete camera class.
This concrete camera class provides the user with an interface to ...
"""
def camera_initialization(self, param):
"""Initialize camera.
Custom initialization for NewCamera...
Args:
param: Some parameter required by NewCamera.
"""
pass
def capture(self):
"""Capture a picture with the camera.
Returns:
The requested image as a numpy.ndarray.
"""
image = np.zeros(1)
return image
def release(self):
"""Release the camera.
"""
pass
def available(self):
"""Check if the camera is available.
Returns:
True if the NewCamera is available, False otherwise.
"""
available = False
return available
Step 3. Add the Factory class to the camera module
This class must inherit from the CameraFactory base class. And, it is very important to keep the name Factory for this class in every camera module so that it can be managed by the abstraction library facade.
from .camerabaseclasses import Camera
from .camerabaseclasses import CameraFactory
class NewCamera(Camera):
"""New concrete camera class.
This concrete camera class provides the user with an interface to ...
"""
def camera_initialization(self, param):
"""Initialize camera.
Custom initialization for NewCamera...
Args:
param: Some parameter required by NewCamera.
"""
pass
def capture(self):
"""Capture a picture with the camera.
Returns:
The requested image as a numpy.ndarray.
"""
image = np.zeros(1)
return image
def release(self):
"""Release the camera.
"""
pass
def available(self):
"""Check if the camera is available.
Returns:
True if the NewCamera is available, False otherwise.
"""
available = False
return available
class Factory(CameraFactory):
"""Concrete factory for NewCamera.
This factory allows the user to create new concrete NewCamera
instances.
"""
def init_camera(self, device_id, *args, **kwargs) -> NewCamera:
"""Initialize new NewCamera object.
Returns:
New NewCamera instance.
"""
return NewCamera(device_id, *args, **kwargs)
Step 4. Add the new camera module to the library facade
To add the new camera module to the library facade, is as simple as adding a new item to the CAMERA_MODULES dictionary. Make sure to follow the <module_name>: <alias> item structure, where the module_name corresponds to the actual name of the python file, and the alias corresponds to the simpler name the user will use to request the calibrator. For example, if the new camera module name is newcamera.py but you want the user to call it ncam, then add newcamera: ncam to the CAMERA_MODULES dictionary.
First, open abstractionlayerfacade.py for editing with your editor of choice:
editor abstractionlayerfacade.py
Then, add newcamera to the facade dictionary.
CAMERA_MODULES = {'opencvcamera': 'opencvcam',
'gstrtspdualcam': 'gstrtspdualcam',
'newcamera': 'ncam'}
.
.
.