Holoscan Sensor Bridge Sensor Support

Holoscan Platform
Holoscan Sensor Bridge
Hardware Connection Bring Up Process Running Demo Performance Applications Adding New Sensors
Holoscan SDK
Introduction Installation and Testing Adding New Operators - CUDA
Contact Us

This article describes the steps to implement support for a new sensor in the NVIDIA Holoscan Sensor Bridge (Hololink). It assumes familiarity with Python and the existing Holoscan/hololink sensor APIs.

Overview

Hololink provides a framework for controlling sensors connected to Hololink devices. To support a new sensor, one typically:

• Creates a sensor object/class that implements required methods (e.g. configuration, start, stop, converter setup).
• Integrates that class into the sensor bridge repository so applications can import and use it.

Hardware considerations

The camera port of the Lattice Holoscan Sensor Bridge provides most of the connections required for modern cameras, like the IMX274. However, please, take into consideration the following:

1. The Lattice Holoscan Sensor Bridge does not provide the AVDD voltage (2.8V). It makes it incompatible with the OV5693 (from the TX devkits) cameras that require this voltage.
2. The Lattice Holoscan Sensor Bridge can support multiple cameras. Nevertheless, for the best performance, it is recommended to stream each camera through a dedicated port. You can find an example of a driver of this style here: IMX274 driver and application.
3. The Lattice Holoscan Sensor Bridge can multiplex multiple cameras through a single network interface. This is the case of the stereo application.

Required Sensor Object Interface

According to the Holoscan Sensor Bridge docs, a sensor class should typically provide the following methods:

• A configure method which uses the control plane to set up the data produced by the sensor. This method is called by the application code to configure the resolutions, framerate, exposure, and others before starting the capture.
• A start method that the data receiver operator calls in its startup, which configures the sensor to begin producing data. This usually means setting the start stream bit to one in many sensors, like IMX477 and IMX219.
• A stop method, called when the data receiver is shut down, that stops the sensor data flow. This usually means setting the start stream bit to zero in many sensors, like IMX477 and IMX219.
• A configure_converter method, which allows the sensor object to configure the next element in the application pipeline. This method is called by the application layer when the pipeline is being set up.
• A get/set_register pair of methods, which allows the implementation of the sensor register R/W for configuration. This is usually utilised by start/stop and configure.

The following is the skeleton of a sensor driver in Hololink:

import hololink

class MyCamera:
    CAMERA_I2C_BUS_ADDRESS = 0x34

    def __init__(self, hololink_channel, hololink_i2c_controller_address):
        self._hololink = hololink_channel.hololink()
        self._i2c = self._hololink.get_i2c(hololink_i2c_controller_address)

    def get_register(self, register):
        # Compose write buffer (big‐endian 2‐byte address)
        write_buffer = bytearray(10)
        serializer = hololink.abstract_os.Serializer(write_buffer)
        serializer.append_u16_be(register)
        read_byte_count = 4
        reply = self._i2c.i2c_transaction(
            self.CAMERA_I2C_BUS_ADDRESS,
            serializer.data(),
            read_byte_count
        )
        deserializer = hololink.abstract_os.Deserializer(reply)
        r = deserializer.next_u32_be()
        return r

    def configure(self, mode, …):
        # Write sensor registers to set mode, format, etc.
        self.set_register(…)
        # etc.

    def start(self):
        # Enable streaming
        self.set_register(…)
    
    def stop(self):
        # Disable streaming
        self.set_register(…)
    
    def configure_converter(self, converter):
        # Inform converter about frame size, metadata overhead etc.
        converter.configure(width * height, …)

You can find examples in the official repository. Here, we would like to point out the IMX477 driver because of its simplicity.

The sensor driver can be saved in the python/hololink/sensors and it requires registration in the __init__.py file of the same directory.

Overall workflow

A high‐level workflow when integrating a new sensor might be:

Sensor Driver:

1. Create the sensor class in hololink/sensors/my_sensor.py.
2. Implement register access and necessary control (initialization, power, clocks).
3. Implement the required interface: configure, start, stop, configure_converter.

Example program to instantiate the sensor:

1. Discover the Hololink channel.
2. Create the sensor with correct I2C/SPI controller parameters.
3. Call sensor.configure(...) for the desired mode.
4. Call sensor.configure_converter(...).
5. Start receiver operators, which under the hood will call sensor.start().
6. Stop at shutdown (sensor.stop()).