Creating a service for loading camera drivers dynamically using NVIDIA Jetson IO Tool
Motivation
The idea of creating a service for loading camera drivers dynamically is that the user only has to worry about disconnecting and connecting the camera sensors to the board. The service will determine which sensor configuration is connected to the board and select the device tree binaries to load the camera drivers and make the capture work successfully.
Introduction
This is an implementation of a service that checks for the i2c bus muxes related to the camera sensor devices. With the buses detected, it looks for an i2c address to identify which sensor is connected and extracts the Device Tree Overlay name. Then, checks if there are overlays available detected by the Jetson IO tool, verifies if there is an overlay applied in the extlinux.config and if not, it applies the overlay related to the physical connection and reboots the board. If there is no overlay related to the physical connection it will restore the extlinux config to the default and any changes should be loaded after the reboot.
Jetson IO tool
The following wiki https://developer.ridgerun.com/wiki/index.php/NVIDIA_Jetson_-_Device_Tree_Overlay gives a detailed explanation about the Device Tree Overlays and their usage in combination with the device tree overlay.
The Jetson IO tool allows configuration for compatible hardware, that displays the compatible hardware screen, which lets the user select from a list of configurations for hardware modules that can be attached to the header. The user could check this as executing the following command:
sudo /opt/nvidia/jetson-io/config-by-hardware.py -l
The above commands could give an output similar to the following:
Header 1 [default]: Jetson 40pin Header Available hardware modules: 1. Adafruit SPH0645LM4H 2. Adafruit UDA1334A 3. FE-PI Audio V1 and Z V2 4. ReSpeaker 4 Mic Array 5. ReSpeaker 4 Mic Linear Array Header 2: Jetson Nano CSI Connector Available hardware modules: 1. Camera X Dual 2. Camera Y Dual 3. Camera X-A and Y-B Header 3: Jetson M.2 Key E Slot No hardware configurations found!
The header of interest for this section is Header 2: Jetson Nano CSI Connector.
Considerations before creating the dynamic driver load service
One consideration before creating the service is the user has to create the camera sensors device tree (dtsi) files. The user can take the imx477.dtsi and imx219.dtsi for the Jetson Xavier JetPack 5.1.4 (L4T 35.6.0) as references at the following paths:
hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-camera-rbpcv3-imx477.dtsi hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-camera-rbpcv2-imx219.dtsi
The overlay at the following path can be taken as a reference too:
hardware/nvidia/platform/t19x/jakku/kernel-dts/tegra194-p3668-all-p3509-0000-camera-imx477-imx219.dts
Also, the user will need to create a depending on the number of sensors that will be connected to the board and the possible physical connections that can be made:
For example:
1. Camera IMX477 Triple 2. Camera IMX219 Triple 3. Cameras IMX477-A and IMX219-B-C 4. Cameras IMX477-A-B and IMX219-C 5. Cameras IMX477-A-C and IMX219-B 6. Cameras IMX219-A and IMX477-B-C 7. Cameras IMX219-A-B and IMX477-C 8. Cameras IMX219-A-C and IMX477-B
Note: The IMX477 and IMX219 sensor names are taking only as a symbolic names for this documentation.
Creating the loading camera drivers service
Now that the Device Tree Overlay was built, we can create a service for loading the camera drivers dynamically using the NVIDIA Jetson IO tool. For this service, we need a way to look at how the sensors are physically connected to the Jetson board. So, we need a way to differentiate which sensor is connected. In this case, we used two different cameras, one of them has an EEPROM at 0x72 i2c address and the other does not. Based on this scenario, this service is created to use the i2c bus muxes related to the camera sensor devices, so the first aspect to implement is to check for those i2c buses and it can be done by taking the following code snippet as a guide:
NUMBER_OF_I2C_BUSES = 3
def look_for_i2c_buses():
mux_delimiter_string = 'mux'
buses_counter = 0
i2c_camera_buses_list = []
try:
# Check for all i2c buses
all_buses = subprocess.run(["i2cdetect", "-l"], stdout=PIPE, stderr=PIPE)
for bus in all_buses.stdout.splitlines():
bus = bus.decode('utf-8')
# Look for the i2c mux buses such for example:
# i2c-10 i2c i2c-2-mux (chan_id 0)
if mux_delimiter_string in bus:
# Slip the first list element
# until the hyphen to get the
# bus number
bus = int(bus.split()[0].rsplit('-',1)[1])
i2c_camera_buses_list.append(bus)
i2c_camera_buses_list = sorted(i2c_camera_buses_list)[:NUMBER_OF_I2C_BUSES]
print(f"The i2c buses found are {i2c_camera_buses_list}")
return i2c_camera_buses_list
except (subprocess.CalledProcessError, FileNotFoundError) as exception:
if isinstance(exception, subprocess.CalledProcessError):
print("subprocess.CalledProcessError occurred in look_for_i2c_buses function")
elif isinstance(exception, FileNotFoundError):
print("i2cdetect -l command not found")
The output of executing the above function should be similar to the following:
The i2c buses found are [10, 11, 12] [10, 11, 12]
After checking the i2c buses, the following two steps consist of checking if the EEPROM at 0x72 exists and building the Device Tree Overlay's name based on the physical connection.
Please, recall that the overlay must have the overlay-name property that specifies a name for the hardware module.
CAMERA_SENSORS_LIST = [' IMX477', ' IMX219']
def check_i2c_address(bus_number, address):
try:
# Indicates the bus /dev/i2c-<bus_number>
bus = SMBus(bus_number)
# Check if the i2c address exists
# if not an exception occurs
bus.read_byte_data(address,0)
print(f"bus {bus.read_byte_data(address,0)}")
bus.close()
return CAMERA_SENSORS_LIST[0]
except IOError:
print(f"traceback.format_exc() {traceback.format_exc()}")
# Check the traceback error Message
# to identify if the device is found but busy
if "busy" in traceback.format_exc():
bus.close()
return CAMERA_SENSORS_LIST[0]
bus.close()
return CAMERA_SENSORS_LIST[1]
def check_sensors_physically_connection():
configuration = "Camera "
preliminar_configuration = ""
sensors_counter = 0
configuration_length = 0
i2c_buses_list = look_for_i2c_buses()
# Look at how the sensors are physically connected
# and create an overlay based on that connection.
# Take one of NVIDIA's overlay names as an example:
# Camera IMX477-A and IMX219-B
# where A and B are the CSI port labels
# The time.sleep is to wait for i2c buses
# to be stable if the service runs immediately
# after the board boots.
time.sleep(30)
for i in i2c_buses_list:
sensor_found = check_i2c_address(i,I2C_EXPANDER_ADDRESS)
# If it is the first time finding a sensor,
# register it as IMX477-A, for example.
if sensor_found not in preliminar_configuration:
preliminar_configuration += sensor_found + '-' + chr(ord('A') + sensors_counter)
else:
# If the sensor is already registered
# Look for the instance index to register
# and register the next findings as:
# IMX477-A-B IMX219-C
delimiter_index = preliminar_configuration.find(sensor_found) + 1
char_counter = delimiter_index
for char in preliminar_configuration[delimiter_index:]:
if char == ' ':
break
char_counter += 1
preliminar_configuration = preliminar_configuration[ : char_counter] + '-' + chr(ord('A') + sensors_counter) + preliminar_configuration[char_counter : ]
sensors_counter += 1
preliminar_configuration = preliminar_configuration.split()
# Check the preliminary configuration found
# to create the overlay. For example:
# Cameras IMX477-A-B and IMX219-C
# or if all the sensors are the same:
# Camera IMX477 Triple
configuration_length = len(preliminar_configuration)
if configuration_length > 1:
configuration = "Cameras "
for i in range(configuration_length):
if (i == configuration_length - 1):
configuration += ' and ' + preliminar_configuration[i]
else:
configuration += preliminar_configuration[i]
else:
configuration += preliminar_configuration[0].split('-')[0] + ' Triple'
print(f"The configuration physically connected is {configuration}")
return configuration
An explanation of the check_i2c_address function may be needed. If the EEPROM at 0x72 exits but is not busy, the read_byte_data will be executed with success. Otherwise, if the EEPROM is busy, the method will fail, but in both cases, this will mean that the sensor connected is the one that has the EEPROM. In this case is the CAMERA_SENSORS_LIST[0].
The output of the above step should be something similar to the following:
The configuration physically connected is Cameras IMX477-A and IMX219-B-C
Note that the above functions use the look_for_i2c_buses function which was already defined at firsthand.
Then, it is time to check for the Device Tree Overlays available in the Jetson IO tool as:
def device_tree_overlays_availables():
try:
# The config-by-hardware.py -l command
# list all the device tree overlays availables
result = subprocess.run(["/opt/nvidia/jetson-io/config-by-hardware.py", "-l"], stdout=PIPE, stderr=PIPE)
configs_availables = []
delimiter = 'Camera'
# Parse the overlay name
# from the Jetson IO output list
for string in result.stdout.splitlines():
string = string.decode('utf-8')
if delimiter in string:
delimiter_index = string.find(delimiter)
configs_availables.append(string[delimiter_index:])
print(f"The device tree overlays found in the JetsonIO tool are {configs_availables}")
return configs_availables
except (subprocess.CalledProcessError, FileNotFoundError) as exception:
if isinstance(exception, subprocess.CalledProcessError):
print("subprocess.CalledProcessError occurred in device_tree_overlays_availables function")
elif isinstance(exception, FileNotFoundError):
print("/opt/nvidia/jetson-io/config-by-hardware.py file not found")
The output of the device_tree_overlays_availables should be something similar to the following:
The device tree overlays found in the JetsonIO tool are ['Camera IMX477 Triple', 'Camera IMX219 Triple', 'Cameras IMX477-A and IMX219-B-C', 'Cameras IMX477-A-B and IMX219-C']
Since the changes of the DT overlay are applied over the original DT and this new DT is used by the Jetson‑IO to create a new entry in /boot/extlinux/extlinux.conf, it is needed to look in the extlinux.conf file if there is any Device Tree Overlay applied. This can be done by using the following function:
def get_extlinux_configuration():
# Look in the extlinux.conf file
# if there is any device tree overlay applied.
# Based on the NVIDIA's example,
# the instance will look like the following:
# <CSI Cameras IMX477-A-B and IMX219-C>
extlinux = os.path.join("/boot", 'extlinux/extlinux.conf')
with open(extlinux, 'r') as f:
while True:
line = f.readline()
if not line:
break
if 'CSI' in line:
return line.split("CSI ",1)[1].rsplit('>',1)[0]
Based on the above output a new Device Tree Overlay should be applied or not. In the case that a Device Tree Overlay has to be applied, the following function can be used:
def apply_new_device_tree_overlay(device_tree_overlay):
try:
# Apply the device tree overlay.
# Only check the processs call
# since is not necessary to
# manipulate the output
# and catch the exception if any
subprocess.check_call(["/opt/nvidia/jetson-io/config-by-hardware.py", "-n 2=", device_tree_overlay], stdout=PIPE, stderr=PIPE)
except (subprocess.CalledProcessError, FileNotFoundError) as exception:
if isinstance(exception, subprocess.CalledProcessError):
print("subprocess.CalledProcessError occurred in apply_new_device_tree_overlay function")
elif isinstance(exception, FileNotFoundError):
print("/opt/nvidia/jetson-io/config-by-hardware.py file not found")
As mentioned before, if the Device Tree Overlay has been successfully applied, a new extlinux.conf file will be created at /boot/extlinux/. Also, the Jetson IO tool creates a backup extlinux file called extlinux.conf.jetson-io-backup that can be used to revert the changes applied.
If the Device Tree Overlay has been applied, the board has to be rebooted so the changes can take effect. This can be finally done in the main function by putting all the functions above detailed together:
def main():
sensors_configuration_connected = check_sensors_physically_connection()
overlays_config_availables = device_tree_overlays_availables()
configuration_already_set = get_extlinux_configuration()
if not overlays_config_availables:
print("There are no overlays configs listed by the JetsonIO tool, "\
"can not continue execution")
return
if sensors_configuration_connected not in overlays_config_availables:
if os.path.exists(EXITLINUX_CONFIG_FILE_BACKUP_NAME):
# Check if the extlinux.confi backup file
# with the original contents exits.
# If so, restore it to avoid any driver to load
# and stay away from any strange behavior
print("The overlay configuration not found by the JetsonIO tool. "\
"There is a previous configuration set so restoring to the "\
"original extlinux.conf file and rebooting the board")
os.system(f'mv {EXITLINUX_CONFIG_FILE_BACKUP_NAME} {EXITLINUX_CONFIG_FILE}')
os.system('reboot')
else:
print("The overlay configuration not found by the JetsonIO tool, not applying any changes")
return
if sensors_configuration_connected == configuration_already_set:
print(f"The overlay configuration {sensors_configuration_connected} is already set, not applying any changes")
return
else:
apply_new_device_tree_overlay(sensors_configuration_connected)
print("No device tree overlay has been found in the /boot/extlinux.conf file.")
print(f"Applying the {sensors_configuration_connected} overlay found and rebooting the board")
os.system('reboot')
if __name__ == '__main__':
main()
Notice that this main function is in charge of verifying if the physical sensors configuration connected has a Device Tree Overlay assigned. The service does not reboot the board if there is not an Overlay already applied. If an Overlay was applied but an Overlay was not found for the new configuration, the service will revert the changes by using the extlinux.conf.jetson-io-backup backup file to avoid any strange behavior to happen.
To make the above a service that works after every board reboot, the following service file has to be created:
[Unit] Description= Sensors configuration detection service to apply the correct device tree overlay and reboot the system if needed. After=multi-user.target [Service] Type=oneshot ExecStart=/usr/bin/python3 /home/nvidia/sensor-configure-detection-service/sensors_config_detection.py [Install] WantedBy=multi-user.target
Enable and start the service by using the following bash script:
#!/bin/bash
# This service will execute the sensors_config_detection.py
# only one time after the Jetson board reboots.
# It works as the last service on boot.
give_permissions_to_service_files () {
chmod 744 sensors_config_detection.py
chmod 644 sensors_config_detection.service
}
enable_and_start_service () {
cp sensors_config_detection.service /etc/systemd/system/
systemctl enable sensors_config_detection.service
systemctl start sensors_config_detection.service
systemctl status sensors_config_detection.service
}
give_permissions_to_service_files
enable_and_start_service
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.
ARTIFICIAL INTELLIGENCE
