NVIDIA Jetson Orin - IMX477 Auvidea J20 board Camera Sensor Driver Support
|
|
Introduction to Sony IMX477 Linux driver
RidgeRun is actively working to enable customers with great software solutions on top of powerful hardware. For this reason, RidgeRun has worked on enabling a triple capture for the V4L2 driver for the Raspberry Pi HQ camera using an Auvidea J20 board. This wiki contains a brief introduction to the sensor and hardware that will be used to use the driver. Also please keep in mind that the driver can be ported to other Jetson platforms if needed. Please contact us if you require more information.
Raspberry Pi HQ camera
Camera Features
- 12.3-Megapixel high-resolution Sony IMX477 sensor.
- 1.55μm × 1.55μm Pixel Size – double the pixel area of IMX219 for improved low-light performance.
- Back-illuminated sensor architecture for improved sensitivity.
- Support for different C- and CS-mount lenses.
- Integrated back-focus adjustment ring and tripod mount.
Sony IMX477 Basic Drive Mode
The Raspberry Pi HQ camera uses the IMX477 CMOS sensor of Sony to capture. Such a sensor provides the modes included in the table below. At this point, RidgeRun is working to support one mode only but others can be added per request.
Drive Mode | Number of active pixels | Maximum frame rate [frame/s] |
Output Interface | ADC [bit] |
---|---|---|---|---|
Full (4:3) (Normal) | 4056 x 3040 approx. 12.33 M pixels |
60 | CSI-2 | 10 |
40 | CSI-2 | 12 | ||
Full (4:3) (DOL-HDR) | 4056 x 3040 approx. 12.33 M pixels |
DOL 2 frame:30 DOL 3 frame:15 |
CSI-2 | 10 |
Full (16:9) 4K2K (Normal) | 4056 x 2288 approx. 9.28 M pixels |
79 | CSI-2 | 10 |
Full (16:9) 4K2K (DOL-HDR) | 4056 x 2288 approx. 9.28 M pixels |
DOL 2 frame:39 DOL 3 frame:19 |
CSI-2 | 10 |
Full (4:3) Binning (Normal) | 2028 x 1520 approx. 3.08 M pixels |
179 | CSI-2 | 10 |
Full (16:9) Binning 1080P (Normal) | 2028 x 1128 approx. 2.29 M pixels |
240 | CSI-2 | 10 |
Full (16:9) Binning 720P (Normal) | 1348 x 750 approx. 1.01 M pixels |
240 | CSI-2 | 10 |
Full (16:9) Scaling 1080P (Normal) | 2024 x 1142 approx. 2.31 M pixels |
79 | CSI-2 | 10 |
Full (16:9) Scalling 720P (Normal) | 1348 x 762 approx. 1.03 M pixels |
79 | CSI-2 | 10 |
Compatibility with NVIDIA®Jetson™ Platforms
R8 resistor
The Raspberry Pi HQ camera module requires a hardware modification in order to work with Jetson Platforms. Jetson Platforms provide 1.8V for reset GPIO in the camera interface, but the camera module requires 3.3V.
To fix this issue a resistor labeled as R8 must be removed from the camera module. If you decide to apply this fix it is under your own risk. RidgeRun is not responsible for any damage caused to your board. The following picture shows the location of the R8 in the module.
Auvidea J20 adapter
The NVIDIA®Jetson Orin™ makes use of a Samtec connector, this driver makes use of the J20 from Auvidea More information recopilated from RidgeRun can be found here.
The driver will work with IMX477 connected to the J1_1, J1_2, and J1_3 ports of the J20.
IMX477 Linux driver for NVIDIA®Jetson Orin™
RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. and Auvidea. as a collaborative initiative. The driver works with three cameras on the J1 ports from the J20. In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These instructions have been validated in the following JetPack versions:
- Jetpack 5.0.2
Download JetPack
The current version of the driver is supported in Jetpack 5.0.2. Porting the driver to other versions and platforms is possible.
1. Download and install the NVIDIA SDK Manager (membership is required).
2. Log in with your NVIDIA membership credentials in the SDK Manager.
3. Select JetPack version 5.0.2 DP and Jetson Orin as shown below and press Continue:
4. Accept the terms and conditions and press Continue. The SDK Manager will download and install the selected components:
5. Select Manual Setup. Create a Make sure the board is in Recovery Mode and then press Flash:
Recovery Mode for Jetson Orin AGX
To set the Orin AGX developer kit in Recovery Mode the following steps are required: |
- Make sure the device is connected to the power adapter but powered off.
- Connect the host computer to the front USB Type-C connector on the device.
- Power the device
- Press and hold the middle (Force Recovery) button.
- Press and hold the left (Power) button.
- Release both buttons.
6. When the OS has been installed successfully in the Jetson Orin AGX, you have multiple options to use the board:
- Use the username and password previously selected to connect over SSH to the corresponding IP (normally 192.168.55.1 over the USB connection).
- Connect over the UART connection opened by attaching the micro-USB, for example, minicom:
sudo minicom -D /dev/ttyUSB0 -b 115200
and use the same username and password - Connect over display.
IMPORTANT:
This directory is usually in |
Installing the Driver - JetPack sources patch
Download the JetPack sources
You can obtain the JetPack 5.0.2 DP sources by doing the following:
1. Go to https://developer.nvidia.com/embedded/jetson-linux-archive and download the source files for your release.
2. Extract the .tbz2 file:
tar -xjf public_sources.tbz2
3. Extract the kernel source file:
cd Linux_for_Tegra/source/public JETSON_SOURCES=$PWD tar –xjf kernel_src.tbz2
This extracts the kernel source to the kernel/ subdirectory.
Patch instructions
Download the patches
RidgeRun has created a repository from where you can download the patch files.
You need to clone it and use the branch for the jetpack.
cd $HOME git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git -b jetpack-5.0.2
Once you have downloaded the repository, copy the files to the $JETSON_SOURCES
directory:
KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/ cp $KERNEL_PATCH/patches_orin/5.0.2_agx_imx477_v0.1.patch $JETSON_SOURCES/
Apply the patches
cd $JETSON_SOURCES git apply 5.0.2_agx_imx477_v0.1.patch
Kernel build instructions
Once the sources have been patched, perform the following steps in order to build and install the driver.
Install the Toolchain
First, download the Bootlin toolchain binaries from the NVIDIA Jetson Linux home page. Go to the bottom of the page and click on the Bootlin Toolchain gcc option as shown in the image below:
After that do:
mkdir $HOME/l4t-gcc cd $HOME/l4t-gcc tar xf <path_to_toolchain_archive>
Build the kernel
- Establish the building paths and create directories:
KERNEL_OUT=$JETSON_SOURCES/kernel_out MODULES_OUT=$JETSON_SOURCES/modules_out mkdir -p $KERNEL_OUT mkdir -p $MODULES_OUT
- Use the following command to install dependencies on Ubuntu:
sudo apt install libncurses-dev
- Export environment variables:
export CROSS_COMPILE_AARCH64_PATH=$HOME/l4t-gcc export CROSS_COMPILE_AARCH64=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-
- Compile kernel:
cd $JETSON_SOURCES ./nvbuild.sh -o $KERNEL_OUT
- Install the modules
make -C $JETSON_SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra INSTALL_MOD_PATH=$MODULES_OUT modules_install
Flash the Jetson
We present two methods to install the customized kernel, one requires a complete flash, and the other is for quick updating while developing.
Complete flash
Update the Linux_for_Tegra directory
# Copy the new compiled kernel. cp -r $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/ # Copy all new compiled DTBs and DTB overlays. cp -r $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/ # Update rootfs with compiled modules, which include the driver. rsync -azPu $MODULES_OUT/lib/modules/5.10.104-tegra $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/ # Update the rootfs with all the binaries (to make sure the /boot/ folder is updated too). cd $JETPACK_DIR/Linux_for_Tegra sudo ./apply_binaries.sh # Copy the module to the correct location. sudo cp $KERNEL_OUT/drivers/media/i2c/nv_imx477.ko $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers/media/i2c/
Flash
Finally, flash the driver
sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1
Manual installation
Update the kernel and DTB files used by /boot/extlinux/extlinux.conf when there is an already running board. This assumes that the Jetson can be reach by SSH using:
ssh jetson
Updating the kernel
Updating the kernel is as simple as copying into the board:
# Update Image on Linux_for_Tegra folder just to sync the flash folders sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/ # Copy the Image to the tmp folder on the board scp $JETPACK_DIR/Linux_for_Tegra/kernel/Image jetson:/tmp/ # Copy the Image to the boot directory on the board sudo mv /tmp/Image /boot/Image
Updating the DTB
The new JetPack uses the FDT property of the /boot/extlinux/extlinux.conf file, and it sets it to a device tree located in /boot/dtb/ with a prefix of kernel_tegra. We can update this DTB with the following:
# Update on Linux_for_Tegra folder with the new DTBs and overlays sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/ # Generate required prefixed DTB using flash script with no flash option cd $JETPACK_DIR/Linux_for_Tegra sudo ./flash.sh -r --no-flash jetson-agx-orin-devkit mmcblk0p1 # Copy the device tree to the board scp $JETPACK_DIR/Linux_for_Tegra/bootloader/kernel_tegra234-p3701-0000-p3737-0000.dtb jetson:/tmp/ # Update the device tree on the board sudo mv /tmp/kernel_tegra234-p3701-0000-p3737-0000.dtb /boot/dtb/
Updating modules
Inside the board run the following to update the modules folder, copying them from the host:
HOST_NAME=host HOST_IP=192.168.55.100 cd /lib/modules/ sudo rsync -azPu $HOST_NAME@$HOST_IP:$MODULES_OUT/lib/modules/5.10.104-tegra .
ISP camera overrides installation
Please, visit the Custom ISP Configuration page in order to learn how to use the camera_overrides.isp
file to configure the ISP.
NOTE: This is causing some timeout issues currently. |
Supported Features
Resolutions and framerates
- 1920x1080 @ 60fps
- 3840x2160 @ 30fps
Controls
- Gain
- Exposure
- Framerate
- Group Hold
Example Pipelines
Find some example pipelines to use the IMX477 on Jetson Xavier NX below:
Display
1920x1080
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink
3840x2160
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink
MP4 Recording
1920x1080
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
3840x2160
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
JPEG snapshots
1920x1080
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
3840x2160
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.