IMX477 J20 Camera Setup Guide for NVIDIA Jetson Orin
|
|
|
Introduction to Sony IMX477 Linux driver
RidgeRun is actively working to enable customers with great software solutions on top of powerful hardware. For this reason, RidgeRun has worked on enabling a triple capture for the V4L2 driver for the Raspberry Pi HQ camera using an Auvidea J20 board. This wiki contains a brief introduction to the sensor and hardware that will be used to use the driver. Also please keep in mind that the driver can be ported to other Jetson platforms if needed. Please contact us if you require more information.
Raspberry Pi HQ camera
Camera Features
- 12.3-Megapixel high-resolution Sony IMX477 sensor.
- 1.55μm × 1.55μm Pixel Size – double the pixel area of IMX219 for improved low-light performance.
- Back-illuminated sensor architecture for improved sensitivity.
- Support for different C- and CS-mount lenses.
- Integrated back-focus adjustment ring and tripod mount.
Sony IMX477 Basic Drive Mode
The Raspberry Pi HQ camera uses the IMX477 CMOS sensor of Sony to capture. Such a sensor provides the modes included in the table below. At this point, RidgeRun is working to support one mode only but others can be added per request.
| Drive Mode | Number of active pixels | Maximum frame rate [frame/s] |
Output Interface | ADC [bit] |
|---|---|---|---|---|
| Full (4:3) (Normal) | 4056 x 3040 approx. 12.33 M pixels |
60 | CSI-2 | 10 |
| 40 | CSI-2 | 12 | ||
| Full (4:3) (DOL-HDR) | 4056 x 3040 approx. 12.33 M pixels |
DOL 2 frame:30 DOL 3 frame:15 |
CSI-2 | 10 |
| Full (16:9) 4K2K (Normal) | 4056 x 2288 approx. 9.28 M pixels |
79 | CSI-2 | 10 |
| Full (16:9) 4K2K (DOL-HDR) | 4056 x 2288 approx. 9.28 M pixels |
DOL 2 frame:39 DOL 3 frame:19 |
CSI-2 | 10 |
| Full (4:3) Binning (Normal) | 2028 x 1520 approx. 3.08 M pixels |
179 | CSI-2 | 10 |
| Full (16:9) Binning 1080P (Normal) | 2028 x 1128 approx. 2.29 M pixels |
240 | CSI-2 | 10 |
| Full (16:9) Binning 720P (Normal) | 1348 x 750 approx. 1.01 M pixels |
240 | CSI-2 | 10 |
| Full (16:9) Scaling 1080P (Normal) | 2024 x 1142 approx. 2.31 M pixels |
79 | CSI-2 | 10 |
| Full (16:9) Scalling 720P (Normal) | 1348 x 762 approx. 1.03 M pixels |
79 | CSI-2 | 10 |
Compatibility with NVIDIA®Jetson™ Platforms
R8 resistor
The Raspberry Pi HQ camera module requires a hardware modification in order to work with Jetson Platforms. Jetson Platforms provide 1.8V for reset GPIO in the camera interface, but the camera module requires 3.3V.
To fix this issue a resistor labeled as R8 must be removed from the camera module. If you decide to apply this fix it is under your own risk. RidgeRun is not responsible for any damage caused to your board. The following picture shows the location of the R8 in the module.
Auvidea J20 adapter
The NVIDIA®Jetson Orin™ makes use of a Samtec connector, this driver makes use of the J20 from Auvidea More information recopilated from RidgeRun can be found here.
The driver will work with IMX477 connected to the J1_1, J1_2, and J1_3 ports of the J20.
IMX477 Linux driver for NVIDIA®Jetson Orin™
RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. and Auvidea. as a collaborative initiative. The driver works with three cameras on the J1 ports from the J20. In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These instructions have been validated in the following JetPack versions:
- Jetpack 5.0.2
Installing the Driver - JetPack sources patch
Obtain the sources
Follow the instructions in our dedicated wiki: Obtaining Sources for JetPack 5.0.2
Patch instructions
Download the patches
RidgeRun has created a repository from where you can download the patch files.
You need to clone it and use the branch for the jetpack.
cd $HOME git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git -b jetpack-5.0.2
Once you have downloaded the repository, copy the files to the $JETSON_SOURCES directory:
KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/ cp $KERNEL_PATCH/patches_orin/5.0.2_agx_imx477_v0.1.patch $JETSON_SOURCES/
Apply the patches
cd $JETSON_SOURCES git apply 5.0.2_agx_imx477_v0.1.patch
Build the kernel, modules, and DTBs
After patching, follow the steps in: Building the Kernel, Modules and DTBs (JetPack 5.0.2)
Flash the Jetson
We present two methods to install the customized kernel, one requires a complete flash, and the other is for quick updating while developing.
Complete flash
Update the Linux_for_Tegra directory
# Copy the new compiled kernel. cp -r $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/ # Copy all new compiled DTBs and DTB overlays. cp -r $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/ # Update rootfs with compiled modules, which include the driver. rsync -azPu $MODULES_OUT/lib/modules/5.10.104-tegra $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/ # Update the rootfs with all the binaries (to make sure the /boot/ folder is updated too). cd $JETPACK_DIR/Linux_for_Tegra sudo ./apply_binaries.sh # Copy the module to the correct location. sudo cp $KERNEL_OUT/drivers/media/i2c/nv_imx477.ko $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers/media/i2c/
Flash
Finally, flash the driver
sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1
Manual installation
Update the kernel and DTB files used by /boot/extlinux/extlinux.conf when there is an already running board. This assumes that the Jetson can be reach by SSH using:
ssh jetson
Updating the kernel
Updating the kernel is as simple as copying into the board:
# Update Image on Linux_for_Tegra folder just to sync the flash folders sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/ # Copy the Image to the tmp folder on the board scp $JETPACK_DIR/Linux_for_Tegra/kernel/Image jetson:/tmp/ # Copy the Image to the boot directory on the board sudo mv /tmp/Image /boot/Image
Updating the DTB
The new JetPack uses the FDT property of the /boot/extlinux/extlinux.conf file, and it sets it to a device tree located in /boot/dtb/ with a prefix of kernel_tegra. We can update this DTB with the following:
# Update on Linux_for_Tegra folder with the new DTBs and overlays sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/ # Generate required prefixed DTB using flash script with no flash option cd $JETPACK_DIR/Linux_for_Tegra sudo ./flash.sh -r --no-flash jetson-agx-orin-devkit mmcblk0p1 # Copy the device tree to the board scp $JETPACK_DIR/Linux_for_Tegra/bootloader/kernel_tegra234-p3701-0000-p3737-0000.dtb jetson:/tmp/ # Update the device tree on the board sudo mv /tmp/kernel_tegra234-p3701-0000-p3737-0000.dtb /boot/dtb/
Updating modules
Inside the board run the following to update the modules folder, copying them from the host:
HOST_NAME=host HOST_IP=192.168.55.100 cd /lib/modules/ sudo rsync -azPu $HOST_NAME@$HOST_IP:$MODULES_OUT/lib/modules/5.10.104-tegra .
ISP camera overrides installation
Please, visit the Custom ISP Configuration page in order to learn how to use the camera_overrides.isp file to configure the ISP.
 | NOTE: This is causing some timeout issues currently. |
Supported Features
Resolutions and framerates
- 1920x1080 @ 60fps
- 3840x2160 @ 30fps
Controls
- Gain
- Exposure
- Framerate
- Group Hold
Example Pipelines
Find some example pipelines to use the IMX477 on Jetson Xavier NX below:
Display
1920x1080
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink
3840x2160
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink
MP4 Recording
1920x1080
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
3840x2160
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
JPEG snapshots
1920x1080
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
3840x2160
SENSOR_ID=0 # 0 to 2 depending on cam port. FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.
ARTIFICIAL INTELLIGENCE
