NVIDIA Jetson Orin - Video Capture and Display - Cameras - IMX477 J20
|
|
Introduction to Sony IMX477 Linux driver
RidgeRun is actively working to enable customers with great software solutions on top of powerful hardware. For this reason, RidgeRun has worked on enabling a triple capture for the V4L2 driver for the Raspberry Pi HQ camera using an Auvidea J20 board. This wiki contains a brief introduction to the sensor and hardware that will be used to use the driver. Also please keep in mind that the driver can be ported to other Jetson platforms if needed. Please contact us if you require more information.
Raspberry Pi HQ camera
Camera Features
- 12.3-Megapixel high-resolution Sony IMX477 sensor.
- 1.55μm × 1.55μm Pixel Size – double the pixel area of IMX219 for improved low-light performance.
- Back-illuminated sensor architecture for improved sensitivity.
- Support for different C- and CS-mount lenses.
- Integrated back-focus adjustment ring and tripod mount.
Sony IMX477 Basic Drive Mode
The Raspberry Pi HQ camera uses the IMX477 CMOS sensor of Sony to capture. Such a sensor provides the modes included in the table below. At this point, RidgeRun is working to support one mode only but others can be added per request.
Drive Mode | Number of active pixels | Maximum frame rate [frame/s] |
Output Interface | ADC [bit] |
---|---|---|---|---|
Full (4:3) (Normal) | 4056 x 3040 approx. 12.33 M pixels |
60 | CSI-2 | 10 |
40 | CSI-2 | 12 | ||
Full (4:3) (DOL-HDR) | 4056 x 3040 approx. 12.33 M pixels |
DOL 2 frame:30 DOL 3 frame:15 |
CSI-2 | 10 |
Full (16:9) 4K2K (Normal) | 4056 x 2288 approx. 9.28 M pixels |
79 | CSI-2 | 10 |
Full (16:9) 4K2K (DOL-HDR) | 4056 x 2288 approx. 9.28 M pixels |
DOL 2 frame:39 DOL 3 frame:19 |
CSI-2 | 10 |
Full (4:3) Binning (Normal) | 2028 x 1520 approx. 3.08 M pixels |
179 | CSI-2 | 10 |
Full (16:9) Binning 1080P (Normal) | 2028 x 1128 approx. 2.29 M pixels |
240 | CSI-2 | 10 |
Full (16:9) Binning 720P (Normal) | 1348 x 750 approx. 1.01 M pixels |
240 | CSI-2 | 10 |
Full (16:9) Scaling 1080P (Normal) | 2024 x 1142 approx. 2.31 M pixels |
79 | CSI-2 | 10 |
Full (16:9) Scalling 720P (Normal) | 1348 x 762 approx. 1.03 M pixels |
79 | CSI-2 | 10 |
Compatibility with NVIDIA®Jetson™ Platforms
R8 resistor
The Raspberry Pi HQ camera module requires a hardware modification in order to work with Jetson Platforms. Jetson Platforms provide 1.8V for reset GPIO in the camera interface, but the camera module requires 3.3V.
To fix this issue a resistor labeled as R8 must be removed from the camera module. If you decide to apply this fix it is under your own risk. RidgeRun is not responsible for any damage caused to your board. The following picture shows the location of the R8 in the module.
Auvidea J20 adapter
The NVIDIA®Jetson Orin™ makes usage of a Samtec connector, this driver makes usage of the J20 from Auvidea: More information recopilated from RidgeRun can be found here
IMX477 Linux driver for NVIDIA®Jetson Orin™
RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. as a collaborative initiative. In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These instructions have been validated in the following JetPack versions:
- Jetpack 5.0.1
Download JetPack
The current version of the driver is supported in Jetpack 5.0.1. Porting the driver to other versions and platforms is possible.
1. Download and install the NVIDIA SDK Manager (membership is required).
2. Log in with your NVIDIA membership credentials in the SDK Manager.
3. Select JetPack version 5.0.1 DP and Jetson Orin as shown below and press Continue:
4. Accept the terms and conditions and press Continue. The SDK Manager will download and install the selected components:
5. Select Manual Setup. Create a Make sure the board is in Recovery Mode and then press Flash:
Recovery Mode for Jetson Orin AGX
To set the Orin AGX developer kit in Recovery Mode the following steps are required: |
- Make sure the device is connected to the power adapter, but powered off.
- Connect the host computer to the front USB Type-C connector on the device.
- Press and hold the middle (Force Recovery) button.
- Press and hold the left (Power) button.
- Release both buttons.
6. When the OS has been installed successfully in the Jetson Orin AGX, you have multiple options to use the board:
- Use the username and password previously selected to connect over SSH to the corresponded IP (normally 192.168.55.1 over the USB connection).
- Connect over the UART connection opened by attaching the micro-USB, for example minicom:
sudo minicom -D /dev/ttyUSB0 -b 115200
and use the same username and password - Connect over display.
IMPORTANT:
This directory is usually in |
Installing the Driver - JetPack sources patch
Download the JetPack sources
You can obtain the JetPack 5.0.1 DP sources by doing the following:
cd $JETPACK_DIR/Linux_for_Tegra/ ./source_sync.sh -t jetson_34.1.1 JETSON_KERNEL_SOURCE=$JETPACK_DIR/Linux_for_Tegra/sources
Fix kernel before patching
The raw kernel has some considerations before patching:
Move the t23x hardware common to common/kernel-dts
cd $JETSON_KERNEL_SOURCE/hardware/nvidia/platform/t23x/ mv common common_tmp/ mkdir -p common mv common_tmp common/kernel-dts
Also you need to create a symbolic link:
cd $JETSON_KERNEL_SOURCE ln -s $(pwd)/kernel/nvethernetrm $(pwd)/kernel/nvidia/drivers/net/ethernet/nvidia/nvethernet/nvethernetrm # OR ln -s ../../../../../../nvethernetrm $(pwd)/kernel/nvidia/drivers/net/ethernet/nvidia/nvethernet/nvethernetrm
Patch instructions
Download the patches
RidgeRun has created a repository from where you can download the patch files.
git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git
Once you have downloaded the repository, move the files to the $JETSON_KERNEL_SOURCE
directory (create the patches directory inside JETSON_KERNEL_SOURCE if necessary):
KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/ mv -v $KERNEL_PATCH/patches_orin/* $JETSON_KERNEL_SOURCE/patches/
Apply the patches
cd $JETSON_KERNEL_SOURCE/patches quilt push -a
Kernel build instructions
Once the sources have been patched, perform the following steps in order to build and install the driver.
Install the Toolchain
mkdir -p /opt/l4t-gcc cd /opt/l4t-gcc/ wget https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/tarballs/aarch64--glibc--stable-2020.08-1.tar.bz2 tar -xvf aarch64--glibc--stable-2020.08-1.tar.bz2 -C .
Build the kernel
- Establish the building paths anc reate directories:
TOOLCHAIN_PREFIX=/opt/l4t-gcc/aarch64--glibc--stable-2020.08-1/bin/aarch64-buildroot-linux-gnu- TEGRA_KERNEL_OUT=$JETSON_KERNEL_SOURCE/../building/kernel_build KERNEL_MODULES_OUT=$JETSON_KERNEL_SOURCE/../building/kernel_modules mkdir -p $TEGRA_KERNEL_OUT mkdir -p $KERNEL_MODULES_OUT
- Use the following command to install dependencies on Ubuntu:
sudo apt install libncurses-dev
- Configure and build kernel:
cd $JETSON_KERNEL_SOURCE make -C kernel/kernel-5.10/ ARCH=arm64 O=$TEGRA_KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${TOOLCHAIN_PREFIX} tegra_defconfig
- Compile kernel, device tree, and modules:
make -C $SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$TEGRA_KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${TOOLCHAIN_PREFIX} -j$(nproc) Image make -C $SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$TEGRA_KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${TOOLCHAIN_PREFIX} -j$(nproc) dtbs make -C $SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$TEGRA_KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${TOOLCHAIN_PREFIX} -j$(nproc) modules make -C $SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$TEGRA_KERNEL_OUT LOCALVERSION=-tegra INSTALL_MOD_PATH=$KERNEL_MODULES_OUT modules_install
Flash the Jetson
Flash the Xavier NX
Please, make sure the board is in Recovery Mode.
- Copy the generated kernel and dtb:
cd $JETSON_KERNEL_SOURCE/../ cp build/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb ./kernel/dtb cp build/arch/arm64/boot/Image ./kernel
- Flash the dtb:
sudo ./flash.sh -r -k kernel-dtb -d kernel/dtb/tegra194-p3668-all-p3509-0000.dtb jetson-xavier-nx-devkit mmcblk0p1
- Copy the kernel image:
scp ./kernel/Image <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/
- Move the Image from
/tmp
to/boot
inside the Xavier NX's filesystem:
sudo mv /tmp/Image /boot/
Flash the Nano
The DTB depends on the Nano revision:
For Jetson Nano B01: replace <dtb> with tegra210-p3448-0000-p3449-0000-b00.dtb For Jetson Nano A02: replace <dtb> with tegra210-p3448-0000-p3449-0000-a02.dtb |
- Make sure your extlinux.conf file at /boot/extlinux/extlinux.conf in your Nano's filesystem looks as follows by adding the FDT line (remember to change the <dtb> depending on your board).
TIMEOUT 30 DEFAULT primary MENU TITLE L4T boot options LABEL primary MENU LABEL primary kernel LINUX /boot/Image INITRD /boot/initrd FDT /boot/<dtb> APPEND ${cbootargs} quiet
- Copy the kernel image and DTB:
cd $JETSON_KERNEL_SOURCE/../ scp build/arch/arm64/boot/Image <nvidia-nano-user>@<nvidia-nano-ip>:/tmp/ scp build/arch/arm64/boot/dts/<dtb> <nvidia-nano-user>@<nvidia-nano-ip>:/tmp/
- Move the files from
/tmp
to/boot
inside the Nano's filesystem:
sudo mv /tmp/Image /boot/ sudo mv /tmp/<dtb> /boot/
Copy the kernel modules
- You can just access your SD card in your PC and copy the KERNEL_MODULES=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.201/* directory content into /lib/modules/4.9.201/ in your SD card to update the modules. Where
[PLATFORM]
must be replaced byXAVIER_NX
orNANO
, and[JETPACK_VERSION]
must be replaced by4.4
,4.4.1
or4.5
.
- Or you can update with your compiled modules If your board is up you can update the kernel modules as follows:
KERNEL_MODULES=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.140 cd $KERNEL_MODULES # use rsync instead of scp to avoid symlink recursion problems rsync --progress -avhe ssh ../4.9.201 <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/
Where [PLATFORM]
must be replaced by XAVIER_NX
or NANO
, and [JETPACK_VERSION]
must be replaced by 4.4
, 4.4.1
or 4.5
.
- Move the modules from
/tmp
to the/lib/modules/
folder inside the Jetson's filesystem and reboot:
sudo mv /tmp/4.9.201/ /lib/modules/ sudo reboot
- If you find that the directory name is /lib/modules/4.9.201-tegra/, change the directory name to /lib/modules/4.9.201/
- After these steps, the driver should be working and you can run some capture tests described below.
- IMPORTANT: You may see some washed-out colors in the captured image. This is because you haven't applied the ISP configuration file. Make sure to install the ISP file that you can find in the repo using these instructions.
ISP camera overrides installation
Please, visit the Custom ISP Configuration page in order to learn how to use the camera_overrides.isp
file to configure the ISP.
NOTE: This is causing some timeout issues currently.
Supported Features
Resolutions and framerates
- 1920x1080 @ 60fps
- 4032x3040 @ 30fps
Controls
- Gain
- Exposure
- Framerate
- Group Hold
Example Pipelines
Find some example pipelines to use the IMX477 on Jetson Xavier NX below:
Display
1920x1080
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink
4032x3040
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink
MP4 Recording
1920x1080
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
4032x3040
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
JPEG snapshots
1920x1080
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
4032x3040
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
RidgeRun Resources | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contact Us
|