NVIDIA Jetson Orin - Video Capture and Display - Cameras - IMX477 J20

Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help.

Introduction to Sony IMX477 Linux driver

RidgeRun is actively working to enable customers with great software solutions on top of powerful hardware. For this reason, RidgeRun has worked on enabling a triple capture for the V4L2 driver for the Raspberry Pi HQ camera using an Auvidea J20 board. This wiki contains a brief introduction to the sensor and hardware that will be used to use the driver. Also please keep in mind that the driver can be ported to other Jetson platforms if needed. Please contact us if you require more information.

Raspberry Pi HQ camera

Camera Features

12.3-Megapixel high-resolution Sony IMX477 sensor.
1.55μm × 1.55μm Pixel Size – double the pixel area of IMX219 for improved low-light performance.
Back-illuminated sensor architecture for improved sensitivity.
Support for different C- and CS-mount lenses.
Integrated back-focus adjustment ring and tripod mount.

Sony IMX477 Basic Drive Mode

The Raspberry Pi HQ camera uses the IMX477 CMOS sensor of Sony to capture. Such a sensor provides the modes included in the table below. At this point, RidgeRun is working to support one mode only but others can be added per request.

Drive Mode	Number of active pixels	Maximum frame rate [frame/s]	Output Interface	ADC [bit]
Full (4:3) (Normal)	4056 x 3040 approx. 12.33 M pixels	60	CSI-2	10
Full (4:3) (Normal)	4056 x 3040 approx. 12.33 M pixels	40	CSI-2	12
Full (4:3) (DOL-HDR)	4056 x 3040 approx. 12.33 M pixels	DOL 2 frame:30 DOL 3 frame:15	CSI-2	10
Full (16:9) 4K2K (Normal)	4056 x 2288 approx. 9.28 M pixels	79	CSI-2	10
Full (16:9) 4K2K (DOL-HDR)	4056 x 2288 approx. 9.28 M pixels	DOL 2 frame:39 DOL 3 frame:19	CSI-2	10
Full (4:3) Binning (Normal)	2028 x 1520 approx. 3.08 M pixels	179	CSI-2	10
Full (16:9) Binning 1080P (Normal)	2028 x 1128 approx. 2.29 M pixels	240	CSI-2	10
Full (16:9) Binning 720P (Normal)	1348 x 750 approx. 1.01 M pixels	240	CSI-2	10
Full (16:9) Scaling 1080P (Normal)	2024 x 1142 approx. 2.31 M pixels	79	CSI-2	10
Full (16:9) Scalling 720P (Normal)	1348 x 762 approx. 1.03 M pixels	79	CSI-2	10

Compatibility with NVIDIA^®Jetson™ Platforms

R8 resistor

The Raspberry Pi HQ camera module requires a hardware modification in order to work with Jetson Platforms. Jetson Platforms provide 1.8V for reset GPIO in the camera interface, but the camera module requires 3.3V.

To fix this issue a resistor labeled as R8 must be removed from the camera module. If you decide to apply this fix it is under your own risk. RidgeRun is not responsible for any damage caused to your board. The following picture shows the location of the R8 in the module.

Figure 1. R8 resistor on RPi HQ camera module.

Auvidea J20 adapter

The NVIDIA^®Jetson Orin™ makes usage of a Samtec connector, this driver makes usage of the J20 from Auvidea: More information recopilated from RidgeRun can be found here

IMX477 Linux driver for NVIDIA^®Jetson Orin™

RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. as a collaborative initiative. In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These instructions have been validated in the following JetPack versions:

Jetpack 5.0.1

Download JetPack

The current version of the driver is supported in Jetpack 5.0.1. Porting the driver to other versions and platforms is possible.

1. Download and install the NVIDIA SDK Manager (membership is required).

2. Log in with your NVIDIA membership credentials in the SDK Manager.

3. Select JetPack version 5.0.1 DP and Jetson Orin as shown below and press Continue:

Figure 1. Required SDK Manager options.

4. Accept the terms and conditions and press Continue. The SDK Manager will download and install the selected components:

Figure 2. SDK Manager download and install software.

5. Select Manual Setup. Create a Make sure the board is in Recovery Mode and then press Flash:

Figure 3. Manual setup to start the flashing process.

Recovery Mode for Jetson Orin AGX

Make sure the device is connected to the power adapter, but powered off.
Connect the host computer to the front USB Type-C connector on the device.
Press and hold the middle (Force Recovery) button.
Press and hold the left (Power) button.
Release both buttons.

6. When the OS has been installed successfully in the Jetson Orin AGX, you have multiple options to use the board:

Use the username and password previously selected to connect over SSH to the corresponded IP (normally 192.168.55.1 over the USB connection).
Connect over the UART connection opened by attaching the micro-USB, for example minicom: sudo minicom -D /dev/ttyUSB0 -b 115200 and use the same username and password
Connect over display.

Installing the Driver - JetPack sources patch

Download the JetPack sources

The sources can be downloaded from NVIDIA's Download Center:

Jetpack 4.4

Jetpack 4.4.1

Jetpack 4.5

1. You must be able to download a file called public_sources.tbz2.

2. Untar the sources

tar -xvf public_sources.tbz2

3. Let's unpack the kernel_src.tbz2 and move the hardware/ and kernel/ folders into Linux_for_Tegra/ inside the JetPack directory $JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/:

cd Linux_for_Tegra/source/public/
tar -xvf kernel_src.tbz2
JETSON_KERNEL_SOURCE=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/source/
mkdir -p $JETSON_KERNEL_SOURCE
mv hardware/ kernel/ $JETSON_KERNEL_SOURCE

Where [PLATFORM] must be replaced by XAVIER_NX or NANO, and [JETPACK_VERSION] must be replaced by 4.4, 4.4.1 or 4.5.

Patch instructions

Download the patches

RidgeRun has created a repository from where you can download the patch files.

git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git

Older versions can be found in jetpack-4.4 and jetpack-4.4.1 branches in the repository. You just have to switch to the jetpack version branch you want to use.

Once you have downloaded the repository, move the files to the $JETSON_KERNEL_SOURCE directory (create the patches directory inside JETSON_KERNEL_SOURCE if necessary):

KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/
mv -v $KERNEL_PATCH/patches_[PLATFORM]/* $JETSON_KERNEL_SOURCE/patches/

Where [PLATFORM] must be replaced by nx or nano.

Apply the patches

cd $JETSON_KERNEL_SOURCE/patches
quilt push

Kernel build instructions

Once the sources have been patched, perform the following steps in order to build and install the driver.

Install the Toolchain

NVIDIA recommends using the Linaro 7.3.1 2018.05 toolchain. Download the pre-built toolchain binaries as follows:

wget http://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz

Execute the following commands to extract the toolchain:

mkdir -p $HOME/l4t-gcc
cd $HOME/l4t-gcc
tar xf gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz

Build the kernel

Create the main directories:

cd $JETSON_KERNEL_SOURCE/../
mkdir -p modules/
mkdir -p packages/
mkdir -p dtb/

Establish the building paths:

CROSS_COMPILE=$HOME/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
KERNEL_OUT=$JETSON_KERNEL_SOURCE/../build
KERNEL_MODULES_OUT=$JETSON_KERNEL_SOURCE/../modules

Use the following command to install dependencies on Ubuntu:

sudo apt install libncurses-dev

Configure and build kernel:

cd $JETSON_KERNEL_SOURCE
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT tegra_defconfig
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT menuconfig

Select:

Device Drivers  --->
  <*> Multimedia support  --->
      NVIDIA overlay Encoders, decoders, sensors and other helper chips  --->
          <*> IMX477 camera sensor support

Press the Y key in order to select the IMX477 option. Go back by hitting the double Esc key until you get the message: Do you want to save your new configuration?, select Yes and press Enter.

Compile kernel, device tree, and modules:

make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 Image
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 dtbs
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 modules
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT modules_install INSTALL_MOD_PATH=$KERNEL_MODULES_OUT

Flash the Jetson

Flash the Xavier NX

Please, make sure the board is in Recovery Mode.

Copy the generated kernel and dtb:

cd $JETSON_KERNEL_SOURCE/../
cp build/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb ./kernel/dtb
cp build/arch/arm64/boot/Image ./kernel

Flash the dtb:

sudo ./flash.sh -r -k kernel-dtb -d kernel/dtb/tegra194-p3668-all-p3509-0000.dtb jetson-xavier-nx-devkit mmcblk0p1

Copy the kernel image:

scp ./kernel/Image <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/

Move the Image from /tmp to /boot inside the Xavier NX's filesystem:

sudo mv /tmp/Image /boot/

Flash the Nano

Make sure your extlinux.conf file at /boot/extlinux/extlinux.conf in your Nano's filesystem looks as follows by adding the FDT line (remember to change the <dtb> depending on your board).

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      FDT /boot/<dtb>
      APPEND ${cbootargs} quiet

Copy the kernel image and DTB:

cd $JETSON_KERNEL_SOURCE/../
scp build/arch/arm64/boot/Image <nvidia-nano-user>@<nvidia-nano-ip>:/tmp/
scp build/arch/arm64/boot/dts/<dtb> <nvidia-nano-user>@<nvidia-nano-ip>:/tmp/

Move the files from /tmp to /boot inside the Nano's filesystem:

sudo mv /tmp/Image /boot/
sudo mv /tmp/<dtb> /boot/

Copy the kernel modules

You can just access your SD card in your PC and copy the KERNEL_MODULES=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.201/* directory content into /lib/modules/4.9.201/ in your SD card to update the modules. Where [PLATFORM] must be replaced by XAVIER_NX or NANO, and [JETPACK_VERSION] must be replaced by 4.4, 4.4.1 or 4.5.

Or you can update with your compiled modules If your board is up you can update the kernel modules as follows:

KERNEL_MODULES=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.140
cd $KERNEL_MODULES
# use rsync instead of scp to avoid symlink recursion problems
rsync --progress -avhe ssh ../4.9.201 <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/

Where [PLATFORM] must be replaced by XAVIER_NX or NANO, and [JETPACK_VERSION] must be replaced by 4.4, 4.4.1 or 4.5.

Move the modules from /tmp to the /lib/modules/ folder inside the Jetson's filesystem and reboot:

sudo mv /tmp/4.9.201/ /lib/modules/
sudo reboot

If you find that the directory name is /lib/modules/4.9.201-tegra/, change the directory name to /lib/modules/4.9.201/
After these steps, the driver should be working and you can run some capture tests described below.
IMPORTANT: You may see some washed-out colors in the captured image. This is because you haven't applied the ISP configuration file. Make sure to install the ISP file that you can find in the repo using these instructions.

ISP camera overrides installation

Please, visit the Custom ISP Configuration page in order to learn how to use the camera_overrides.isp file to configure the ISP.

Supported Features

Resolutions and framerates

1920x1080 @ 60fps
4032x3040 @ 30fps

Controls

Gain
Exposure
Framerate
Group Hold

Example Pipelines

Find some example pipelines to use the IMX477 on Jetson Xavier NX below:

Display

1920x1080

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink

4032x3040

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink

MP4 Recording

1920x1080

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4

4032x3040

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4

JPEG snapshots

1920x1080

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
NUMBER_OF_SNAPSHOTS=20
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg

4032x3040

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode
NUMBER_OF_SNAPSHOTS=20
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg

RidgeRun Resources