RidgeRun Developer Wiki

NVIDIA Jetson Orin/Video Capture and Display/Cameras/IMX477 J20: Difference between revisions

< NVIDIA Jetson Orin‎ | Video Capture and Display
mNo edit summary
 
(54 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<noinclude>
<noinclude>
{{NVIDIA Jetson Orin/Head|previous=Video Capture and Display/Cameras/IMX219|next=Video Capture and Display/Software Support|keywords=jetpack,flashing,image,flash,flashing board}}
{{NVIDIA Jetson Orin/Head|previous=Video Capture and Display|next=GStreamer Pipelines|metakeywords=imx477, j20, mipi camera, Auvidea, Auvidea J20 board, Raspberry Pi HQ camera, IMX477 Linux driver for NVIDIA, IMX477 Linux driver}}
</noinclude>
</noinclude>


__TOC__
{{DISPLAYTITLE:NVIDIA Jetson Orin - IMX477 Auvidea J20 board Camera Sensor Driver Support|noerror}}
 


<table>
<table>
Line 82: Line 81:
=== Auvidea J20 adapter ===
=== Auvidea J20 adapter ===


The NVIDIA<sup>®</sup>Jetson Orin™ makes usage of a Samtec connector, this driver makes usage of the J20 from Auvidea: More information recopilated from RidgeRun can be found [[Getting_started_guide_for_Auvidea_J20_board|here]]
The NVIDIA<sup>®</sup>Jetson Orin™ makes use of a Samtec connector, this driver makes use of the J20 from Auvidea
More information recopilated from RidgeRun can be found [[Getting_started_guide_for_Auvidea_J20_board|here]].
 
The driver will work with IMX477 connected to the J1_1, J1_2, and J1_3 ports of the J20.


==IMX477 Linux driver for NVIDIA<sup>®</sup>Jetson Orin™==
==IMX477 Linux driver for NVIDIA<sup>®</sup>Jetson Orin™==


RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. as a collaborative initiative.
RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. and Auvidea. as a collaborative initiative. The driver works with three cameras on the J1 ports from the J20.  
In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These  instructions have been validated in the following JetPack versions:
In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These  instructions have been validated in the following JetPack versions:
* Jetpack 5.0.1
* Jetpack 5.0.2


===Download JetPack===
===Download JetPack===


The current version of the driver is supported in Jetpack 5.0.1. Porting the driver to other versions and platforms is possible.
The current version of the driver is supported in Jetpack 5.0.2. Porting the driver to other versions and platforms is possible.


1. Download and install the [https://developer.nvidia.com/nvidia-sdk-manager NVIDIA SDK Manager] (membership is required).
1. Download and install the [https://developer.nvidia.com/nvidia-sdk-manager NVIDIA SDK Manager] (membership is required).


2. Put your NVIDIA membership credentials in the SDK Manager main window:
2. Log in with your NVIDIA membership credentials in the SDK Manager.
[[Image:SDKMCredentials.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 2'''. Required SDK Manager credentials.</div>
3. Select JetPack version and Jetson [Xavier NX | Nano] as shown below and press ''Continue'':
[[Image:SDKMOptions2.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 3'''. Required SDK Manager options.</div>
4. Accept the terms and conditions and press ''Continue''. The SDK Manager will download and install the selected components:
[[Image:SDKMInstall.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 4'''. SDK Manager download and install software.</div>
5. Select '''Manual Setup'''. Make sure the board is in ''Recovery Mode'' and then press '''Flash''':


[[Image:ManualOption.png|thumb|center|650px]]  
3. Select JetPack version 5.0.2 DP and Jetson Orin as shown below and press ''Continue'':
<div style="text-align: center;">'''Figure 5'''. Manual setup to start the flashing process.</div>
[[Image:Sdk1.png|thumb|center|650px]]  
<div style="text-align: center;">'''Figure 1'''. Required SDK Manager options.</div>


====Recovery Mode for Jetson NX====
4. Accept the terms and conditions and press ''Continue''. The SDK Manager will download and install the selected components:
[[Image:imx477_sdkmanager2.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 2'''. SDK Manager download and install software.</div>
5. Select '''Manual Setup'''. Create a  Make sure the board is in ''Recovery Mode'' and then press '''Flash''':
[[Image:imx477_sdkmanager3.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 3'''. Manual setup to start the flashing process.</div>


{{Ambox
====Recovery Mode for Jetson Orin AGX====
|type=notice
|small=left
|issue=To set the Xavier NX developer kit in '''Recovery Mode''' the following steps are required:
|style=width:unset;
}}
 
# Ensure the device is powered off and the power adapter is disconnected.
# Verify that a microSD card is inserted in the Jetson Xavier NX module's card slot.
# Place a jumper across the Force Recovery Mode pins (9 and 10) on the button header [J14]. Please, see Figure 6 below.
# Connect your host computer to the developer kit's USB Micro-B connector.
# Connect the power supply to the power jack [J16]. The developer kit automatically powers on in Force Recovery Mode.
# Remove the jumper from the Force Recovery Mode pins.
# Use the <code>lsusb</code> command to check if the <code>NVidia corp</code> message is shown.<br>
<br>
[[Image:recmodeNX.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 6'''. Jumper between pins 9 and 10 to get the board on Recovery Mode (top view).</div><br>
 
====Recovery Mode for Jetson Nano====
* To set the Jetson Nano in ''Recovery Mode'', follow these steps:
# Ensure the device is powered off and the power adapter is disconnected.
# Verify that a microSD card is inserted in the Jetson Nano SD card slot.
# Place a jumper across the Force Recovery pins.
# Enable the use of the Power Jack [J15] by placing a jumper across the Power Select Header [J48] pins.
# Connect your host computer to the device's Micro USB connector.
# Connect the power supply to the power jack [J25]. The developer kit automatically powers on in Force Recovery Mode.
# Remove the jumper from the Force Recovery Mode pins.


{{Ambox
{{Ambox
|type=notice
|type=notice
|small=left
|small=left
|issue=
|issue=To set the Orin AGX developer kit in '''Recovery Mode''' the following steps are required:
*For the B01 carrier board, the force recovery pins are the 9 and 10 pins (FC REC and GND) of the Button Header. The B01 Button Header is located on the edge of the carrier board under the heat sink and SD card slot.<br>
*For the A02 carrier board, the force recovery pins are the 3 and 4 pins of the Button Header.
|style=width:unset;
|style=width:unset;
}}
}}
<br>
6. When the OS has been installed successfully in the [Xavier NX | Nano], you will be asked to enter the board IP, a username and a password, as shown below in Figure7:
[[Image:OSIPOption.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 7'''. Required username and password for your profile in the board.</div>
7. Don't close the window of Figure 7 and don't modify its content yet. You need to connect a screen to the board via HDMI, a keyboard/mouse via USB port, and a network cable via Ethernet port. You can remove the jumper that keeps the board in ''Recovery Mode'' and then reboot the Xavier board at this point if necessary.


8. After reboot, you will be asked to accept the terms of the license' on the board's screen. Select the checkbox and press ''Continue''.
# Make sure the device is connected to the power adapter but powered off.
# Connect the host computer to the front USB Type-C connector on the device.
# Power the device
# Press and hold the middle (Force Recovery) button.
# Press and hold the left (Power) button.
# Release both buttons.


9. Select your language in the next screen and press ''Continue'':
6. When the OS has been installed successfully in the Jetson Orin AGX, you have multiple options to use the board:


[[Image:languageNX.png|thumb|center|650px]]
# Use the username and password previously selected to connect over SSH to the corresponding IP (normally 192.168.55.1 over the USB connection).  
<div style="text-align: center;">'''Figure 8'''. Language selection.</div>
# Connect over the UART connection opened by attaching the micro-USB, for example, minicom: <code>sudo minicom -D /dev/ttyUSB0 -b 115200</code> and use the same username and password
 
# Connect over display.
10. Select your keyboard layout and press ''Continue'':
 
[[Image:keyboardNX.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 9'''. Keyboard layout selection.</div>
 
11. Select your location and press ''Continue'':
 
[[Image:locationNX.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 10'''. Location selection.</div>
 
12. Enter a username and a password. Then, press ''Continue'':
 
[[Image:usernameNX.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 11'''. User credentials for Ubuntu.</div>
 
13. Wait for the system to apply the changes. The board will restart automatically.
 
14. Now you are able to access Ubuntu as a new user in the [Xavier NX | Nano]. You can find the board IP by using <code>ifconfig</code> from a terminal.
 
15. Get back to the window presented in Figure 7 (in your host PC) and enter the IP, username, and password that the [Xavier NX | Nano] has been configured with. Then press ''Install''.
 
[[Image:preinstallNX.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 12'''. User credentials to start the software installation process.</div>
 
16. The software installation process should finish successfully, as shown in Figure 13:
 
[[Image:finishNX.png|thumb|center|650px]]
<div style="text-align: center;">'''Figure 13'''. Finishing software install.</div>


<br>
<br>
Line 198: Line 138:
|small=left
|small=left
|issue='''IMPORTANT:'''
|issue='''IMPORTANT:'''
*From now on we will refer as <code>$JETPACK_DIR/</code> to the directory where the SDK Manager placed the <code>JetPack_4.4_Linux_JETSON_[PLATFORM]_DEVKIT</code> / <code>JetPack_4.4.1_Linux_JETSON_[PLATFORM]_DEVKIT</code> / <code>JetPack_4.5_Linux_JETSON_[PLATFORM]_DEVKIT</code>folder.<br>
*From now on we will refer as <code>$JETPACK_DIR</code> to the directory where the SDK Manager placed the <code>JetPack_5.0.2_DP_Linux_JETSON_AGX_ORIN_TARGETS<br>
       Where <code>[PLATFORM]</code> must be replaced by <code>XAVIER_NX</code> or <code>NANO</code>. This directory is usually <code>$HOME/nvidia/nvidia_sdk.<code>
       This directory is usually in <code>$HOME/nvidia/nvidia_sdk<code>
|style=width:unset;
|style=width:unset;
}}
}}
<br>
<br>


== Installing the Driver - Option A: Debian Packages (Recommended) ==
== Installing the Driver - JetPack sources patch==
 
This is the simplest way to install the IMX477 driver on Jetson platforms.
 
===Downloading the debian packages===
 
You can download the Debian packages according to your platform from the following link:
 
[https://drive.google.com/drive/folders/1E1bE57VlXqadLGg8sSw8KwW7AJRWjbmG?usp=sharing Kernel Debian Packages]
 
===Installing the debian packages===
 
====Jetson Xavier NX====
 
* Copy the kernel and dtb Debian packages to your board filesystem:
 
<pre>
scp debian_nx/nvidia-l4t-kernel_4.9.140-tegra-32.4.3-20200625213407_arm64.deb [jetson-username]@[jetson-ip]:/home/nvidia/
scp debian_nx/nvidia-l4t-kernel-dtbs_4.9.140-tegra-32.4.3-20200625213407_arm64.deb [jetson-username]@[jetson-ip]:/home/nvidia/
</pre>
 
* Install the debian packages in your board:
<pre>
sudo apt-get install --reinstall ./nvidia-l4t-kernel_4.9.140-tegra-32.4.3-20200625213407_arm64.deb
sudo apt-get install --reinstall ./nvidia-l4t-kernel-dtbs_4.9.140-tegra-32.4.3-20200625213407_arm64.deb
</pre>
 
* Enable the dtb changes by modifying the <code>/boot/extlinux/extlinux.conf</code> file:
<pre>
TIMEOUT 30
DEFAULT primary
 
MENU TITLE L4T boot options
 
LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
      FDT /boot/tegra194-p3668-all-p3509-0000.dtb
</pre>
 
* Reboot your board:
<pre>
sudo reboot
</pre>
 
====Jetson Nano====
 
* Copy the kernel and dtb debian packages to your board filesystem:
 
<pre>
scp debian_nano/nvidia-l4t-kernel_4.9.140-tegra-32.4.3-20200625213809_arm64.deb [jetson-username]@[jetson-ip]:/home/nvidia/
scp debian_nano/nvidia-l4t-kernel-dtbs_4.9.140-tegra-32.4.3-20200625213809_arm64.deb [jetson-username]@[jetson-ip]:/home/nvidia/
</pre>
 
* Install the debian packages in your board:
<pre>
sudo apt-get install --reinstall ./nvidia-l4t-kernel_4.9.140-tegra-32.4.3-20200625213809_arm64.deb
sudo apt-get install --reinstall ./nvidia-l4t-kernel-dtbs_4.9.140-tegra-32.4.3-20200625213809_arm64.deb
</pre>
 
* Enable the dtb changes by modifying the <code>/boot/extlinux/extlinux.conf</code> file:
<br>
{{Ambox
|type=notice
|small=left
|issue='''The FDT depends on the Nano revision:'''
'''For Jetson Nano B01:''' replace <dtb path> with /boot/tegra210-p3448-0000-p3449-0000-b00.dtb
 
'''For Jetson Nano A02:''' replace <dtb path> with /boot/tegra210-p3448-0000-p3449-0000-a02.dtb
|style=width:unset;
}}
<br>
<pre>
TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0
      FDT <dtb path>
</pre>
 
* Reboot your board:
<pre>
sudo reboot
</pre>
 
* After these steps, the driver should be working and you can run some capture tests described below.
* '''IMPORTANT''': You may see some washed-out colors in the captured image. This is because you haven't applied the ISP configuration file. Make sure to install the ISP file that you can find in the repo using [https://developer.ridgerun.com/wiki/index.php?title=Raspberry_Pi_HQ_camera_IMX477_Linux_driver_for_Jetson#ISP_camera_overrides_installation these instructions].
 
== Installing the Driver - Option B: JetPack sources patch==


===Download the JetPack sources===
===Download the JetPack sources===


The sources can be downloaded from NVIDIA's Download Center:
You can obtain the JetPack 5.0.2 DP sources by doing the following:
 
Jetpack 4.4
* [https://developer.nvidia.com/embedded/L4T/r32_Release_v4.3/sources/T210/public_sources.tbz2 Jetson Nano sources]
* [https://developer.nvidia.com/embedded/L4T/r32_Release_v4.3/sources/T186/public_sources.tbz2 Jetson Xavier NX sources]
Jetpack 4.4.1
* [https://developer.nvidia.com/embedded/L4T/r32_Release_v4.4/r32_Release_v4.4-GMC3/Sources/T210/public_sources.tbz2 Jetson Nano sources]
* [https://developer.nvidia.com/embedded/L4T/r32_Release_v4.4/r32_Release_v4.4-GMC3/Sources/T186/public_sources.tbz2 Jetson Xavier NX sources]
Jetpack 4.5
* [https://developer.nvidia.com/embedded/L4T/r32_Release_v5.0/sources/T210/public_sources.tbz2 Jetson Nano sources]
* [https://developer.nvidia.com/embedded/L4T/r32_Release_v5.0/sources/T186/public_sources.tbz2 Jetson Xavier NX sources]


1. You must be able to download a file called <code>public_sources.tbz2</code>.
1. Go to https://developer.nvidia.com/embedded/jetson-linux-archive and download the source files for your release.


2. Untar the sources
2. Extract the .tbz2 file:
<pre>
<pre>
tar -xvf public_sources.tbz2
tar -xjf public_sources.tbz2
</pre>
</pre>
 
3. Extract the kernel source file:
3. Let's unpack the <code>kernel_src.tbz2</code> and move the <code>hardware/</code> and <code>kernel/</code> folders into <code>Linux_for_Tegra/</code> inside the JetPack directory <code>$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/</code>:
 
<pre>
<pre>
cd Linux_for_Tegra/source/public/
cd Linux_for_Tegra/source/public
tar -xvf kernel_src.tbz2
JETSON_SOURCES=$PWD
JETSON_KERNEL_SOURCE=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/source/
tar –xjf kernel_src.tbz2
mkdir -p $JETSON_KERNEL_SOURCE
mv hardware/ kernel/ $JETSON_KERNEL_SOURCE
</pre>
</pre>
 
This extracts the kernel source to the kernel/ subdirectory.
Where <code>[PLATFORM]</code> must be replaced by <code>XAVIER_NX</code> or <code>NANO</code>, and <code>[JETPACK_VERSION]</code> must be replaced by <code>4.4</code>, <code>4.4.1</code> or <code>4.5</code>.


===Patch instructions===
===Patch instructions===
Line 337: Line 168:
====Download the patches====
====Download the patches====


RidgeRun has created a [https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git repository] from where you can download the patch files.
RidgeRun has created a [https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3/tree/jetpack-5.0.2 repository] from where you can download the patch files.


You need to clone it and use the branch for the jetpack.
<pre>
<pre>
git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git
cd $HOME
git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git -b jetpack-5.0.2
</pre>
</pre>


Older versions can be found in '''jetpack-4.4''' and '''jetpack-4.4.1''' branches in the repository. You just have to switch to the jetpack version branch you want to use.
Once you have downloaded the repository, copy the files to the <code>$JETSON_SOURCES</code> directory:
 
Once you have downloaded the repository, move the files to the <code>$JETSON_KERNEL_SOURCE</code> directory (create the patches directory inside JETSON_KERNEL_SOURCE if necessary):


<pre>
<pre>
KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/
KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/
mv -v $KERNEL_PATCH/patches_[PLATFORM]/* $JETSON_KERNEL_SOURCE/patches/
cp $KERNEL_PATCH/patches_orin/5.0.2_agx_imx477_v0.1.patch $JETSON_SOURCES/
</pre>
</pre>
Where <code>[PLATFORM]</code> must be replaced by <code>nx</code> or <code>nano</code>.


====Apply the patches====
====Apply the patches====
<pre>
<pre>
cd $JETSON_KERNEL_SOURCE/patches
cd $JETSON_SOURCES
quilt push
git apply 5.0.2_agx_imx477_v0.1.patch
</pre>
</pre>


Line 364: Line 194:


====Install the Toolchain====
====Install the Toolchain====
First, download the [https://developer.nvidia.com/embedded/jetson-linux Bootlin toolchain binaries] from the NVIDIA Jetson Linux home page. Go to the bottom of the page and click on the Bootlin Toolchain gcc option as shown in the image below:
<br>
[[File:Toolchain.png|thumb|center|650px]]


NVIDIA recommends using the Linaro 7.3.1 2018.05 toolchain. Download the pre-built toolchain binaries as follows:
After that do:
<pre>
<syntaxhighlight lang=bash>
wget http://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
mkdir $HOME/l4t-gcc
</pre>
 
Execute the following commands to extract the toolchain:
<pre>
mkdir -p $HOME/l4t-gcc
cd $HOME/l4t-gcc
cd $HOME/l4t-gcc
tar xf gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
tar xf <path_to_toolchain_archive>
</pre>
</syntaxhighlight>


====Build the kernel====
====Build the kernel====


* Create the main directories:
* Establish the building paths and create directories:
<pre>
cd $JETSON_KERNEL_SOURCE/../
mkdir -p modules/
mkdir -p packages/
mkdir -p dtb/
</pre>
 
* Establish the building paths:
<pre>
<pre>
CROSS_COMPILE=$HOME/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
KERNEL_OUT=$JETSON_SOURCES/kernel_out
KERNEL_OUT=$JETSON_KERNEL_SOURCE/../build
MODULES_OUT=$JETSON_SOURCES/modules_out
KERNEL_MODULES_OUT=$JETSON_KERNEL_SOURCE/../modules
mkdir -p $KERNEL_OUT
mkdir -p $MODULES_OUT
</pre>
</pre>


Line 399: Line 220:
</pre>
</pre>


* Configure and build kernel:
* Export environment variables:
<pre>
<pre>
cd $JETSON_KERNEL_SOURCE
export CROSS_COMPILE_AARCH64_PATH=$HOME/l4t-gcc
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT tegra_defconfig
export CROSS_COMPILE_AARCH64=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT menuconfig
</pre>
</pre>


* Select:
* Compile kernel:
<pre>
<pre>
Device Drivers  --->
cd $JETSON_SOURCES
  <*> Multimedia support  --->
./nvbuild.sh -o $KERNEL_OUT
      NVIDIA overlay Encoders, decoders, sensors and other helper chips  --->
          <*> IMX477 camera sensor support
</pre>
</pre>
Press the ''Y'' key in order to select the IMX477 option.
Go back by hitting the double ''Esc'' key until you get the message: ''Do you want to save your new configuration?'', select ''Yes'' and press ''Enter''.
* Compile kernel, device tree, and modules:


*Install the modules
<pre>
<pre>
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 Image
make -C $JETSON_SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra INSTALL_MOD_PATH=$MODULES_OUT modules_install
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 dtbs
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 modules
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT modules_install INSTALL_MOD_PATH=$KERNEL_MODULES_OUT
</pre>
</pre>


===Flash the Jetson===
===Flash the Jetson===


====Flash the Xavier NX====
We present two methods to install the customized kernel, one requires a complete flash, and the other is for quick updating while developing.


Please, make sure the board is in [[Raspberry_Pi_HQ_camera_IMX477_Linux_driver_for_Jetson#Recovery_Mode_for_Jetson_NX|Recovery Mode]].
==== Complete flash ====


* Copy the generated kernel and dtb:
'''Update the Linux_for_Tegra directory'''


<pre>
<syntaxhighlight lang=bash>
cd $JETSON_KERNEL_SOURCE/../
# Copy the new compiled kernel.
cp build/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb ./kernel/dtb
cp -r $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/
cp build/arch/arm64/boot/Image ./kernel
</pre>


* Flash the dtb:
# Copy all new compiled DTBs and DTB overlays.
<pre>
cp -r $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/
sudo ./flash.sh -r -k kernel-dtb -d kernel/dtb/tegra194-p3668-all-p3509-0000.dtb jetson-xavier-nx-devkit mmcblk0p1
</pre>


* Copy the kernel image:
# Update rootfs with compiled modules, which include the driver.
<pre>
rsync -azPu $MODULES_OUT/lib/modules/5.10.104-tegra $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/
scp ./kernel/Image <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/
</pre>


* Move the Image from <code>/tmp</code> to <code>/boot</code> inside the Xavier NX's filesystem:
# Update the rootfs with all the binaries (to make sure the /boot/ folder is updated too).
<pre>
cd $JETPACK_DIR/Linux_for_Tegra
sudo mv /tmp/Image /boot/
sudo ./apply_binaries.sh
</pre>


====Flash the Nano====
# Copy the module to the correct location.
<br>
sudo cp $KERNEL_OUT/drivers/media/i2c/nv_imx477.ko $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers/media/i2c/
{{Ambox
</syntaxhighlight>
|type=notice
|small=left
|issue='''The DTB depends on the Nano revision:'''
'''For Jetson Nano B01:''' replace <dtb> with tegra210-p3448-0000-p3449-0000-b00.dtb
 
'''For Jetson Nano A02:''' replace <dtb> with tegra210-p3448-0000-p3449-0000-a02.dtb
|style=width:unset;
}}
<br>
* Make sure your extlinux.conf file at '''/boot/extlinux/extlinux.conf''' in your Nano's filesystem looks as follows by adding the FDT line (remember to change the <dtb> depending on your board).


<pre>
'''Flash'''
TIMEOUT 30
DEFAULT primary


MENU TITLE L4T boot options
Finally, flash the driver
<syntaxhighlight lang=bash>
sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1
</syntaxhighlight>


LABEL primary
==== Manual installation ====
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      FDT /boot/<dtb>
      APPEND ${cbootargs} quiet
</pre>


* Copy the kernel image and DTB:
Update the kernel and DTB files used by /boot/extlinux/extlinux.conf when there is an already running board.
This assumes that the Jetson can be reach by SSH using:


<pre>
<pre>
cd $JETSON_KERNEL_SOURCE/../
ssh jetson
scp build/arch/arm64/boot/Image <nvidia-nano-user>@<nvidia-nano-ip>:/tmp/
scp build/arch/arm64/boot/dts/<dtb> <nvidia-nano-user>@<nvidia-nano-ip>:/tmp/
</pre>
</pre>


* Move the files from <code>/tmp</code> to <code>/boot</code> inside the Nano's filesystem:
'''Updating the kernel'''
 
Updating the kernel is as simple as copying into the board:


<pre>
<syntaxhighlight lang=bash>
sudo mv /tmp/Image /boot/
# Update Image on Linux_for_Tegra folder just to sync the flash folders
sudo mv /tmp/<dtb> /boot/
sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/
</pre>


===Copy the kernel modules===
# Copy the Image to the tmp folder on the board
scp $JETPACK_DIR/Linux_for_Tegra/kernel/Image jetson:/tmp/


* You can just access your SD card in your PC and copy the '''KERNEL_MODULES=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.201/*''' directory content into '''/lib/modules/4.9.201/''' in your SD card to update the modules. Where <code>[PLATFORM]</code> must be replaced by <code>XAVIER_NX</code> or <code>NANO</code>, and <code>[JETPACK_VERSION]</code> must be replaced by <code>4.4</code>, <code>4.4.1</code> or <code>4.5</code>.
# Copy the Image to the boot directory on the board
sudo mv /tmp/Image /boot/Image
</syntaxhighlight>
'''Updating the DTB'''


* Or you can update with your compiled modules  If your board is up you can update the kernel modules as follows:
The new JetPack uses the FDT property of the /boot/extlinux/extlinux.conf file, and it sets it to a device tree located in /boot/dtb/ with a prefix of kernel_tegra. We can update this DTB with the following:
<syntaxhighlight lang=bash>
# Update on Linux_for_Tegra folder with the new DTBs and overlays
sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/


<pre>
# Generate required prefixed DTB using flash script with no flash option
KERNEL_MODULES=$JETPACK_DIR/JetPack_[JETPACK_VERSION]_Linux_JETSON_[PLATFORM]_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.140
cd  $JETPACK_DIR/Linux_for_Tegra
cd $KERNEL_MODULES
sudo ./flash.sh -r --no-flash jetson-agx-orin-devkit mmcblk0p1
# use rsync instead of scp to avoid symlink recursion problems
rsync --progress -avhe ssh ../4.9.201 <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/
</pre>


Where <code>[PLATFORM]</code> must be replaced by <code>XAVIER_NX</code> or <code>NANO</code>, and <code>[JETPACK_VERSION]</code> must be replaced by <code>4.4</code>, <code>4.4.1</code> or <code>4.5</code>.
# Copy the device tree to the board
scp $JETPACK_DIR/Linux_for_Tegra/bootloader/kernel_tegra234-p3701-0000-p3737-0000.dtb jetson:/tmp/


# Update the device tree on the board
sudo mv /tmp/kernel_tegra234-p3701-0000-p3737-0000.dtb /boot/dtb/
</syntaxhighlight>


* Move the modules from <code>/tmp</code> to the <code>/lib/modules/</code> folder inside the Jetson's filesystem and reboot:
==== Updating modules ====


<pre>
Inside the board run the following to update the modules folder, copying them from the host:
sudo mv /tmp/4.9.201/ /lib/modules/
sudo reboot
</pre>


* If you find that the directory name is '''/lib/modules/4.9.201-tegra/''', change the directory name to '''/lib/modules/4.9.201/'''
<syntaxhighlight lang=bash>
* After these steps, the driver should be working and you can run some capture tests described below.
HOST_NAME=host
* '''IMPORTANT''': You may see some washed-out colors in the captured image. This is because you haven't applied the ISP configuration file. Make sure to install the ISP file that you can find in the repo using [https://developer.ridgerun.com/wiki/index.php?title=Raspberry_Pi_HQ_camera_IMX477_Linux_driver_for_Jetson#ISP_camera_overrides_installation these instructions].
HOST_IP=192.168.55.100
cd /lib/modules/
sudo rsync -azPu $HOST_NAME@$HOST_IP:$MODULES_OUT/lib/modules/5.10.104-tegra .
</syntaxhighlight>


==ISP camera overrides installation==
==ISP camera overrides installation==


Please, visit the [https://developer.ridgerun.com/wiki/index.php?title=JetsonTX2/HowTo%27s/NVIDIA_Jetson_ISP_Control#Custom_ISP_Configuration Custom ISP Configuration page] in order to learn how to use the <code>camera_overrides.isp</code> file to configure the ISP.
Please, visit the [[JetsonTX2/HowTo%27s/NVIDIA_Jetson_ISP_Control#Custom_ISP_Configuration | Custom ISP Configuration page]] in order to learn how to use the <code>camera_overrides.isp</code> file to configure the ISP.
 
{{Ambox
|type=notice
|small=left
|issue='''NOTE:''' This is causing some timeout issues currently.
|style=width:unset;
}}


==Supported Features==
==Supported Features==
Line 533: Line 338:


* 1920x1080 @ 60fps
* 1920x1080 @ 60fps
* 4032x3040 @ 30fps
* 3840x2160 @ 30fps


===Controls===
===Controls===
Line 550: Line 355:
====1920x1080====
====1920x1080====
<syntaxhighlight lang=bash>
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
SENSOR_ID=0 # 0 to 2 depending on cam port.
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink
</syntaxhighlight>
</syntaxhighlight>


====4032x3040====
====3840x2160====


<syntaxhighlight lang=bash>
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
SENSOR_ID=0 # 0 to 2 depending on cam port.
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink
</syntaxhighlight>
</syntaxhighlight>


Line 567: Line 372:
====1920x1080====
====1920x1080====
<syntaxhighlight lang=bash>
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
SENSOR_ID=0 # 0 to 2 depending on cam port.
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
</syntaxhighlight>
</syntaxhighlight>


====4032x3040====
====3840x2160====


<syntaxhighlight lang=bash>
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
SENSOR_ID=0 # 0 to 2 depending on cam port.
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
</syntaxhighlight>
</syntaxhighlight>


Line 584: Line 389:
====1920x1080====
====1920x1080====
<syntaxhighlight lang=bash>
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
SENSOR_ID=0 # 0 to 2 depending on cam port.
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
NUMBER_OF_SNAPSHOTS=20
NUMBER_OF_SNAPSHOTS=20
Line 590: Line 395:
</syntaxhighlight>
</syntaxhighlight>


====4032x3040====
====3840x2160====


<syntaxhighlight lang=bash>
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
SENSOR_ID=0 # 0 to 2 depending on cam port.
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
NUMBER_OF_SNAPSHOTS=20
NUMBER_OF_SNAPSHOTS=20
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
</syntaxhighlight>
</syntaxhighlight>


Line 606: Line 411:


<noinclude>
<noinclude>
{{NVIDIA Jetson Orin/Foot|Video Capture and Display/Cameras/IMX219|Video Capture and Display/Software Support}}
{{NVIDIA Jetson Orin/Foot|Video Capture and Display|GStreamer Pipelines}}
</noinclude>
</noinclude>

Latest revision as of 09:47, 11 February 2023


Previous: Video Capture and Display Index Next: GStreamer Pipelines






Problems running the pipelines shown on this page? Please see our GStreamer Debugging guide for help.

Introduction to Sony IMX477 Linux driver

RidgeRun is actively working to enable customers with great software solutions on top of powerful hardware. For this reason, RidgeRun has worked on enabling a triple capture for the V4L2 driver for the Raspberry Pi HQ camera using an Auvidea J20 board. This wiki contains a brief introduction to the sensor and hardware that will be used to use the driver. Also please keep in mind that the driver can be ported to other Jetson platforms if needed. Please contact us if you require more information.

Raspberry Pi HQ camera


 

Camera Features

  • 12.3-Megapixel high-resolution Sony IMX477 sensor.
  • 1.55μm × 1.55μm Pixel Size – double the pixel area of IMX219 for improved low-light performance.
  • Back-illuminated sensor architecture for improved sensitivity.
  • Support for different C- and CS-mount lenses.
  • Integrated back-focus adjustment ring and tripod mount.
 

Sony IMX477 Basic Drive Mode

The Raspberry Pi HQ camera uses the IMX477 CMOS sensor of Sony to capture. Such a sensor provides the modes included in the table below. At this point, RidgeRun is working to support one mode only but others can be added per request.

Drive Mode Number of active pixels Maximum frame rate
[frame/s]		 Output Interface ADC
[bit]
Full (4:3) (Normal) 4056 x 3040
approx. 12.33 M pixels 		60 CSI-2 10
40 CSI-2 12
Full (4:3) (DOL-HDR) 4056 x 3040
approx. 12.33 M pixels		 DOL 2 frame:30
DOL 3 frame:15		 CSI-2 10
Full (16:9) 4K2K (Normal) 4056 x 2288
approx. 9.28 M pixels		 79 CSI-2 10
Full (16:9) 4K2K (DOL-HDR) 4056 x 2288
approx. 9.28 M pixels		 DOL 2 frame:39
DOL 3 frame:19		 CSI-2 10
Full (4:3) Binning (Normal) 2028 x 1520
approx. 3.08 M pixels		 179 CSI-2 10
Full (16:9) Binning 1080P (Normal) 2028 x 1128
approx. 2.29 M pixels		 240 CSI-2 10
Full (16:9) Binning 720P (Normal) 1348 x 750
approx. 1.01 M pixels		 240 CSI-2 10
Full (16:9) Scaling 1080P (Normal) 2024 x 1142
approx. 2.31 M pixels		 79 CSI-2 10
Full (16:9) Scalling 720P (Normal) 1348 x 762
approx. 1.03 M pixels		 79 CSI-2 10

Compatibility with NVIDIA®Jetson™ Platforms

R8 resistor

The Raspberry Pi HQ camera module requires a hardware modification in order to work with Jetson Platforms. Jetson Platforms provide 1.8V for reset GPIO in the camera interface, but the camera module requires 3.3V.

To fix this issue a resistor labeled as R8 must be removed from the camera module. If you decide to apply this fix it is under your own risk. RidgeRun is not responsible for any damage caused to your board. The following picture shows the location of the R8 in the module.

 
Figure 1. R8 resistor on RPi HQ camera module.

Auvidea J20 adapter

The NVIDIA®Jetson Orin™ makes use of a Samtec connector, this driver makes use of the J20 from Auvidea More information recopilated from RidgeRun can be found here.

The driver will work with IMX477 connected to the J1_1, J1_2, and J1_3 ports of the J20.

IMX477 Linux driver for NVIDIA®Jetson Orin™

RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. and Auvidea. as a collaborative initiative. The driver works with three cameras on the J1 ports from the J20. In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These instructions have been validated in the following JetPack versions:

  • Jetpack 5.0.2

Download JetPack

The current version of the driver is supported in Jetpack 5.0.2. Porting the driver to other versions and platforms is possible.

1. Download and install the NVIDIA SDK Manager (membership is required).

2. Log in with your NVIDIA membership credentials in the SDK Manager.

3. Select JetPack version 5.0.2 DP and Jetson Orin as shown below and press Continue:

 
Figure 1. Required SDK Manager options.

4. Accept the terms and conditions and press Continue. The SDK Manager will download and install the selected components:

 
Figure 2. SDK Manager download and install software.

5. Select Manual Setup. Create a Make sure the board is in Recovery Mode and then press Flash:

 
Figure 3. Manual setup to start the flashing process.

Recovery Mode for Jetson Orin AGX

  1. Make sure the device is connected to the power adapter but powered off.
  2. Connect the host computer to the front USB Type-C connector on the device.
  3. Power the device
  4. Press and hold the middle (Force Recovery) button.
  5. Press and hold the left (Power) button.
  6. Release both buttons.

6. When the OS has been installed successfully in the Jetson Orin AGX, you have multiple options to use the board:

  1. Use the username and password previously selected to connect over SSH to the corresponding IP (normally 192.168.55.1 over the USB connection).
  2. Connect over the UART connection opened by attaching the micro-USB, for example, minicom: sudo minicom -D /dev/ttyUSB0 -b 115200 and use the same username and password
  3. Connect over display.



Installing the Driver - JetPack sources patch

Download the JetPack sources

You can obtain the JetPack 5.0.2 DP sources by doing the following:

1. Go to https://developer.nvidia.com/embedded/jetson-linux-archive and download the source files for your release.

2. Extract the .tbz2 file: 

tar -xjf public_sources.tbz2

3. Extract the kernel source file: 

cd Linux_for_Tegra/source/public
JETSON_SOURCES=$PWD
tar –xjf kernel_src.tbz2

This extracts the kernel source to the kernel/ subdirectory.

Patch instructions

Download the patches

RidgeRun has created a repository from where you can download the patch files.

You need to clone it and use the branch for the jetpack. 

cd $HOME
git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git -b jetpack-5.0.2

Once you have downloaded the repository, copy the files to the $JETSON_SOURCES directory: 

KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/
cp $KERNEL_PATCH/patches_orin/5.0.2_agx_imx477_v0.1.patch $JETSON_SOURCES/

Apply the patches

cd $JETSON_SOURCES
git apply 5.0.2_agx_imx477_v0.1.patch

Kernel build instructions

Once the sources have been patched, perform the following steps in order to build and install the driver.

Install the Toolchain

First, download the Bootlin toolchain binaries from the NVIDIA Jetson Linux home page. Go to the bottom of the page and click on the Bootlin Toolchain gcc option as shown in the image below:

 

After that do: 

mkdir $HOME/l4t-gcc
cd $HOME/l4t-gcc
tar xf <path_to_toolchain_archive>

Build the kernel

  • Establish the building paths and create directories:
KERNEL_OUT=$JETSON_SOURCES/kernel_out
MODULES_OUT=$JETSON_SOURCES/modules_out
mkdir -p $KERNEL_OUT
mkdir -p $MODULES_OUT
  • Use the following command to install dependencies on Ubuntu:
sudo apt install libncurses-dev
  • Export environment variables:
export CROSS_COMPILE_AARCH64_PATH=$HOME/l4t-gcc
export CROSS_COMPILE_AARCH64=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-
  • Compile kernel:
cd $JETSON_SOURCES
./nvbuild.sh -o $KERNEL_OUT
  • Install the modules
make -C $JETSON_SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra INSTALL_MOD_PATH=$MODULES_OUT modules_install

Flash the Jetson

We present two methods to install the customized kernel, one requires a complete flash, and the other is for quick updating while developing.

Complete flash

Update the Linux_for_Tegra directory 

# Copy the new compiled kernel.
cp -r $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/

# Copy all new compiled DTBs and DTB overlays. 
cp -r $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/

# Update rootfs with compiled modules, which include the driver.
rsync -azPu $MODULES_OUT/lib/modules/5.10.104-tegra $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/

# Update the rootfs with all the binaries (to make sure the /boot/ folder is updated too).
cd $JETPACK_DIR/Linux_for_Tegra
sudo ./apply_binaries.sh

# Copy the module to the correct location.
sudo cp $KERNEL_OUT/drivers/media/i2c/nv_imx477.ko $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers/media/i2c/

Flash

Finally, flash the driver 

sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1

Manual installation

Update the kernel and DTB files used by /boot/extlinux/extlinux.conf when there is an already running board. This assumes that the Jetson can be reach by SSH using: 

ssh jetson

Updating the kernel

Updating the kernel is as simple as copying into the board: 

# Update Image on Linux_for_Tegra folder just to sync the flash folders
sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/

# Copy the Image to the tmp folder on the board
scp $JETPACK_DIR/Linux_for_Tegra/kernel/Image jetson:/tmp/

# Copy the Image to the boot directory on the board
sudo mv /tmp/Image /boot/Image

Updating the DTB

The new JetPack uses the FDT property of the /boot/extlinux/extlinux.conf file, and it sets it to a device tree located in /boot/dtb/ with a prefix of kernel_tegra. We can update this DTB with the following: 

# Update on Linux_for_Tegra folder with the new DTBs and overlays
sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/

# Generate required prefixed DTB using flash script with no flash option
cd  $JETPACK_DIR/Linux_for_Tegra
sudo ./flash.sh -r --no-flash jetson-agx-orin-devkit mmcblk0p1

# Copy the device tree to the board
scp $JETPACK_DIR/Linux_for_Tegra/bootloader/kernel_tegra234-p3701-0000-p3737-0000.dtb jetson:/tmp/

# Update the device tree on the board
sudo mv /tmp/kernel_tegra234-p3701-0000-p3737-0000.dtb /boot/dtb/

Updating modules

Inside the board run the following to update the modules folder, copying them from the host: 

HOST_NAME=host
HOST_IP=192.168.55.100
cd /lib/modules/
sudo rsync -azPu $HOST_NAME@$HOST_IP:$MODULES_OUT/lib/modules/5.10.104-tegra .

ISP camera overrides installation

Please, visit the Custom ISP Configuration page in order to learn how to use the camera_overrides.isp file to configure the ISP.

Supported Features

Resolutions and framerates

  • 1920x1080 @ 60fps
  • 3840x2160 @ 30fps

Controls

  • Gain
  • Exposure
  • Framerate
  • Group Hold

Example Pipelines

Find some example pipelines to use the IMX477 on Jetson Xavier NX below:

Display

1920x1080

SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink

3840x2160

SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink

MP4 Recording

1920x1080

SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4

3840x2160

SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4

JPEG snapshots

1920x1080

SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
NUMBER_OF_SNAPSHOTS=20
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg

3840x2160

SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
NUMBER_OF_SNAPSHOTS=20
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg


RidgeRun Resources

Quick Start Client Engagement Process RidgeRun Blog Homepage
Technical and Sales Support RidgeRun Online Store RidgeRun Videos Contact Us
RidgeRun.ai: Artificial Intelligence | Generative AI | Machine Learning

  Contact Us

Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering information is available at RidgeRun Engineering Services, RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page.  
 




Previous: Video Capture and Display Index Next: GStreamer Pipelines



Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=NVIDIA_Jetson_Orin/Video_Capture_and_Display/Cameras/IMX477_J20&oldid=46365"