Raspberry Pi HQ camera IMX477 Linux driver for Jetson
|
|
Introduction to Raspberry Pi HQ camera Sony IMX477 Linux driver
RidgeRun is actively working to enable customers with great software solutions on top of powerful hardware. For this reason RidgeRun is currently working to create the V4L2 driver for the Raspberry Pi HQ camera. This wiki contains a brief introduction to the sensor and hardware that will be used to develop the driver. Please keep in mind that the driver can be ported to other Jetson platforms if needed. Please contact us if you require more information.
Raspberry Pi HQ camera
Camera Features
- 12.3-Megapixel high-resolution Sony IMX477 sensor.
- 1.55μm × 1.55μm Pixel Size – double the pixel area of IMX219 for improved low-light performance.
- Back-illuminated sensor architecture for improved sensitivity.
- Support for different C- and CS-mount lenses.
- Integrated back-focus adjustment ring and tripod mount.
Sony IMX477 Basic Drive Mode
The Raspberry Pi HQ camera uses the IMX477 CMOS sensor of Sony to capture. Such a sensor provides the modes included in the table below. At this point RidgeRun is working to support one mode only but others can be added per request.
Drive Mode | Number of active pixels | Maximum frame rate [frame/s] |
Output Interface | ADC [bit] |
---|---|---|---|---|
Full (4:3) (Normal) | 4056 x 3040 approx. 12.33 M pixels |
60 | CSI-2 | 10 |
40 | CSI-2 | 12 | ||
Full (4:3) (DOL-HDR) | 4056 x 3040 approx. 12.33 M pixels |
DOL 2 frame:30 DOL 3 frame:15 |
CSI-2 | 10 |
Full (16:9) 4K2K (Normal) | 4056 x 2288 approx. 9.28 M pixels |
79 | CSI-2 | 10 |
Full (16:9) 4K2K (DOL-HDR) | 4056 x 2288 approx. 9.28 M pixels |
DOL 2 frame:39 DOL 3 frame:19 |
CSI-2 | 10 |
Full (4:3) Binning (Normal) | 2028 x 1520 approx. 3.08 M pixels |
179 | CSI-2 | 10 |
Full (16:9) Binning 1080P (Normal) | 2028 x 1128 approx. 2.29 M pixels |
240 | CSI-2 | 10 |
Full (16:9) Binning 720P (Normal) | 1348 x 750 approx. 1.01 M pixels |
240 | CSI-2 | 10 |
Full (16:9) Scaling 1080P (Normal) | 2024 x 1142 approx. 2.31 M pixels |
79 | CSI-2 | 10 |
Full (16:9) Scalling 720P (Normal) | 1348 x 762 approx. 1.03 M pixels |
79 | CSI-2 | 10 |
Compatibility with NVIDIA®Jetson™ Platforms
The Raspberry Pi HQ camera module requires a hardware modification in order to work with Jetson Platforms. Both Jetson Nano and Jetson Xavier NX provides 1.8V for reset GPIO in the camera interface, but the camera module requires 3.3V.
To fix this issue a resistor labeled as R8 must be removed from the camera module. If you decide to apply this fix it is under your own risk. RidgeRun is not responsible for any damage caused to your board. The following picture shows the location of the R8 in the module.
Raspberry Pi HQ camera IMX477 Linux driver for NVIDIA®Jetson Xavier™ NX
RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. as a collaborative initiative. In this section you will find the instructions to patch the JetPack 4.4 sources enabling the IMX477 sensor driver for the Jetson Xavier NX.
Download the JetPack 4.4
The current version of the driver is supported in Jetpack 4.4 production and Jetpack 4.4 developer preview (DP). Porting the driver to other versions and platforms is possible.
1. Download and install the NVIDIA SDK Manager (membership is required).
2. Put your NVIDIA membership credentials in the SDK Manager main window:
3. Select JetPack 4.4 and Jetson Xavier NX as shown below and press Continue:
4. Accept the terms and conditions and press Continue. The SDK Manager will download and install the selected components:
5. Select Manual Setup. Make sure the board is in Recovery Mode and then press Flash:
Recovery Mode for Jetson NX
- NOTE: To set the Xavier NX developer kit in Recovery Mode the following steps are required:
- Ensure the device is powered off and the power adapter is disconnected.
- Verify that a microSD Card is inserted in the Jetson Xavier NX module's card slot.
- Place a jumper across the Force Recovery Mode pins (9 and 10) on the button header [J14]. Please, see Figure 6 below.
- Connect your host computer to the developer kit's USB Micro-B connector.
- Connect the power supply to the power jack [J16]. The developer kit automatically powers on in Force Recovery Mode.
- Remove the jumper from the Force Recovery Mode pins.
- Use the
lsusb
command to check if theNVidia corp
message is shown.
6. When the OS has been installed successfully in the Xavier NX, you will be asked to enter the board IP, a username and a password, as shown below in Figure7:
7. Don't close the window of Figure 7 and don't modify its content yet. Yo need to connect a screen to the board via HDMI, a keyboard/mouse via USB port, and a network cable via Ethernet port. You can remove the jumper that keeps the board in Recovery Mode and then reboot the Xavier board at this point if necessary.
8. After reboot you will be asked to accept the licences terms in Xavier NX's screen. Select the checkbox and press Continue.
9. Select you language in the next screen and press Continue:
10. Select you keyboard layout and press Continue:
11. Select you location and press Continue:
12. Enter a username and a password. Then, press Continue:
13. Wait for the system to apply the changes. The board will restart automatically.
14. Now you are able to access Ubuntu as a new user in the Xavier NX. You can find the board IP by using ifconfig
from a terminal.
15. Get back to the window presented in Figure 7 (in your host PC) and enter the IP, username and password which the Xavier NX has been configured with. Then press Install.
16. The software installation process shoud finish successfully, as shown in Figure 13:
- IMPORTANT: From now on we will refer as
$JETPACK_DIR/
to the directory where the SDK Manager placed theJetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT
folder.
Getting the kernel patch
RidgeRun has created a repository from where you can download the patch files with the following structure:
. ├── li-camera-calibration-files │ └── camera_overrides.isp ├── patches │ ├── 4.4.2_nx_imx477.patch │ └── series └── README.md
Download the JetPack 4.4 sources
The sources can be downloaded from the NVIDIA's Download Center:
1. You must be able to download a file called public_sources.tbz2
.
2. Untar the sources
tar -xvf public_sources.tbz2
3. Move the hardware/
and kernel/
folders into Linux_for_Tegra/
inside the JetPack directory $JETPACK_DIR/
:
cd Linux_for_Tegra/source/public/ tar -xvf kernel_src.tbz2 JETSON_KERNEL_SOURCE=$JETPACK_DIR/JetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT/Linux_for_Tegra/source/ mkdir -p $JETSON_KERNEL_SOURCE mv hardware/ kernel/ $JETSON_KERNEL_SOURCE
Patch instructions
Download the patches
cd $HOME git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git KERNEL_PATCH=$HOME/NVIDIA-Jetson-IMX477-RPIV3/ cp -r $KERNEL_PATCH/patches/ $JETSON_KERNEL_SOURCE
Apply the patches
cd $JETSON_KERNEL_SOURCE quilt push
Kernel build instructions
Once the sources have been patched, perform the following steps in order to build and install the driver.
Install the Toolchain
NVIDIA recommends using the Linaro 7.3.1 2018.05 toolchain. Download the pre-built toolchain binaries as follows:
wget http://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
Execute the following commands to extract the toolchain:
mkdir -p $HOME/l4t-gcc cd $HOME/l4t-gcc tar xf gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
Build the kernel
- Create the main directories:
cd $JETSON_KERNEL_SOURCE/../ mkdir -p modules/ mkdir -p packages/ mkdir -p dtb/
- Establish the building paths:
CROSS_COMPILE=$HOME/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- KERNEL_OUT=$JETSON_KERNEL_SOURCE/../build KERNEL_MODULES_OUT=$JETSON_KERNEL_SOURCE/../modules
- Configure and build kernel:
cd $JETSON_KERNEL_SOURCE make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT tegra_defconfig make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT menuconfig
- Select:
Device Drivers ---> <*> Multimedia support ---> NVIDIA overlay Encoders, decoders, sensors and other helper chips ---> <*> IMX477 camera sensor support
Press the Y key in order to select the IMX477 option. Go back by hitting double Esc key until you get the message: Do you want to save your new configuration?, select Yes and press Enter.
- Compile kernel, device tree and modules:
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 Image make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 dtbs make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=${CROSS_COMPILE} -j4 modules make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT modules_install INSTALL_MOD_PATH=$KERNEL_MODULES_OUT
- Copy the generated kernel and dtb:
cd $JETSON_KERNEL_SOURCE/../ cp build/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb ./kernel/dtb cp build/arch/arm64/boot/Image ./kernel
Flash the Jetson
Flash the Xavier NX
Please, make sure the board is in Recovery Mode.
- Flash the dtb:
sudo ./flash.sh -r -k kernel-dtb -d kernel/dtb/tegra194-p3668-all-p3509-0000.dtb jetson-xavier-nx-devkit mmcblk0p1
- Copy the kernel image:
scp ./kernel/Image <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/
- Move the Image from
/tmp
to/boot
inside the Xavier NX's filesystem:
sudo mv /tmp/Image /boot/
Flash the Nano
- Make sure your extlinux.conf file in /boot/extlinux.conf looks like the following:
TIMEOUT 30 DEFAULT primary MENU TITLE L4T boot options LABEL primary MENU LABEL primary kernel LINUX /boot/Image INITRD /boot/initrd FDT /boot/tegra210-p3448-0000-p3449-0000-b00.dtb APPEND ${cbootargs} quiet
- Copy the kernel image and DTB:
scp ./kernel/Image ./kernel/dtb/tegra210-p3448-0000-p3449-0000-b00.dtb <nvidia-nano-user>@<nvidia-nano-ip>:/tmp/
- Move the files from
/tmp
to/boot
inside the Nano's filesystem:
sudo mv /tmp/Image /boot/ sudo mv /tmp/tegra210-p3448-0000-p3449-0000-b00.dtb /boot/
Copy the kernel modules
- You can also just access your SD card in your PC and copy the KERNEL_MODULES=$JETPACK_DIR/JetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.140-tegra/* directory content into /lib/modules/4.9.140/ in your SD card to update the modules.
- Or you can update with your compiled modules . If your board is up you can update the kernel modules as follows:
KERNEL_MODULES=$JETPACK_DIR/JetPack_4.4_DP_Linux_DP_JETSON_XAVIER_NX_DEVKIT/Linux_for_Tegra/modules/lib/modules/4.9.140 cd $KERNEL_MODULES scp -r ../4.9.140/ <nvidia-nx-user>@<nvidia-nx-ip>:/tmp/
- Move the modules from
/tmp
to the/lib/modules/
folder inside the Jetson's filesystem:
sudo cp -rf /tmp/4.9.140/ /lib/modules/
ISP camera overrides installation
Please, visit the Custom ISP Configuration page in order to learn how to use the camera_overrides.isp
file to configure the ISP.
Supported Features
Resolutions and framerates
- 1920x1080 @ 60fps
- 4032x3040 @ 30fps
Controls
- Gain
- Exposure
- Framerate
- Group Hold
Example Pipelines
Find some example pipelines to use the IMX477 on Jetson Xavier NX below:
Display
1920x1080
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink
4032x3040
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink
MP4 Recording
1920x1080
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
4032x3040
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
JPEG snapshots
1920x1080
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
4032x3040
SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode NUMBER_OF_SNAPSHOTS=20 gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
RidgeRun Resources | |||||
Contact Us
|