Compiling Jetson TX1/TX2 source code
Introduction
In this wiki page you are going to find the instructions to download the source code to rebuild the tegra X1 images using jetpack, several parts of this wiki were based in the document called Start_L4T_Docs.html given by Nvidia in L4T. This page should be considered a work in progress
Platform
Baseboard: P2597 Jetson module: P2180
Downloading the code
File called Start_L4T_Docs.html inside of Jetpack contains a good description of the packages that it contains. Let's call $DEVDIR the path where you installed your development directory or Jetpack.
1) In order to download the source code you can run the script called source_sync.sh
$DEVDIR/TX1/Linux_for_Tegra_tx1$ ./source_sync.sh
This will download the bootloader and kernel.
2) It downloads all the kernel tree but you need to specify the tag, looking into the script it seems that the tags should be specified with k and u parameters
Use: source_sync.sh [options] Available general options are, -h : help -e : exit on sync error -d DIR : root of source is DIR By default, all sources are downloaded. Only specified sources are downloaded, if one or more of the following options are mentioned. -k [TAG]: Download kernel source and optionally sync to TAG -u [TAG]: Download u-boot source and optionally sync to TAG
Repositories
k:kernel:nv-tegra.nvidia.com/linux-3.10.git: u:u-boot:nv-tegra.nvidia.com/3rdparty/u-boot.git:
you can see the repositories on the web looking for:
http://nv-tegra.nvidia.com/gitweb/?o=age http://nv-tegra.nvidia.com/gitweb/?p=linux-3.10.git;a=summary http://nv-tegra.nvidia.com/gitweb/?p=3rdparty/u-boot.git;a=summary
3) It will download the git repositories for kernel and uboot but you need to specify the branch or the tag
Branch:
cd $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source git branch -a git checkout l4t/l4t-r23.2 cd $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/u-boot_source git branch -a git checkout l4t/l4t-r23.2
Or if you want to download a tag then the command should look like:
./source_sync.sh -k tegra-l4t-r23.2 -u tegra-l4t-r23.2
Toolchain
Linaro or code sourcery toolchain's can be used, however, it is recommended to use the Linaro toolchain because it is newer and produces more optimized assembler code
1. Download the Linaro toolchain. You need to install two toolchains, one which is the 64bits toolchain for ARM, and the 32bits toolchain for ARM as well. In this case version 5.3-2016.02 will be used:
2. Install the toolchain
sudo mkdir /opt/linaro sudo chmod -R 775 /opt/linaro sudo chown -R $USER /opt/linaro cp gcc-linaro-5.3-2016.02-x86_64_aarch64-linux-gnu.tar.xz /opt/linaro cp gcc-linaro-5.3-2016.02-x86_64_arm-linux-gnueabihf.tar.xz /opt/linaro cd /opt/linaro/ tar -xf gcc-linaro-5.3-2016.02-x86_64_aarch64-linux-gnu.tar.xz tar -xf gcc-linaro-5.3-2016.02-x86_64_arm-linux-gnueabihf.tar.xz
Bootloader
The bootloader is a combination of NVIDIA T-Boot (nvtboot) and U-Boot. You can read about the BOOT FLOW in the nvtboot wiki page.
According to the devtalk forum there are two ways to boot the board: fast boot and uboot
- Fastboot is simpler, and offers no options during boot. It strictly looks for a kernel in the GPT partition from the install's "-k 6" option. This would probably be the way to go for a non-interactive embedded appliance (if not bare metal booting).
- U-boot offers more options, and looks for a zImage in the /boot partition. Boot configuration can also be edited here, for example I threw in a couple of zImage variants (like debugging version and network options added), and edited extlinux.conf...and magically I can boot to any of those kernels at the serial console boot prompt (provided my eye is fast). No flash was required for any kernel beyond the first kernel.
Inside of the bootloader directory you will find several binaries or components, to get a description of these please read the nvtboot wiki page.
Kernel
In order to compile the kernel please follow these setps:
1) Specify the path to the toolchain:
mkdir -p $DEVDIR/images/modules mkdir -p $DEVDIR/images/packages export CROSS_COMPILE=/opt/linaro/gcc-linaro-5.3-2016.02-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu- export CROSS32CC=/opt/linaro/gcc-linaro-5.3-2016.02-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc export KERNEL_MODULES_OUT=$DEVDIR/images/modules export ARCH=arm64
2) Clean your kernel and configuration
cd $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source/ make mrproper
3) Apply following change to avoid error: r7 cannot be used in asm here
Index: kernel_source/arch/arm64/kernel/vdso32/Makefile =================================================================== --- kernel_source.orig/arch/arm64/kernel/vdso32/Makefile 2016-04-08 21:28:52.651992663 -0600 +++ kernel_source/arch/arm64/kernel/vdso32/Makefile 2016-04-11 12:20:03.377388110 -0600 @@ -11,7 +11,7 @@ GCOV_PROFILE := n -ccflags-y := -shared -fPIC -fno-common -fno-builtin -march=armv7-a +ccflags-y := -shared -fPIC -fomit-frame-pointer -fno-common -fno-builtin -march=armv7-a ccflags-y += -nostdlib -Wl,-soname=linux-vdso32.so.1 \ $(call cc-ldoption, -Wl$(comma)--hash-style=sysv) asflags-y := -D__VDSO32__ -s
4) Apply following change to avoid error: logical not is only applied to the left hand side of comparison
Index: kernel_source/drivers/platform/tegra/tegra21_clocks.c =================================================================== --- kernel_source.orig/drivers/platform/tegra/tegra21_clocks.c 2016-04-08 21:28:58.755992489 -0600 +++ kernel_source/drivers/platform/tegra/tegra21_clocks.c 2016-04-11 12:49:59.561337028 -0600 @@ -1062,7 +1062,7 @@ */ static void tegra21_cpu_clk_init(struct clk *c) { - c->state = (!is_lp_cluster() == (c->u.cpu.mode == MODE_G)) ? ON : OFF; + c->state = ((!is_lp_cluster()) == (c->u.cpu.mode == MODE_G)) ? ON : OFF; } static int tegra21_cpu_clk_enable(struct clk *c)
5) Configure your kernel
make tegra21_defconfig make menuconfig
If you want to compare the configuration of your current kernel vrs the kernel that you are going to build (tegra21_defconfig) you can run:
zcat /proc/config.gz > config #tegra meld config tegra21_defconfig #host
6) Compile kernel, device tree and modules
make zImage make dtbs make modules make modules_install INSTALL_MOD_PATH=$KERNEL_MODULES_OUT
7) Save your binaries in the images directory
cp arch/arm64/boot/Image arch/arm64/boot/zImage $DEVDIR/images/
8) Create kernel_supplements.tbz2 with kernel modules, this package is needed by Jetpack
cd $DEVDIR/images/modules tar -cjf kernel_supplements.tbz2 * mv kernel_supplements.tbz2 $DEVDIR/images/packages
9) Besides creating the modules and the kernel image you need to create again your kernel_headers.tbz2 file (needed by Jetpack). By default when you patch the kernel code and you don't check your changes in the kernel will add a -dirty suffix to the release version (check ls $DEVDIR/images/modules/lib/modules/ for example). For this specific reason you need to generate your headers tarball again changing it the release version.
cd $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel tar -xf kernel_headers.tbz2 mv linux-headers-3.10.67-g458d45c linux-headers-3.10.67-g458d45c-dirty tar -cjf kernel_headers_custom.tbz2 linux-headers-3.10.67-g458d45c-dirty mv kernel_headers_custom.tbz2 $DEVDIR/images/packages rm -rf linux-headers-3.10.67-g458d45c-dirty
If you want to do this you can run the following before compiling the kernel:
echo "-gbaaa7aa"> .scmversion
10) Create the dtb directory
mkdir -p $DEVDIR/images/dtb cp $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source/arch/arm64/boot/dts/*.dtb $DEVDIR/images/dtb/
11) Copy the Device Tree Compiler (DTC)
cp $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source/scripts/dtc/dtc $DEVDIR/images/dtc
12) Create a backup of the images and packages included in Jetpack
mkdir -p $DEVDIR/images/packages-backup cp -rf $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/* $DEVDIR/images/packages-backup
13) Finally overwrite the default images with your own images to install and boot them using jetpack
cd $DEVDIR/images rm -rf $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/dtb cp -rf Image zImage packages/kernel_headers_custom.tbz2 packages/kernel_supplements.tbz2 dtb/ dtc $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/ cp -rf packages/kernel_headers_custom.tbz2 $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/kernel_headers.tbz2
14) Run the applybinaries script to generate the image to flash
cd $DEVDIR/TX1/Linux_for_Tegra_tx1/ sudo./apply_binaries.sh
Output:
Using rootfs directory of: /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs Extracting the NVIDIA user space components to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs Extracting the BSP test tools to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs Extracting the NVIDIA gst test applications to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs Extracting the configuration files for the supplied root filesystem to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs Creating a symbolic link nvgstplayer pointing to nvgstplayer-0.10 Creating a symbolic link nvgstcapture pointing to nvgstcapture-0.10 Adding symlink libcuda.so --> libcuda.so.1.1 in target rootfs Adding symlink libGL.so --> libGL.so.1 in target rootfs Adding symlink libcuda.so --> tegra/libcuda.so in target rootfs Adding symlink libEGL.so --> libEGL.so.1 in target rootfs Extracting the firmwares and kernel modules to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs Extracting the kernel headers to /usr/src in target rootfs Installing zImage into /boot in target rootfs Installing Image into /boot in target rootfs Installing the board *.dtb files into /boot in target rootfs Success!
Flash your image with Jetpack
You can flash your image with Jetpack now, just run Jetpack as the first time that you install it:
./JetPack-L4T-2.1-linux-x64.run
It will notice that everything is already build and will install your new images. If you have problems detecting the IP address wait like 2 minutes and it will give you the option to enter it manually. Username and password: ubuntu. You can see the ipaddress connecting a keyboard, mouse and monitor to the board at this point since ubuntu would be already booted.
DTB
Looking into the file $DEVDIR/TX1/Linux_for_Tegra_tx1/jetson-tx1.conf the dtb used by jetson x1 is: tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
ODMDATA=0x84000; NET_BSF=; EMMC_BSF=; EMMC_BCT=P2180_A00_LP4_DSC_204Mhz.cfg; EMMC_CFG=gnu_linux_tegraboot_emmc_full.xml; BOOTPARTSIZE=8388608; EMMCSIZE=31276924928; ITS_FILE=; SYSBOOTFILE=p2371-2180-devkit/extlinux.conf; DTB_FILE=tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb DFLT_KERNEL=Image; ROOTFSSIZE=14GiB; CMDLINE_ADD="fbcon=map:0"; UBOOT_TEXT_BASE=0x8010e000; UIMAGE_LABEL="Linux-tegra21"; target_board="t210ref"; ROOT_DEV="mmcblk0p1 ------------ internal eMMC. sda1 ----------------- external USB devices. (USB memory stick, HDD) eth0 ----------------- nfsroot via RJ45 Ethernet port. eth1 ----------------- nfsroot via USB Ethernet interface."; TEGRABOOT="bootloader/${target_board}/nvtboot.bin"; SOSFILE="bootloader/nvtboot_recovery.bin"; WB0BOOT="bootloader/${target_board}/warmboot.bin"; FLASHAPP="bootloader/tegraflash.py"; FLASHER="bootloader/${target_board}/cboot.bin"; BOOTLOADER="bootloader/${target_board}/p2371-2180/u-boot-dtb.bin"; UBOOT_WITH_TBOOT=yes; BCFFILE="bootloader/${target_board}/cfg/board_config_p2597-devkit.xml"; INITRD="bootloader/l4t_initrd.img"; TBCFILE="bootloader/nvtboot_cpu.bin"; BPFFILE="bootloader/bpmp.bin"; TOSFILE="bootloader/tos.img"; EKSFILE="bootloader/eks.img"; FBFILE="bootloader/reserved_fb.xml";
Filesystem
Jetpack comes with a sample filesystem with X and ubuntu and Canonical keeps updating the packages available, you can also use your own filesystem. In both cases you have to install on it the Nvidia binaries, the instructions to do that are in the file Start_L4T_Docs.html, in the Getting started section. It mentions:
Extract the sample file system to the rootfs directory with this command:
$ sudo tar jxpf ../../Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2
Run the apply_binaries.sh script to copy the NVIDIA user space libraries into the target file system:
$ cd .. $ sudo ./apply_binaries.sh
If you are using a different rootfs, or if you have already configured your rootfs, apply the NVIDIA user space libraries by setting the LDK_ROOTFS_DIR environment variable to point to your rootfs. Then run the script, as shown above, to copy the binaries into your target file system.
If the apply_binaries.sh script installs the binaries correctly, the last message output from the script is “Success!”.
You can create your own ubuntu filesystem using .deb packages as sources following this instructions. The sample filesystem provided is a 32 bits filesystem, however, the kernel is a 64bits kernel. According to devtalk nvidia will be releasing a 64bits filesystem
To receive notifications of new packages
1. Locate and edit the following file:
/etc/apt/sources.list
2. Add the following line:
deb http://ports.ubuntu.com/ubuntu-ports <distribution>-updates main universe
Where <distribution> is the name of the Ubuntu distribution your rootfs is based on. For example, for a rootfs based on the Trusty Tahr distribution of Ubuntu, add the line:
deb http://ports.ubuntu.com/ubuntu-ports trusty-updates main universe
Prerequisite
You have attached an Ethernet cable to the device through either the Ethernet port (if available) or through the USB Ethernet adapter. To install more packages
- Boot the target device.
- Verify your Ethernet connection.
- Update the package list by executing:
$ sudo apt-get update
Note: Ensure that you run sudo apt-get update and not apt-get upgrade, which upgrades already installed packages. Do not confuse the two commands.
- Install packages using apt-get. For example, to install wget execute this command:
$ sudo apt-get install wget
Flashing the board
Nvidia provides several scripts to help to generate the images and to flash the resulting images. Including if you are going to install uboot or fastboot. First the board needs to be in recovery mode and then the main script to flash it is $DEVDIR/Linux_for_Tegra_tx1/flash.sh
1. Recovery mode
To put the board into force USB Recovery Mode:
1. Power down the device. If connected, remove the AC adapter from the device. The device must be powered OFF, and not in a suspend or sleep state. 2. Connect the Micro-B plug on the USB cable to the Recovery (USB Micro-B) Port on the device and the other end to an available USB port on the host PC. 3. Connect the power adapter to the device. 4. With the system powered on: 5. Press and hold the RECOVERY FORCE button. 6. While depressing the RECOVERY FORCE button, press and release the RESET buon. 7. Wait 2 seconds and release the RECOVERY FORCE button
2. To flash the filesystem in the emmc you need to run:
sudo ./flash.sh jetson-tx1 mmcblk0p1
or in the usb
sudo ./flash.sh jetson-tx1 sda1 #it can be other mount point
to use fastboot use the -L option. To see all the options run it with -h
sudo ./flash.sh -L bootloader/<platform>/fastboot.bin <platform> <rootdev>
For NFS you need to use the -N option
References
https://devtalk.nvidia.com/default/topic/929186/jetson-tx1/jetson-tx1-kernel-compilation/ https://devtalk.nvidia.com/default/topic/762653/?comment=4654303 https://devtalk.nvidia.com/default/topic/901677/building-tx1-kernel-from-source/?offset=17 https://devtalk.nvidia.com/default/topic/760180/newbie-building-l4t-from-source-which-branch-tag-/ https://wiki.ubuntu.com/ARM/RootfsFromScratch https://devtalk.nvidia.com/default/topic/929186/jetson-tx1/jetson-tx1-kernel-compilation/post/4854603/#4854603 https://tls.mbed.org/kb/development/arm-thumb-error-r7-cannot-be-used-in-asm-here https://devtalk.nvidia.com/default/topic/914941/jetson-tx1/custom-kernel-compilations/post/4799773/#4799773 https://devtalk.nvidia.com/default/topic/914941/jetson-tx1/custom-kernel-compilations/2 https://devtalk.nvidia.com/default/topic/897280/jetson-tx1/jetson-tx1-with-l4t-23-1-doesn-t-support-native-aarch64-binaries-/post/4732697/#4732697 http://elinux.org/Jetson_TX1 https://devtalk.nvidia.com/default/topic/901677/jetson-tx1/building-tx1-kernel-from-source/post/4749509/#4749509