RidgeRun Developer Wiki

Compiling Jetson TX1/TX2 source code

Revision as of 17:59, 23 April 2016 by Dsoto (talk | contribs) (→‎References)

Introduction

In this wiki page you are going to find the instructions to download the source code to rebuild the tegra X1 images using jetpack, several parts of this wiki were based in the document called Start_L4T_Docs.html given by Nvidia in L4T. This page should be considered a work in progress

Platform 

Baseboard: P2371
Jetson module: P2180
Soc: tegra210 or tegra21x

Downloading the code

File called Start_L4T_Docs.html inside of Jetpack contains a good description of the packages that it contains. Let's call $DEVDIR the path where you installed your development directory or Jetpack.

1) In order to download the source code you can run the script called source_sync.sh 

$DEVDIR/TX1/Linux_for_Tegra_tx1$ ./source_sync.sh

This will download the bootloader and kernel.

2) It downloads all the kernel tree but you need to specify the tag, looking into the script it seems that the tags should be specified with k and u parameters 

Use: source_sync.sh [options]
Available general options are,
     -h     :     help
     -e     : exit on sync error
     -d DIR : root of source is DIR

By default, all sources are downloaded.
Only specified sources are downloaded, if one or more of the following options are mentioned.

     -k [TAG]: Download kernel source and optionally sync to TAG
     -u [TAG]: Download u-boot source and optionally sync to TAG

Repositories 

k:kernel:nv-tegra.nvidia.com/linux-3.10.git:
u:u-boot:nv-tegra.nvidia.com/3rdparty/u-boot.git:

you can see the repositories on the web looking for: 

http://nv-tegra.nvidia.com/gitweb/?o=age

http://nv-tegra.nvidia.com/gitweb/?p=linux-3.10.git;a=summary
http://nv-tegra.nvidia.com/gitweb/?p=3rdparty/u-boot.git;a=summary

3) It will download the git repositories for kernel and uboot but you need to specify the branch or the tag

Branch: 

cd $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source
git branch -a
git checkout l4t/l4t-r23.2

cd $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/u-boot_source
git branch -a
git checkout l4t/l4t-r23.2

Or if you want to download a tag then the command should look like: 

./source_sync.sh -k tegra-l4t-r23.2 -u tegra-l4t-r23.2

Toolchain

Linaro or code sourcery toolchain's can be used, however, it is recommended to use the Linaro toolchain because it is newer and produces more optimized assembler code

1. Download the Linaro toolchain. You need to install two toolchains, one which is the 64bits toolchain for ARM, and the 32bits toolchain for ARM as well. In this case version 5.3-2016.02 will be used:

2. Install the toolchain 

sudo mkdir /opt/linaro
sudo chmod -R 775 /opt/linaro
sudo chown -R $USER /opt/linaro
cp gcc-linaro-5.3-2016.02-x86_64_aarch64-linux-gnu.tar.xz /opt/linaro
cp gcc-linaro-5.3-2016.02-x86_64_arm-linux-gnueabihf.tar.xz /opt/linaro
cd /opt/linaro/
tar -xf gcc-linaro-5.3-2016.02-x86_64_aarch64-linux-gnu.tar.xz
tar -xf gcc-linaro-5.3-2016.02-x86_64_arm-linux-gnueabihf.tar.xz

Kernel

In order to compile the kernel please follow these setps:

1) Specify the path to the toolchain: 

mkdir -p $DEVDIR/images/modules 
mkdir -p $DEVDIR/images/packages 
export CROSS_COMPILE=/opt/linaro/gcc-linaro-5.3-2016.02-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
export CROSS32CC=/opt/linaro/gcc-linaro-5.3-2016.02-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc
export KERNEL_MODULES_OUT=$DEVDIR/images/modules
export ARCH=arm64

2) Clean your kernel and configuration 

cd $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source/
make mrproper

3) Apply following change to avoid error: r7 cannot be used in asm here 

Index: kernel_source/arch/arm64/kernel/vdso32/Makefile
===================================================================
--- kernel_source.orig/arch/arm64/kernel/vdso32/Makefile        2016-04-08 21:28:52.651992663 -0600
+++ kernel_source/arch/arm64/kernel/vdso32/Makefile     2016-04-11 12:20:03.377388110 -0600
@@ -11,7 +11,7 @@
 
 GCOV_PROFILE := n
 
-ccflags-y := -shared -fPIC -fno-common -fno-builtin -march=armv7-a
+ccflags-y := -shared -fPIC -fomit-frame-pointer -fno-common -fno-builtin -march=armv7-a
 ccflags-y += -nostdlib -Wl,-soname=linux-vdso32.so.1 \
                $(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
 asflags-y := -D__VDSO32__ -s

4) Apply following change to avoid error: logical not is only applied to the left hand side of comparison 

Index: kernel_source/drivers/platform/tegra/tegra21_clocks.c
===================================================================
--- kernel_source.orig/drivers/platform/tegra/tegra21_clocks.c  2016-04-08 21:28:58.755992489 -0600
+++ kernel_source/drivers/platform/tegra/tegra21_clocks.c       2016-04-11 12:49:59.561337028 -0600
@@ -1062,7 +1062,7 @@
  */
 static void tegra21_cpu_clk_init(struct clk *c)
 {
-       c->state = (!is_lp_cluster() == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
+       c->state = ((!is_lp_cluster()) == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
 }
 
 static int tegra21_cpu_clk_enable(struct clk *c)

5) Configure your kernel 

make tegra21_defconfig
make menuconfig

If you want to compare the configuration of your current kernel vrs the kernel that you are going to build (tegra21_defconfig) you can run: 

zcat /proc/config.gz > config #tegra
meld config tegra21_defconfig #host

6) Compile kernel, device tree and modules 

make zImage
make dtbs
make modules
make modules_install INSTALL_MOD_PATH=$KERNEL_MODULES_OUT

7) Save your binaries in the images directory 

cp arch/arm64/boot/Image arch/arm64/boot/zImage $DEVDIR/images/

8) Create kernel_supplements.tbz2 with kernel modules, this package is needed by Jetpack 

cd $DEVDIR/images/modules/lib/modules/
ls .

define a variable with the name of the modules directory 

export KERNEL_MODULES_NAME=3.10.67-gbaaa7aa-dirty

fix the symbolic links on the kernel modules directory 

cd $DEVDIR/images/modules/lib/modules/$KERNEL_MODULES_NAME
rm build source

create tarball 

cd $DEVDIR/images/modules/
tar -cjf kernel_supplements.tbz2 *
mv kernel_supplements.tbz2 $DEVDIR/images/packages

9) Besides creating the modules and the kernel image you need to create again your kernel_headers.tbz2 file (needed by Jetpack). By default when you patch the kernel code and you don't check your changes in the kernel will add a -dirty suffix to the release version (check ls $DEVDIR/images/modules/lib/modules/ for example). For this specific reason you need to generate your headers tarball again changing it the release version. 

cd $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel
tar -xf kernel_headers.tbz2
mv linux-headers-3.10.67-g458d45c linux-headers-$KERNEL_MODULES_NAME
tar -cjf kernel_headers_custom.tbz2 linux-headers-$KERNEL_MODULES_NAME
mv kernel_headers_custom.tbz2 $DEVDIR/images/packages
rm -rf linux-headers-$KERNEL_MODULES_NAME

If you don't want to do this you can run the following before compiling the kernel: 

echo "-gbaaa7aa"> .scmversion

10) Create the dtb directory 

mkdir -p $DEVDIR/images/dtb
cp $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source/arch/arm64/boot/dts/*.dtb $DEVDIR/images/dtb/

11) Copy the Device Tree Compiler (DTC) 

cp $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/kernel_source/scripts/dtc/dtc $DEVDIR/images/dtc

12) Create a backup of the images and packages included in Jetpack 

mkdir -p $DEVDIR/images/packages-backup
cp -rf $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/* $DEVDIR/images/packages-backup

13) Finally overwrite the default images with your own images to install and boot them using jetpack 

cd $DEVDIR/images
rm -rf $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/dtb
cp -rf Image zImage packages/kernel_supplements.tbz2 dtb/ dtc $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/
cp -rf packages/kernel_headers_custom.tbz2 $DEVDIR/TX1/Linux_for_Tegra_tx1/kernel/kernel_headers.tbz2

14) Run the applybinaries script to generate the image to flash 

cd $DEVDIR/TX1/Linux_for_Tegra_tx1/
sudo ./apply_binaries.sh

Output: 

Using rootfs directory of: /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the NVIDIA user space components to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the BSP test tools to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the NVIDIA gst test applications to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the configuration files for the supplied root filesystem to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs
Creating a symbolic link nvgstplayer pointing to nvgstplayer-0.10
Creating a symbolic link nvgstcapture pointing to nvgstcapture-0.10
Adding symlink libcuda.so --> libcuda.so.1.1 in target rootfs
Adding symlink libGL.so --> libGL.so.1 in target rootfs
Adding symlink libcuda.so --> tegra/libcuda.so in target rootfs
Adding symlink libEGL.so --> libEGL.so.1 in target rootfs
Extracting the firmwares and kernel modules to /home/dsoto/devdirs/tegra/Jetpack/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the kernel headers to /usr/src in target rootfs
Installing zImage into /boot in target rootfs
Installing Image into /boot in target rootfs
Installing the board *.dtb files into /boot in target rootfs
Success!

Flash your image with Jetpack

You can flash your image with Jetpack now instead of using the scripts directly, however this takes time (around 10min) because we will need jetpack generating the system.img again

1. Backup your system.img 

cd $DEVDIR/TX1/Linux_for_Tegra_tx1/bootloader/
mv system.img.raw system.img $DEVDIR/images/packages-backup/

2. Just run Jetpack as the first time that you install it: 

cd $DEVDIR
./JetPack-L4T-2.1-linux-x64.run

See the Jetpack output while flashing

It will notice that everything is already build and will install your new images. If you have problems detecting the IP address wait like 2 minutes and it will give you the option to enter it manually. Username and password: ubuntu. You can see the ipaddress connecting a keyboard, mouse and monitor to the board at this point since ubuntu would be already booted.

DTB

Looking into the file $DEVDIR/TX1/Linux_for_Tegra_tx1/jetson-tx1.conf the dtb used by jetson x1 is: tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb 

ODMDATA=0x84000;
NET_BSF=;
EMMC_BSF=;
EMMC_BCT=P2180_A00_LP4_DSC_204Mhz.cfg;
EMMC_CFG=gnu_linux_tegraboot_emmc_full.xml;
BOOTPARTSIZE=8388608;
EMMCSIZE=31276924928;
ITS_FILE=;
SYSBOOTFILE=p2371-2180-devkit/extlinux.conf;
DTB_FILE=tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
DFLT_KERNEL=Image;
ROOTFSSIZE=14GiB;
CMDLINE_ADD="fbcon=map:0";
UBOOT_TEXT_BASE=0x8010e000;
UIMAGE_LABEL="Linux-tegra21";
target_board="t210ref";
ROOT_DEV="mmcblk0p1 ------------ internal eMMC.
        sda1 ----------------- external USB devices. (USB memory stick, HDD)
        eth0 ----------------- nfsroot via RJ45 Ethernet port.
        eth1 ----------------- nfsroot via USB Ethernet interface.";
TEGRABOOT="bootloader/${target_board}/nvtboot.bin";
SOSFILE="bootloader/nvtboot_recovery.bin";
WB0BOOT="bootloader/${target_board}/warmboot.bin";
FLASHAPP="bootloader/tegraflash.py";
FLASHER="bootloader/${target_board}/cboot.bin";
BOOTLOADER="bootloader/${target_board}/p2371-2180/u-boot-dtb.bin";
UBOOT_WITH_TBOOT=yes;
BCFFILE="bootloader/${target_board}/cfg/board_config_p2597-devkit.xml";
INITRD="bootloader/l4t_initrd.img";
TBCFILE="bootloader/nvtboot_cpu.bin";
BPFFILE="bootloader/bpmp.bin";
TOSFILE="bootloader/tos.img";
EKSFILE="bootloader/eks.img";
FBFILE="bootloader/reserved_fb.xml";

Bootloader

The bootloader is a combination of NVIDIA T-Boot (nvtboot) and U-Boot. You can read about the BOOT FLOW in the nvtboot wiki page.

According to the devtalk forum there are two ways to boot the board: fast boot and uboot

  • Fastboot is simpler, and offers no options during boot. It strictly looks for a kernel in the GPT partition from the install's "-k 6" option. This would probably be the way to go for a non-interactive embedded appliance (if not bare metal booting).
  • U-boot offers more options, and looks for a zImage in the /boot partition. Boot configuration can also be edited here, for example I threw in a couple of zImage variants (like debugging version and network options added), and edited extlinux.conf...and magically I can boot to any of those kernels at the serial console boot prompt (provided my eye is fast). No flash was required for any kernel beyond the first kernel.

Inside of the bootloader directory you will find several binaries or components, to get a description of these please read the nvtboot wiki page.

Steps

1. Backup original uboot files. You see which platform you are using checking the file in $DEVDIR/TX1/Linux_for_Tegra_tx1/jetson-tx1.conf, it says: 

SYSBOOTFILE=p2371-2180-devkit/extlinux.conf;

copy the files to the packages-backup directory 

cp -rf $DEVDIR/TX1/Linux_for_Tegra_tx1/bootloader/t210ref/p2371-2180-devkit $DEVDIR/images/packages-backup/

2. Go to uboot and set environment variables 

cd $DEVDIR/TX1/Linux_for_Tegra_tx1/sources/u-boot_source/
export CROSS_COMPILE=/opt/linaro/gcc-linaro-5.3-2016.02-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
export ARCH=arm64

3. Clean and compile 

make distclean
make p2371-2180_defconfig
make

Filesystem

Jetpack comes with a sample filesystem with X and ubuntu and Canonical keeps updating the packages available, you can also use your own filesystem. In both cases you have to install on it the Nvidia binaries, the instructions to do that are in the file Start_L4T_Docs.html, in the Getting started section. It mentions:

Extract the sample file system to the rootfs directory with this command: 

$ sudo tar jxpf ../../Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2

Run the apply_binaries.sh script to copy the NVIDIA user space libraries into the target file system: 

$ cd ..
$ sudo ./apply_binaries.sh

If you are using a different rootfs, or if you have already configured your rootfs, apply the NVIDIA user space libraries by setting the LDK_ROOTFS_DIR environment variable to point to your rootfs. Then run the script, as shown above, to copy the binaries into your target file system.

If the apply_binaries.sh script installs the binaries correctly, the last message output from the script is “Success!”.

You can create your own ubuntu filesystem using .deb packages as sources following this instructions. The sample filesystem provided is a 32 bits filesystem, however, the kernel is a 64bits kernel. According to devtalk nvidia will be releasing a 64bits filesystem

To receive notifications of new packages

1. Locate and edit the following file: 

/etc/apt/sources.list

2. Add the following line: 

deb http://ports.ubuntu.com/ubuntu-ports <distribution>-updates main universe

Where <distribution> is the name of the Ubuntu distribution your rootfs is based on. For example, for a rootfs based on the Trusty Tahr distribution of Ubuntu, add the line: 

deb http://ports.ubuntu.com/ubuntu-ports trusty-updates main universe

Prerequisite

You have attached an Ethernet cable to the device through either the Ethernet port (if available) or through the USB Ethernet adapter. To install more packages

  • Boot the target device.
  • Verify your Ethernet connection.
  • Update the package list by executing:
$ sudo apt-get update

Note: Ensure that you run sudo apt-get update and not apt-get upgrade, which upgrades already installed packages. Do not confuse the two commands.

  • Install packages using apt-get. For example, to install wget execute this command:
$ sudo apt-get install wget

Flashing the board

Nvidia provides several scripts to help to generate the images and to flash the resulting images. Including if you are going to install uboot or fastboot. First the board needs to be in recovery mode and then the main script to flash it is $DEVDIR/Linux_for_Tegra_tx1/flash.sh

1. Recovery mode

To put the board into force USB Recovery Mode: 

1. Power down the device. If connected, remove the AC adapter from the device. The device must be
powered OFF, and not in a suspend or sleep state.
2. Connect the Micro-B plug on the USB cable to the Recovery (USB Micro-B) Port on the device and
the other end to an available USB port on the host PC.
3. Connect the power adapter to the device.
4. With the system powered on:
5. Press and hold the RECOVERY FORCE button.
6. While depressing the RECOVERY FORCE button, press and release the RESET buon.
7. Wait 2 seconds and release the RECOVERY FORCE button

2. To flash the filesystem in the emmc you need to run: 

sudo ./flash.sh jetson-tx1 mmcblk0p1

or in the usb 

sudo ./flash.sh jetson-tx1 sda1 #it can be other mount point

to use fastboot use the -L option. To see all the options run it with -h 

sudo ./flash.sh -L bootloader/<platform>/fastboot.bin <platform> <rootdev>

For NFS you need to use the -N option

References

https://devtalk.nvidia.com/default/topic/929186/jetson-tx1/jetson-tx1-kernel-compilation/
https://devtalk.nvidia.com/default/topic/762653/?comment=4654303
https://devtalk.nvidia.com/default/topic/901677/building-tx1-kernel-from-source/?offset=17
https://devtalk.nvidia.com/default/topic/760180/newbie-building-l4t-from-source-which-branch-tag-/
https://wiki.ubuntu.com/ARM/RootfsFromScratch
https://devtalk.nvidia.com/default/topic/929186/jetson-tx1/jetson-tx1-kernel-compilation/post/4854603/#4854603
https://tls.mbed.org/kb/development/arm-thumb-error-r7-cannot-be-used-in-asm-here
https://devtalk.nvidia.com/default/topic/914941/jetson-tx1/custom-kernel-compilations/post/4799773/#4799773
https://devtalk.nvidia.com/default/topic/914941/jetson-tx1/custom-kernel-compilations/2
https://devtalk.nvidia.com/default/topic/897280/jetson-tx1/jetson-tx1-with-l4t-23-1-doesn-t-support-native-aarch64-binaries-/post/4732697/#4732697
http://elinux.org/Jetson_TX1
https://devtalk.nvidia.com/default/topic/901677/jetson-tx1/building-tx1-kernel-from-source/post/4749509/#4749509
https://devtalk.nvidia.com/default/topic/905345/jetson-tx1/built-tx1-u-boot-from-source/

Articles related

Gstreamer pipelines for Tegra X1
Compile gstreamer on tegra X1
Tegra X1
Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=Compiling_Jetson_TX1/TX2_source_code&oldid=9324"