How to use kgdb in NVIDIA Jetson platforms

Introduction

This document explains how to setup and run KGDB to debug kernel issues in NVIDIA Jetson platforms.

Overview

Because printk is not enough! I do believe printk is a valid debugging tool as many others (despite lot of people might not agree with me) but it has its own limitations, and there are moments where we need to add some extra horsepower to our development. In the case of kernel-debug, there are lot of highly advanced tools such as JTAG debuggers, however no all people have access to a JTAG debugger, and thus KGDB comes as a good alternative when some additional inspection is required.

Kernel setup

In order to use KGDB you need to recompile the kernel source code in order to add debug symbols and enable KGDB, the recommended configuration is as follows:

CONFIG_DEBUG_INFO=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_DEFAULT_ENABLE=0x1
CONFIG_KDB_KEYBOARD=y
CONFIG_KDB_CONTINUE_CATASTROPHIC=0

You can read more about how to compile the Jetpack source code at: Compiling Jetson TX1/TX2 source code

Using KGDB with GDB

Setting up the serial connection

In order to use KGDB over the serial connection, you need to connect an FTDI cable to your board's UART port and to a host PC. The following image shows how to connect the FTDI cable to your Jetson:

The pin connection should match the following for Jetson TX1/TX2 (make sure to look for the right pin connection for any other Jetson board):

J21 Pin 8  (TXD) -> Cable RXD
J21 Pin 10 (RXD) -> Cable TXD
J21 Pin 9  (GND) -> Cable GND

KGDB Boot Start

Change your boot parameter to include the following options:

kgdboc=ttyS0,115200 kgdbwait oops=panic panic=0

You can change it at /boot/exlinux/exlinux.conf. An example exlinux.conf file is shown below:

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1125 core_edp_ma=4000 tegra_fbmem2=0x800000@0x92cb1000 is_hdmi_initialised=1  root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1    root=/dev/mmcblk2p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1 quiet kgdboc=ttyS0,115200 kgdbwait oops=panic panic=0

When you reboot your board it "won't start" because the kernel will halt waiting for the debug session to start.

KGDB Manual Start

KGDB can be also enabled manually from file system configuration, for this, follow the next steps at the Jetson side:

$ sudo su
$ echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
$ echo g > /proc/sysrq-trigger

This will halt the kernel and leave the system waiting for the GDB session to start.

GDB Session Setup

Now we just need to start the GDB session at the host side, for this, we need to use the Toolchain's GDB binary, in my case I am using Toolchain GCC Linaro 7.3.1 as follows:

$ /opt/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gdb <path to vmlinux image>

Once you run it, you should get an output as follows:

$ /opt/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gdb vmlinux
GNU gdb (Linaro_GDB-2018.05) 8.1.0.20180612-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from vmlinux...done.
(gdb)

Make sure you get the following line:

Reading symbols from vmlinux...done.

Now let's connect to the target KGDB session, for this we need to use the serial device linked to our FTDI adapter in the host side, in my case it is ttyUSB1 so we run it as follows:

(gdb) target remote /dev/ttyUSB1

This will open the KGDB session and now you can start debugging. Using KGDB is pretty much the same than using GDB, so the first step is to set any breakpoints you need, for example:

(gdb) break uvc_video_setup

And then let the kernel run so the system can continue execution as normal:

 (gdb) cont

From now on, the overall debug process is similar to using GDB! Enjoy!

Contact Us

Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering information is available at RidgeRun Engineering Services, RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page.