How to use kgdb
I have found kgdb a great way to learn about various parts of the kernel, such as how driver probing works. That said, others have strong opinions:
Do _not_ use kgdb - which is the modus operandi of every sane kernel developer on the planet.
quote from Thomas Gleixner (kernel contributions include High Resolution Timers, Realtime-Linux with Ingo Molnar, and NAND flash driver), who clearly thinks kgdb isn't needed.
Overview of key steps
Commands to be run on the Ubuntu host have a yellow background. Commands to be run on the ARM target have an aqua background.
Configure kernel for kgdb support
Add to kernel command line in order to stop kernel boot and attach kgdb
kgdboc=ttyS0,115200 kgdbwait
- Set CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO:
If you say Y here the resulting kernel image will include
debugging info resulting in a larger kernel image.
This adds debug symbols to the kernel and modules (gcc -g), and
is needed if you intend to use kernel crashdump or binary object
tools like crash, kgdb, LKCD, gdb, etc on the kernel.
Say Y here only if you plan to debug the kernel.
If unsure, say N.
Symbol: DEBUG_INFO [=y]
Prompt: Compile the kernel with debug info
Defined at lib/Kconfig.debug:575
Depends on: DEBUG_KERNEL
Location:
-> Kernel configuration
-> Kernel hacking
- Set CONFIG_KGDB=y
menuconfig KGDB
bool "KGDB: kernel debugging with remote gdb"
depends on HAVE_ARCH_KGDB
depends on DEBUG_KERNEL && EXPERIMENTAL
help
If you say Y here, it will be possible to remotely debug the
kernel using gdb.
- Set CONFIG_KGDB_CONSOLE=y
config KGDB_SERIAL_CONSOLE
tristate "KGDB: use kgdb over the serial console"
select CONSOLE_POLL
select MAGIC_SYSRQ
default y
help
Share a serial console with kgdb. Sysrq-g must be used
to break in initially.
Code Optimization
Code optimization will harm the debugger stepping in a high level language (C in this case), because the high level statements necessarily map to exact one sequence of assembler statements. This results in jumping around with the stepping cursor in a debug session when sequential instruction processing is expected.
The only way to overcome this issue is to compile the code with -O0.
However, it is not possible to compile the whole kernel with -O0 since some few special code sequences can only be compiled with optimization.
Further, it is neither necessary nor a good idea to compile the whole kernel with -O0 for two reasons:
- In practice only a driver or e special subsystem contains the buggy code that has to be stepped through exactly - The kernel may behave different from the productive -O2 or -Os compilation that could lead to wrong conclusions
The kernel is compiled with '-Os' instead of '-O2' if CONFIG_CC_OPTIMIZE_FOR_SIZE is enabled. One may disable this option to have a weaker optimization but it's a better practice to add the following line to the Makefile of the driver/subsystem of interest:
ccflags-y := -O0
This will add -O0 after -O2 respectively -Os set generally by the kernel.
As stated by the GCC documentation:
The last option is effective:
If you use multiple -O options, with or without level numbers, the last such option is the one that is effective.
Additional debug info
It is possible to provide more debug info for gdb by using the -ggdb<LEVEL> option instead of -g e.g. including macro definitions.
Just change the top level Makefile of the kernel before compiling as follows:
... ifdef CONFIG_DEBUG_INFO #KBUILD_CFLAGS += -g KBUILD_CFLAGS += -ggdb3 KBUILD_AFLAGS += -gdwarf-2 endif ...
Setup runtime for kgdb usage
- Boot into Linux
- Enable inetd so you can telnet into hardware
- On the target, switch console tty device for use with kgdb
echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc echo g > /proc/sysrq-trigger
The SysRq command is also useful in order to stop kernel execution, sometimes it is normal to use at gdb the continue commands and the kernel will continue running without finding a break point, in order to stop kernel execution when desired send the SysRq command and kernel will stop at the gdb console:
echo g > /proc/sysrq-trigger
- On the host, exit terminal emulator (picocom) and run gdb
cd $DEVDIR/kernel/linux-* arm-linux-gnueabi-gdb -tui # or maybe arm-none-linux-gnueabi-gdb -tui file vmlinux set remotebaud 115200 target remote /dev/ttyUSB3 break uart_register_driver continue
In telnet window, cause the breakpoint to be tripped
KDbg as GUI front end
NOTE: KDbg has an issue on Ubuntu 14.04.
Problem can be solved as described here
by removing the image file:
sudo rm /usr/share/kde4/apps/kdbg/icons/hicolor/22x22/actions/pulse.mng
For a GUI front end KDbg vcan be used.
Installation in Ubuntu:
sudo apt-get install kdbg
To use the cross debugger instead of the host's gdb the global settings must be modified.
Start the front end:
kdbg
Change the global settings:
Settings -> Global Options...
The Global Options dialog appears.
Setup How to invoke GDB: as follows if e.g. the Code Sourcery arm-2009q1 under /opt/codesourcery is used as toolchain:
/opt/codesourcery/arm-2009q1/bin/arm-none-linux-gnueabi-gdb --fullname --nx
For a different toolchain modify the path and the toolchain prefix acordingly.
Leave kdbg.
Now, if kgdb is started and waiting for a connection, you can connect with the KDbg front end as follows:
kdbg -r /dev/ttyUSB0 kernel/linux-2.6.32.17-psp03.01.01.39/vmlinux
Adjust the kernel's vmlinux path and the serial port according to your configuration.
After about 5 seconds the source file kernel/kgdb.c should open stopping the execution at:
...
void kgdb_breakpoint(void)
{
...
arch_kgdb_breakpoint();
...
}
...
Loadable modules and kgdb
Load the module and learn the load address.
modprobe sc16is7 lsmod
You should get output something like
sc16is7 9793 0 - Live 0xbf000000
where 0xbf000000 is the module starting virtual address.
Tell the gdb about the module
add-symbol-file drivers/serial/sc16is7.o 0xbf000000
Hints
- If you want to interrupt a running kernel and you get back to the gdb prompt
echo g > /proc/sysrq-trigger
- Don't try to debug parts of the kernel that deal with scheduling/execution control. Don't even try to step over them with the next function. Instead, set a breakpoint after the function call to get gdb back in control. Some of the scheduling/execution control functions to watch out for include:
queue_work() spin_lock() spin_unlock()
As you are stepping though code and approach one of these functions, such as:
│199 int tty_buffer_request_room(struct tty_struct *tty, size_t size)
│200 {
│201 struct tty_buffer *b, *n;
│202 int left;
│203 unsigned long flags;
│204
>│205 spin_lock_irqsave(&tty->buf.lock, flags);
│206
│207 /* OPTIMISATION: We could keep a per tty "zero" sized buffer
│208 remove this conditional if its worth it. This would be
│209 to the callers */
│210 if ((b = tty->buf.tail) != NULL)
(gdb) s
tty_buffer_request_room (tty=0xc21b1000, size=1) at drivers/char/tty_buffer.c:200
(gdb)
You can skip the function by setting a temporary breakpoing on a line of code after the function call, such as:
(gdb) tbreak 210 Breakpoint 2 at 0xc018e548: file drivers/char/tty_buffer.c, line 210. (gdb) continue Continuing. tty_buffer_request_room (tty=0xc21b1000, size=1) at drivers/char/tty_buffer.c:210 (gdb)
Then you can continue using gdb normally.