Preparing Yocto Development Environment for Debugging

Description

Setting up an environment with the appropriate tools can help developers in the process of debugging and overall analyzing a system's behavior. Yocto supports several debugging capabilities which are aimed to produce information useful for the development process. These Yocto features available by adding a few additional packages and features as discussed in the following sections.

General Environment Setup

In order to create an appropriate development environment, Yocto allows to include useful packages and generate specific information required by the development tools. Additionally, it is well known that for the process of debugging, it is also necessary to disable optimizations and enable debug symbols. The following is a list of additional configuration to be added to the build/conf/local.conf file. You can choose which ones to include depending on your application and desired environment. The following example enables ALL of the recommended development/debugging tools.

$YOCTO_BUILD/conf/local.conf:

EXTRA_IMAGE_FEATURES += "\
      dbg-pkgs \       # adds -dbg packages for all installed packages and symbol information for debugging and profiling.
      tools-debug \    # adds debugging tools like gdb and strace.
      tools-profile \  # add profiling tools (oprofile, exmap, lttng valgrind (x86 only))
      tools-testapps \ # add useful testing tools (ts_print, aplay, arecord etc.)
      debug-tweaks \   # make image for suitable of development, like setting an empty root password
      tools-sdk \      # OPTIONAL: adds development tools (gcc, make, pkgconfig, etc)  
      dev-pkgs"        # OPTIONAL: adds -dev packages for all installed packages

# Specifies to build packages with debugging information
DEBUG_BUILD = "1"

# Do not remove debug symbols
INHIBIT_PACKAGE_STRIP = "1"

# OPTIONAL: Do not split debug symbols in a separate file
INHIBIT_PACKAGE_DEBUG_SPLIT= "1"

EXTRA_IMAGE_FEATURES: By editing this variable, we can add the desired packages to be included in our generated image. Bear in mind, however, that adding these will increase the size and time needed for building the image. In the following sections, there is a set list of necessary packages for the recommended tools.
DEBUG_BUILD: Enables full debug and backtrace capabilities for all programs and libraries in the image, by modifying the SELECTED_OPTIMIZATION variable, setting it to "DEBUG_OPTIMIZATION".
INHIBIT_PACKAGE_STRIP: This is needed to ensure that in the process of building, debugging symbols aren't stripped by Yocto from the binaries it packages. Note: If this is only needed for a specific recipe, it can be added by using OVERRIDES.
INHIBIT_PACKAGE_DEBUG_SPLIT: Adds the symbols and debug info files onto the filesystem as separate files instead of having them embedded with the executables.

Finally, for the general environment setup, depending on where and what we are debugging, it might be useful to have bitbake emit debugging output, which can be achieved by adding the BBDEBUG option to the local.conf as follows. You can also create a build history git repository that stores meta-data about the current project, including package dependencies, size and generated sub-packages, stored in the $PROJECT_DIR/build/buildhistory.

$YOCTO_BUILD/conf/local.conf:

BBDEBUG = "yes"

INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

Recommended debugging tools

Once the environment is prepared to generate the appropriate information, most additional tools can be included by adding the tool's recipe to the respective meta-layer and adding it to IMAGE_INSTALL on the desired image recipe. Note: If you are not using your own recipe, you can append it to the already existing one by creating a $IMAGE_NAME.bbappend on your meta-layer. Most of these tools' recipes are already available on a very generic format, and can be modified if necessary to adjust to a specific setup or hardware. You can check your current Yocto build and its meta-layers to see if any of them have the desired recipe, or if it is added to your layer path by running.

bitbake -e $PACKAGE_NAME

For some tools, like GDB and Strace, the process is different, which will be presented in the following sections, since they are already easily included by Yocto.

Tracing and profiling

Strace

Strace allows to observe one or multiple running processes at the level of system (kernel) calls. You can obtain information about your program while it is executing, or you can save your output and analyze it afterwards. It is mostly used for problems outside of the binary itself; for example, knowing which configuration files were read, which files or libraries were ready before a program crash, input data, kernel interfaces, frequency and time consumed by system cals, and so on.

For setting up the environment to use it, remember that the package "tools-debug" already includes it, so you should add to your local.conf file:

$YOCTO_BUILD/conf/local.conf:

EXTRA_IMAGE_FEATURES += "tools-debug"

If you want to only add strace, make sure the following are added to your image recipe file:

On image recipe:

IMAGE_INSTALL += "strace procps"

Adding "procps" includes pgrep, a process utility that will make our debugging easier by looking up process IDs by name.

Perf

Tracing and profiling are fundamental on developing, and it is the developer’s responsibility to write code that provides the highest possible performance, so it becomes necessary to analyze programs to find bottlenecks and optimize those regions of code. This analysis can be achieved by using Perf. More specifically, it can be used to solve performance and troubleshooting functions, and answer a variety of questions about code-paths, allocated memory, TCP retransmits, calls to kernel functions,

The following options are the minimum needed for running perf.

$YOCTO_BUILD/conf/local.conf:

EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs"
INHIBIT_PACKAGE_STRIP = "1"
PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'

Perf is included in the 'tools-profile' package, but it can also be manually included by modifying the image recipe.

On local image:

IMAGE_INSTALL += "perf"

Additionally, you might need to modify your kernel configuration in order to get useful information. You can modify it with the command:

$ bitbake -c menuconfig virtual/kernel

Make sure your debug information is available (CONFIG_DEBUG_INFO=y), which you can configure under 'Kernel hacking -> Compile the kernel with debug info' or, on newer versions, 'Kernel hacking -> Compile-time checks and compiler options-> Compile the kernel with debug info'. Also, make sure perf_events are enabled, which you can check under 'General setup -> Kernel Performance Events and Counters' and 'Kernel features -> Enable hardware performance counter support for perf events'. Finally, you can enable trace calls and kprobes/uprobes for dynamic events under 'Kernel hacking -> tracers'

Memory Analysis

Valgrind

Valgrind is a very powerful tool whenever we are dealing with memory leaks and overall trying to find errors of work with memory, but it can also be used for performance profiling, finding synchronization errors in multi-threading programs and analysis of memory consumption.

As you would do for any other package that you want to include in your image and is not part of the default configuration, you need to add the recipe to build it. The source will depend on what board you a using, and you must add the recipe to your meta-layer. Valgrind also needs some of the previously defined features on your local.conf, specifically:

$YOCTO_BUILD/conf/local.conf:

EXTRA_IMAGE_FEATURES += " dbg-pkgs tools-debug "

Also, remember to add it to your image recipe, as follows:

On your image recipe:

IMAGE_INSTALL += "valgrind"

Network analysis

Tcpdump

Tcpdump is a powerful command-line packet analyzer; and libpcap, a portable C/C++ library for network traffic capture. Is used to display TCP\IP and other network packets being transmitted over the network attached to the system, as well as to display available interfaces and capture packets.

You can add it to your target image by including the corresponding recipe on your meta-layer. To add it to your image, you need to add to your image recipe:

On your image recipe:

IMAGE_INSTALL += "tcpdump"

Kernel debugging

Kdump

Kdump is very useful for analyzing kernel crashes. It uses kexec to quickly boot to a dump-capture kernel whenever a dump of the system kernel's memory needs to be taken (for example, on system panics). The system kernel's memory image is preserved across the reboot and is accessible to the dump-capture kernel. The crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. For Yocto, it is available in the kexec-tools package, and it can be included in your image by adding the recipe to your meta-layer, and modifying your recipe by adding the line:

On your image recipe:

IMAGE_INSTALL += "kexec-tools"

KGDB

KGDB is intended to be used for kernel debugging, as a source level debugger along with GDB. It also allows to place breakpoints in kernel code and perform limited execution stepping.

In order to include KGDB on our image, it is necessary to modify the kernel configuration by running the following command:

bitbake -c menuconfig virtual/kernel

The settings are as follows:

Kernel hacking -> KGDB: Kernel debugger (Check it and Enter):
Check: KGDB: use kgdb over the serial console
Check: KGDB_KDB: include kdb frontend for kgdb        
Enable: Kernel debugging and Magic SysRq key
Enable: Compile the kernel with debug info

Finally, it is necessary to add SSH to the image. This can be done by adding to the $YOCTO_BUILD/conf/local.conf:

$YOCTO_BUILD/conf/local.conf

CORE_IMAGE_EXTRA_INSTALL += "openssh"

Debugging

GDB

GDB is a powerful debugging tool that allows us to see what is going on "inside" another running program, or what it was doing at the moment it crashed. When building with Yocto, it can either used either on the target, or remotely by using gdbserver, which runs on a remote computer (the host GDB), allowing to run or stop a program and read memory regions. Due to all of these processing running on the host, it becomes quite agile on the target in terms of speed and space, given that the binaries on the target don't need their debugging symbols and information.

Setting up environment for debugging on target

As previously mentioned, GDB is included in the package "tools-debug", so commonly the EXTRA_IMAGE_FEATURES would be set as follows:

$YOCTO_BUILD/conf/local.conf:

EXTRA_IMAGE_FEATURES = "tools-debug debug-tweaks dbg-pkgs"

Notice also that the package "dbg-pkgs" will help us include the debug symbols for all the packages. To enable debugging symbols for a specific package, the -dbg package must be included. This can be done by modifying the recipe, or the local.conf file, by adding the following line:

On your image recipe:

IMAGE_INSTALL += " $PACKAGE_NAME $PACKAGE_NAME-dbg"

Setting up environment for remote debugging (GDBServer)

The host GDB must have access to all debugging information and unstripped binaries, libraries and target must be compiled with no optimizations, all of which were mentioned on the general environment setup. For more information, refer to the official documentation. The steps are listed here for convenience:

Configure your build system to construct the companion debug filesystem by editing the local.conf file:

$YOCTO_BUILD/conf/local.conf:

IMAGE_GEN_DEBUGFS = "1"
IMAGE_FSTYPES_DEBUGFS = "tar.bz2"

These options cause the OpenEmbedded build system to generate a special companion filesystem fragment, which contains the matching source and debug symbols to your deployable filesystem. The build system does this by looking at what is in the deployed filesystem, and pulling the corresponding -dbg packages.

The companion debug filesystem is not a complete filesystem, but only contains the debug fragments. This filesystem must be combined with the full filesystem for debugging. Subsequent steps in this procedure show how to combine the partial filesystem with the full filesystem.

Configure the system to include gdbserver in the target filesystem by adding gdbserver to either the local.conf or the recipe:

On your image recipe:

IMAGE_INSTALL += “gdbserver"

Construct the image and companion filesystem

bitbake $IMAGE_NAME

Build the cross GDB component and make it available for debugging. Build the SDK that matches the image. Building the SDK is best for a production build that can be used later for debugging, especially during long term maintenance:

bitbake -c populate_sdk $IMAGE_NAME

Alternatively, you can build the minimal toolchain components that match the target. Doing so creates a smaller than typical SDK and only contains a minimal set of components with which to build simple test applications, as well as run the debugger:

bitbake meta-toolchain

A final method (the quickest, but no recommended for long-term maintenance) is to build Gdb itself within the build system, producing a temporary copy of cross-gdb you can use for debugging during development.

bitbake gdb-cross-architecture

Set up the debugfs by running the following set of commands:

mkdir debugfs
cd debugfs
tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2

RidgeRun Resources