Preparing Yocto Development Environment for Debugging

Description

Setting up an environment with the appropiate tools can help developers in the process of debugging and overall analyzing a system's behavior. Yocto supports several of which are aimed to produce information useful for the development process, available by adding a few additional packages and features as discussed in the following sections.

General Environment Setup

In order to create an appropriate development environment, Yocto allows to include useful packages and generate specific information required by the development tools. Additionally, it is well known that for the process of debugging, it is also necessary to disable optimizations and enable debug symbols. The following is a list of additional configuration to be added to the build/conf/local.conf file. You can choose which ones to include depending on your application and desired environment.

EXTRA_IMAGE_FEATURES += "\
      dbg-pkgs \       # adds -dbg packages for all installed packages and symbol information for debugging and profiling.
      dev-pkgs \       # adds -dev packages for all installed packages
      tools-sdk \      # adds development tools (gcc, make, pkgconfig, etc)  
      tools-debug \    # adds debugging tools like gdb and strace.
      tools-profile \  # add profiling tools (oprofile, exmap, lttng valgrind (x86 only))
      tools-testapps \ # add useful testing tools (ts_print, aplay, arecord etc.)
      debug-tweaks"    # make image for suitable of development, like setting an empty root password

# Making images useful for debugging.
SELECTED_OPTIMIZATION = "${DEBUG_OPTIMIZATION}" 
DEBUG_BUILD = "1"    
# For not removing debug symbols
INHIBIT_PACKAGE_STRIP = "1"

EXTRA_IMAGE_FEATURES

By editing this variable, we can add the desired packages to be included in our generated image. Bear in mind, however, that adding these will increase the size and time needed for building the image. In the following sections, there is a set list of necessary packages for the recommended tools.

DEBUG_BUILD

Enables full debug and backtrace capabilities for all programs and libraries in the image.

INHIBIT_PACKAGE_STRIP

This is needed to ensure that in the process of building, debugging symbols aren't stripped by Yocto from the binaries it packages. Note: If this is only needed for a specific recipe, it can be added by using OVERRIDES.

INHIBIT_PACKAGE_DEBUG_SPLIT

Adds the symbols and debug info files onto the filesystem as separate files instead of having them embedded with the executables.

Finally, for the general environment setup, depending on where and what we are debugging, it might be useful to have bitbake emit debugging output, which can be achieved by adding the BBDEBUG option to the local.conf as follows. You can also create a build history git repository that stores meta-data about the current project, including package dependencies, size and generated sub-packages, stored in the <project_Dir>/build/buildhistory.

BBDEBUG = "yes"

INHERIT += "buildhistory"
BUILDHISTORY_COMMIT = "1"

Recommended debugging tools

Once the environment is prepared to generate the appropriate information, most additional tools can be included by adding the tool's recipe to the respective meta-layer and adding it to IMAGE_INSTALL on the desired image recipe. Note: If you are not using your own recipe, you can append it to the already existing one by creating a <image_name>.bbappend on your meta-layer. Most of these tools' recipes are already available on a very generic format, and can be modified if necessary to adjust to a specific setup or hardware. You can check your current Yocto build and its meta-layers to see if any of them have the desired recipe, or if it is added to your layer path by running.

bitbake -e <package_name>

For some tools, like GDB and Strace, the process is different, which will be presented in the following sections, since they are already easily included by Yocto.

Strace

Strace is mostly used for problems outside of the binary itself; for example configuration files, input data, and kernel interfaces. This recipe will explain how to use it. For setting up the environment to use it, remember that the package "tools-debug" already includes it, so you should add to your local.conf file:

EXTRA_IMAGE_FEATURES += "tools-debug"

If you want to only add strace, make sure the following are added to your image recipe file:

IMAGE_INSTALL += "strace procps"

Adding "procps" includes pgrep, a process utility that will make our debugging easier by looking up process IDs by name.

Perf

Profiling and tracing can be achieved using Perf. The basic setup can be done manually, or you can also use the available at the git repository from the Yocto Project. The following options are the minimum needed for running perf.

EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs"
INHIBIT_PACKAGE_STRIP = "1"
PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'

Valgrind

As you would do for any other package that you want to include in your image and is not part of the default configuration, you need to add the recipe to build it. The source will depend on what board you a using, and you must add the recipe to your meta-layer. Valgrind also needs some of the previously defined features on your local.conf, specifically:

EXTRA_IMAGE_FEATURES += " dbg-pkgs tools-debug "

Also, remember to add it to your image recipe, as follows:

IMAGE_INSTALL += "valgrind"

Tcpdump

Tcpdump is a powerful command-line packet analyzer; and libpcap, a portable C/C++ library for network traffic capture. You can add it to your target image by including the corresponding recipe on your meta-layer. To add it to your image, you need to add to your image recipe:

IMAGE_INSTALL += "tcpdump"

Kdump

Kdump uses kexec to quickly boot to a dump-capture kernel whenever a dump of the system kernel's memory needs to be taken (for example, when the system panics). The system kernel's memory image is preserved across the reboot and is accessible to the dump-capture kernel. You can find more information on how to use it in the [Yocto Project repository] For Yocto, it is available in the kexec-tools package, and it can be included in your image by adding the recipe to your meta-layer, and modifying your recipe by adding the line:

IMAGE_INSTALL += "kexec-tools"

GDB

GDB is a powerful debugging tool that allows us to see what is going on "inside" another running program, or what it was doing at the moment it crashed. When building with Yocto, it can either used either on the target, or remotely by using gdbserver, which runs on a remote computer (the host GDB), allowing to run or stop a program and read memory regions. Due to all of these processing running on the host, it becomes quite agile on the target in terms of speed and space, given that the binaries on the target don't need their debugging symbols and information.

Setting up environment for debugging on target

As previously mentioned, GDB is included in the package "tools-debug", so commonly the EXTRA_IMAGE_FEATURES would be set as follows:

EXTRA_IMAGE_FEATURES = "tools-debug debug-tweaks dbg-pkgs"

Notice also that the package "dbg-pkgs" will help us include the debug symbols for all the packages. To enable debugging symbols for a specific package, the -dbg package must be included. This can be done by modifying the recipe, or the local.conf file, by adding the following line:

IMAGE_INSTALL += " <package_name> <package_name>-dbg"

Setting up environment for remote debugging (GDBServer)

The host GDB must have access to all debugging information and unstripped binaries, libraries and target must be compiled with no optimizations, all of which were mentioned on the general environment setup. For more information, refer to the official documentation. The steps are listed here for convenience:

• Configure your build system to construct the companion debug filesystem by editing the local.conf file

IMAGE_GEN_DEBUGFS = "1"
IMAGE_FSTYPES_DEBUGFS = "tar.bz2"

These options cause the OpenEmbedded build system to generate a special companion filesystem fragment, which contains the matching source and debug symbols to your deployable filesystem. The build system does this by looking at what is in the deployed filesystem, and pulling the corresponding -dbg packages.

The companion debug filesystem is not a complete filesystem, but only contains the debug fragments. This filesystem must be combined with the full filesystem for debugging. Subsequent steps in this procedure show how to combine the partial filesystem with the full filesystem.

• Configure the system to include gdbserver in the target filesystem by adding gdbserver to either the local.conf or the recipe:

IMAGE_INSTALL += “gdbserver"

• Construct the image and companion filesystem

bitbake <image_name>

• Build the cross GDB component and make it available for debugging. Build the SDK that matches the image. Building the SDK is best for a production build that can be used later for debugging, especially during long term maintenance:

bitbake -c populate_sdk <image_name>

Alternatively, you can build the minimal toolchain components that match the target. Doing so creates a smaller than typical SDK and only contains a minimal set of components with which to build simple test applications, as well as run the debugger:

bitbake meta-toolchain

A final method (the quickest, but no recommended for long-term maintenance) is to build Gdb itself within the build system, producing a temporary copy of cross-gdb you can use for debugging during development.

bitbake gdb-cross-architecture

• Set up the debugfs by running the following set of commands:

mkdir debugfs
cd debugfs
tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2