GStreamer Debugging Using GDB
|
|
Overview
The focus of the information on this page is to communicate GStreamer shared library debugging, not describing how to build GStreamer or how to use GDB. For that reason, there is a large cut-and-paste set of commands that are not well explained except for the parts that are of interest to GStreamer shared library debugging.
The debugging session is done on a Ubuntu host. The inspiration of the Ubuntu debugging session was to watch a buffer get passed down the GStreamer pipeline. To have GStreamer shared libraries that are built with debug symbols, the libraries had to be rebuilt.
Enabling Debug Symbols
Debugging Symbols for System Libraries
sudo apt-get install -y libgstreamer1.0-0-dbg gstreamer1.0-libav-dbg gstreamer1.0-plugins-bad-dbg gstreamer1.0-plugins-base-dbg gstreamer1.0-plugins-good-dbg gstreamer1.0-plugins-ugly-dbg
Debugging Symbols for your Project
Meson
The default build type for Meson projects is to enable debug symbols and disable optimizations. If you want to configure it explicitly, use these options:
meson --buildtype debug $EXTRA_PARAMETERS
Autotools
./configure CFLAGS='-ggdb3 -O0' CXXFLAGS='-ggdb3 -O0' $EXTRA_PARAMETERS
Deprecated
 | The instructions below are deprecated: |
Complexities
Several unexpected challenges were encountered that are worth documenting.
GStreamer Libraries with Debugging Symbols
The Ubuntu Workstation GStreamer shared libraries are not built with debug symbols. In order to set breakpoints and step though the GStreamer shared libraries, they need to be rebuilt.
Not Corrupting Ubuntu Workstation
When we build and install the debug versions of the GStreamer library, we don't want to have that library affecting any other Ubuntu applications, like VLC. To avoid polluting the Ubuntu workstation, we install the custom build in a locale directory using the --prefix configure parameter and then we have to tell the GStreamer application where to find the shared GStreamer libraries from the custom build using the GST_PLUGIN_SYSTEM_PATH shell environment variable.
You can't set a breakpoint in a shared library unless the library is already loaded. To address this issue, the LD_PRELOAD environment variable is set to the list of shared libraries we want pre-loaded for debugging.
You can't list the shared libraries to preload unless you can identify them. A simple way to do that is using 'strace and grabbing just of system calls of interest - namely open() calls.
In a simplistic example (using Ubuntu's normal GStreamer components):
strace gst-launch videotestsrc num-buffers=30 ! ximagesink 2>&1 | grep '^open.*gst' | fgrep -v ENOENT
Which produces output like:
open("/usr/lib/libgstreamer-0.10.so.0", O_RDONLY) = 3
open("/home/tfischer/.gstreamer-0.10/registry.i486.bin", O_RDONLY|O_LARGEFILE) = 3
open("/usr/lib/gstreamer-0.10/libgstvideotestsrc.so", O_RDONLY) = 3
open("/usr/lib/libgstbase-0.10.so.0", O_RDONLY) = 3
open("/usr/lib/gstreamer-0.10/libgstximagesink.so", O_RDONLY) = 3
open("/usr/lib/libgstinterfaces-0.10.so.0", O_RDONLY) = 3
open("/usr/lib/libgstvideo-0.10.so.0", O_RDONLY) = 3
GStreamer Registry Cache
Since you will have two sets of GStreamer shared libraries on your system, be careful what ends up in your GStreamer Registry Cache. You can stop the cache from being updated via:
GST_REGISTRY_UPDATE=no
For this debugging session, I am not sure it makes any difference.
Shell Environment Variables
The syntax of a simple shell command is defined (man bash) as
A simple command is a sequence of optional variable assignments followed by blank-separated words, redirections, and terminated by a control operator.
I used something like
LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10" gst-launch testsrc | testsink
Successfully, but when I went to use the debugger, like
LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10" gdb gst-launch testsrc | testsink
I found the LD_PRELOAD had no effect.
Instead, I found I had to first set the variable, then on a separate command line, invoke gdb (again, simplified example):
export LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10" gdb gst-launch testsrc | testsink
Which, to make matters more confusing, didn't work unless I exported LD_PRELOAD (which I don't understand why at the moment).
- GNU DDD is a graphical front-end for command-line debuggers such as GDB,
My preferred graphical front-end for GDB is DDD. However, when I try to use it, I get the following types of errors:
warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. Error while mapping shared library sections: /home/tfischer/projects/gst/fs/lib/libgstreamer-0.10.so.0.20.0: No such file or directory. ... warning: .dynamic section for "/local/home/tfischer/work/foo/fs/fs/usr/lib/libgobject-2.0.so.0" is not at the expected address (wrong library or version mismatch?)
Note that libgstreamer-0.10.so.0.20.0 is exactly where the error message says it can't be found, and for some reason, ddd causes gdb to try to pull in ARM libraries from a path I have no idea how DDD/GDB even knows exists.
I have a hunch if I invoke DDD differently it would work fine. It also appears that DDD is currently an abandoned project.
By comparison, the first two commands work, but the third command, with ddd, fails as shown above.
gdb $DEVDIR/fs/bin/gstd insight $DEVDIR/fs/bin/gstd # Above work, below fails ddd $DEVDIR/fs/bin/gstd
Setup Host Development Directory
# Install host tools sudo apt-get install liboil0.3-dev strace libdbus-1-dev libdbus-glib-1-dev libreadline-dev unzip # Create DEVDIR and get needed packages and a movie file DEVDIR=$HOME/projects/gst mkdir -p $DEVDIR/downloads $DEVDIR/fs cd $DEVDIR/downloads wget http://gstreamer.freedesktop.org/src/gstreamer/gstreamer-0.10.23.tar.bz2 wget http://gstreamer.freedesktop.org/src/gstreamer/gstreamer-0.10.23.tar.bz2.md5 wget http://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-0.10.14.tar.bz2 wget http://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-0.10.14.tar.bz2.md5 wget http://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-0.10.23.tar.bz2 wget http://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-0.10.23.tar.bz2.md5 md5sum -c *.md5 wget http://downloads.dvdloc8.com/trailers/divxdigest/serenity_hd_dvd-trailer.zip unzip serenity_hd_dvd-trailer.zip rm serenity_hd_dvd-trailer.zip mv "Serenity - HD DVD Trailer.mp4" `echo Serenity - HD DVD Trailer.mp4 | tr ' ' '_'` # Unpack packages cd $DEVDIR tar -xjf downloads/gstreamer-0.10.23.tar.bz2 tar -xjf downloads/gst-plugins-base-0.10.23.tar.bz2 tar -xjf downloads/gst-plugins-good-0.10.14.tar.bz2 git clone git://gstd.git.sourceforge.net/gitroot/gstd/gstd gstd # Configure and build GStreamer cd $DEVDIR/gstreamer-0.10.23 CFLAGS=-g \ ./configure -prefix=$DEVDIR/fs --enable-debug --disable-examples --disable-tests make -j 3 make install # Configure and build base plug-ins cd $DEVDIR/gst-plugins-base-0.10.23 GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib \ GST_PLUGIN_PATH= \ CFLAGS=-g \ ./configure --prefix=$DEVDIR/fs --enable-debug --disable-examples --disable-tests make -j 3 make install # Configure and build good plug-ins cd $DEVDIR/gst-plugins-good-0.10.14 GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib \ GST_PLUGIN_PATH= \ CFLAGS=-g \ ./configure --prefix=$DEVDIR/fs --enable-debug --disable-examples --disable-tests make -j 3 make install # Configure and build GStreamer Daemon cd $DEVDIR/gstd GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib \ GST_PLUGIN_PATH= \ CFLAGS=-g \ ./configure --prefix=$DEVDIR/fs make -j 3 sudo make install # need to be root due to connection to system dbus sudo /etc/init.d/dbus force-reload # activate GStreamer Daemon D-Bus Configuration
Exploring and Verifying Custom GStreamer Build
Setup shell environment
I gathered all the shell environment variables that need to be set, as described in the Complixities section and a few more used for the example pipeline.
DEVDIR=$HOME/projects/gst GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib GST_REGISTRY_UPDATE=no export LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstcoreelements.so $DEVDIR/fs/lib/libgstbase-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstqtdemux.so $DEVDIR/fs/lib/libgstrtp-0.10.so.0.16.0 $DEVDIR/fs/lib/libgsttag-0.10.so.0.16.0 $DEVDIR/fs/lib/libgstaudio-0.10.so.0.16.0" AV_FILE=$DEVDIR/downloads/Serenity_-_HD_DVD_Trailer.mp4 MULTICAST_IP_ADDR=224.1.1.1 VIDEO_UDP_PORT=5000 PIPELINE="filesrc ! queue ! qtdemux name=dem dem. ! queue ! rtph264pay ! udpsink host=$MULTICAST_IP_ADDR port=$VIDEO_UDP_PORT auto-multicast=true"
List available elements
Should see around 308 features.
$DEVDIR/fs/bin/gst-inspect
Simple gst-launch video test
Make sure our custom build works.
$DEVDIR/fs/bin/gst-launch videotestsrc num-buffers=30 ! ximagesink
Verify libraries contain debugging symbols
objdump -h $DEVDIR/fs/lib/libgstreamer-0.10.so.0.20.0
You should see around nine headers that start with .debug_.
Identify which custom plug-ins are being used
strace $DEVDIR/fs/bin/gst-launch $PIPELINE 2>&1 | grep '^open.*gst' | fgrep -v ENOENT
Verify the output showing GStreamer shared libraries being used only come from $DEVDIR/fs/lib.
Verify GStreamer Daemon is working
killall -9 gstd $DEVDIR/fs/bin/gstd & $DEVDIR/fs/bin/gst-client create "videotestsrc ! ffmpegcolorspace ! ximagesink" $DEVDIR/fs/bin/gst-client -p 0 play sleep 1 $DEVDIR/fs/bin/gst-client -p 0 destroy killall -9 gstd
If you see an error like any of the following
GStreamer-CRITICAL **: gst_type_find_register: assertion `func != NULL' failed GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed Streamer-CRITICAL **: gst_element_register: assertion `g_type_is_a (type, GST_TYPE_ELEMENT)' failed
I found they went away if I tried a few more times (weird).
Example Debugging Session
# Start gdb debugging session using ddd graphical front-end # Note: starting gstd in the background then attaching gdb/ddd # gets around a problem where starting gstd using ddd causes # LD_PRELOAD to be ignored. killall -9 gstd $DEVDIR/fs/bin/gstd & ddd $DEVDIR/fs/bin/gstd $! # set a gdb breakpoint in filesrc element b gst_file_src_set_location run # In another window, create a pipeline using gst-client (first remembering to set up the environment) DEVDIR=$HOME/projects/gst GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib GST_REGISTRY_UPDATE=no export LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstcoreelements.so $DEVDIR/fs/lib/libgstbase-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstqtdemux.so $DEVDIR/fs/lib/libgstrtp-0.10.so.0.16.0 $DEVDIR/fs/lib/libgsttag-0.10.so.0.16.0 $DEVDIR/fs/lib/libgstaudio-0.10.so.0.16.0" AV_FILE=$DEVDIR/downloads/Serenity_-_HD_DVD_Trailer.mp4 MULTICAST_IP_ADDR=224.1.1.1 VIDEO_UDP_PORT=5000 PIPELINE="filesrc name=src ! queue ! qtdemux name=dem dem. ! queue ! rtph264pay ! udpsink host=$MULTICAST_IP_ADDR port=$VIDEO_UDP_PORT auto-multicast=true" $DEVDIR/fs/bin/gst-client create "$PIPELINE" # Still in the other window, set the name of the file to be read by the filesrc element $DEVDIR/fs/bin/gst-client -p 0 set src location string $AV_FILE
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.
ARTIFICIAL INTELLIGENCE
