Revision as of 09:06, 21 March 2014

Links

GStreamer debugging tools - http://docs.gstreamer.com/display/GstSDK/Basic+tutorial+11%3A+Debugging+tools
Good collection of useful hints (including core dumps, using gdb) for GStreamer applications -http://wiki.buzztrax.org/index.php/Debugging
GStreamer application debugging - http://www.gstreamer.net/data/doc/gstreamer/head/gstreamer/html/gst-running.html
GStreamer debugging (helpful syntax) - http://www.gstreamer.net/data/doc/gstreamer/head/manual/html/section-checklist-debug.html
Flumotion approach - http://www.flumotion.net/doc/flumotion/manual/en/trunk/html/chapter-debug.html

GStreamer debugging approaches

Some of these approaches are only useful when you are running a pipeline and audio and/or video stops at an unexpected place in the data stream.

Use standard GStreamer debug output with filter

To see what debug can be enabled, add --gst-debug-help to your GStreamer application arguments, such as:

gst-launch --gst-debug-help

Then you can see which debug you are interested, such as reference counting and buffer movement, and enable all debug output (level output 5).

gst-launch videotestsrc num-buffers=3 ! fakesink --gst-debug=GST_REFCOUNTING:5,GST_BUFFER:5 --gst-debug-no-color=1  2>&1 | grep "\->0" > log.txt

Gets useful data, but typically slows pipeline performance to the point of being not usable.

Another way you can generate the same output is set the GST_DEBUG shell variable:

GST_DEBUG=GST_REFCOUNTING:5 gst-launch videotestsrc num-buffers=3 ! fakesink

Resolving erroneous pipeline could not link

If you get an error like:

WARNING: erroneous pipeline: could not link capsfilter0 to ffenc_mjpeg0

that means the two elements couldn't exchange GStreamer buffers because they couldn't find a common format they both support. To get an insight on what formats are supported, you can add

 --gst-debug=GST_CAPS:4

Create DOT file to see exact pipeline and capabilities used

In order to generate a DOT file, set the GST_DEBUG_DUMP_DOT_DIR environment variable to point to the folder where you want the .dot files to be stored.

1) Install "dot" tool on your host machine...to do so simply install graphviz:

sudo apt-get install graphviz

2) On your target board, set the environment dot dump variable:

export GST_DEBUG_DUMP_DOT_DIR=/root
mkdir -p $GST_DEBUG_DUMP_DOT_DIR

3) Run your pipeline, a .dot file will be generated according to each of the state changes that occur showing how the caps are negotiated during the process.

4) Convert the dot files to PNG image files or SVG graphics files

dot -Tpng input.dot > output.png

dot -Tsvg input.dot > output.svg

Using SVG allows you to n infinitely zoom to be able to read all the text. You can use inkscape to view and edit SVG files.

Use gst-tracelib library to log key pipeline behavior

The gst-tracelib library hooks into gstreamer key functions and logs the function usage behavior. When the application exits it displays some general statistics. Further analysis can be done based on the data written to the log file. Detailed usage description in the source code README file.

gst-tracelib logs

dataflow, messages, queries and events
caps set and get activity
pipeline topology changes
resource usage

Example usage:

export GSTTL_HIDE="caps;chk;topo"
export GSTTL_LOG_SIZE=1048576
AV_FILE=/SD/content/MoMen-dm365.mov

LD_PRELOAD=/usr/lib/gst-tracelib/libgsttracelib.so gst-launch filesrc location = $AV_FILE ! qtdemux name=demux ! queue ! dmaidec_h264 numOutputBufs=12 ! \
priority nice=-10 ! queue ! priority nice=-10 ! dmaiperf ! TIDmaiVideoSink accelFrameCopy=true \
videoOutput=DVI videoStd=720P_60 demux.audio_00 ! queue ! priority nice=-5 ! dmaidec_aac ! alsasink

By default, the log file is named /tmp/gsttl.log. Copy the log file somewhere on your host PC where the scripts gsttl_splitlog.py and gsttl_plt.sh are available (or include the path where the scripts are on $PATH). You can use the following command to create a png file with the information gathered by gst-tracelib:

rm -rf gsttl gsttl.png; gsttl_splitlog.py; gsttl_plot.sh | gnuplot; evince gsttl.png

Watch system interrupts

Run the telnet daemon on the target - likely:

/etc/init.d/inetd start

# or

inetd

telnet into the target hardware and watch the interrupt count

while sleep 1 ; do cat /proc/interrupts ; done

If the pipeline is suppose to be running, the changes in the interrupt count may provide clues as to what is going on.

Capturing a core dump

If your GStreamer application is crashing with a seg fault or similar condition, enable saving a core dump before running the application.

ulimit -c 10000
mkdir -m 777 /root/dumps
echo "/root/dumps/%e.core" > /proc/sys/kernel/core_pattern

Once you have a /root/dumps/*.core file, copy it to your and and inspect it with

ddd  -debugger arm-linux-gnueabi-gdb $GSTREAMER_APPLIATION

Then in gdb,

target core <core file>
bt

and see what function caused the core dump.

Building application and GStreamer libraries with debug enabled

Before using gdb, you need to build your application and the GStreamer libraries with debug enabled.

Make sure you optimization level is not set to -O3
Add -ggdb to your CFLAGS
Remove -fomit-frame-pointer flag from CFLAGS

For TI DMAI plugin and other gstreamer components in the RidgeRun SDK, the can be done via:

cd $DEVDIR/proprietary/gst-dmai-plugins
make clean
make APPS_CFLAGS=-ggdb build install

Use gdb to attach to GStreamer application and examine all the threads

The SDK Debugging Guide provides detailed instructions. I found other helpful gdbserver information as well.

Built your application with symbols (-g) and no optimization (-O0). Use GStreamer libraries that are built with symbols.

Attach to your running GStreamer application using gdbserver

ps
PID=4512 # set the right value based on your application's PID
gdbserver :2345 --attach $PID

Then start the cross compile version of the GNU debugger, like

arm-linux-gnueabi-gdb -tui $GSTREAMER_APP

and get the gdb debugger connected to gdb server on the target

set solib-absolute-prefix <path to devdir>/fs/fs
file <path to file>
target remote <ip address>:2345

List the threads and do a back trace on each one

info threads
bt
thread 2
bt

Use gdbserver in multi-process mode to attach to GStreamer application and examine all the threads

If the info threads command only shows one thread running, you can see if using gstserver in multi-process mode will work better.

On the target (assuming your program name is gstd), first list the PIDs for all the threads in your application:

PROG=gstd
ls /proc/`pidof $PROG`/task

Then start gdbserver in multi-process mode:

gdbserver --multi :2345

Then on the host start the cross compile version of the GNU debugger, like

arm-linux-gnueabi-gdb -tui $GSTREAMER_APP

and get the gdb debugger connected to gdb server on the target

set solib-absolute-prefix <path to devdir>/fs/fs
file <path to file>
target extended-remote <ip address>:2345

Now you can attach to a specific PID, do a backtrack, snoop around, and detach from that PID. Once detach, you can do the same thing again using a different PID from the output of the ls command above.

attach <pid>
bt
detach

Once you detach, the PID will run normally.

Exit locked GStreamer application and see what works stand alone

If you have a GStreamer application that locks up and doesn't run correctly even after you exit the program (possibly with cntl-C) and restart, then it is possible some kernel provided resource is the culprit. For example, if you are using a defective ALSA audio out driver, you might find the GStreamer pipeline locks up in the middle. If you exit the GStreamer application and try a simple audio application, like aplay, you might be able to identify the source of your problem.

Examine a history of what transpired just prior to lockup

If you have a GStreamer application that locks up and can change the pipeline to include a (at this point mythical) recent activity history logger. Such a logger element could be put anywhere in the pipeline. The logger would have circular buffers to keep track of all potentially interesting recent history, such as pad activity, bus activity, and any other relevant information. The circular buffer entries would all be timestamped. When some event occurs (a file exists, a message/signal is received, etc), the element would dump the history, and continue capturing new data.

This idea is after the pipeline locks up, you could cause the history logger to dump it data, and then get an idea of what is suppose to be happening that isn't not occurring.

Activate verbose from application

Some elements (like fakesink and identity) print out information when you enable the verbose output (running the pipeline through gst-launch with the option -v), in order to activate this output manually from a C application you can use:

g_signal_connect( pipeline, "deep-notify", G_CALLBACK( gst_object_default_deep_notify ), NULL );

Enable DMAI Davinci Multimedia Application Interface debug output

If you are using the TI DMAI GStreamer plug-in for the TI Davinci and OMAP SoCs, then you can enable DMAI debug to get more information if a hardware accelerated encoder or decoder is throwing an error.

DMAI_DEBUG=2 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4

Enable CE Codec Engine debug output

If you are using the TI DMAI GStreamer plug-in for the TI Davinci and OMAP SoCs, then you can enable CE debug to get more information if a hardware accelerated encoder or decoder is throwing an error. Note that the stack is GStreamer DMAI plug-in which uses DMAI which uses CE which uses codecs that take advantage of the hardware accelerators.

CE_DEBUG=3 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4

And you can combine DMAI and CE debug

CE_DEBUG=3 DMAI_DEBUG=2 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4

To write the debug to a file on tmpfs, which should reduce the performance impact of enabling debug, add

2>&1 | cat - > /tmp/gst-debug.txt

to the end of the gst-launch command.

Use NFS mount to make it easy to transfer files

If you are creating files on a network connected embedded device, you can use NFS mount (which is different than NFS root mount) so that once you generate a file, it is easy to get it on your desktop machine. For any real-time debugging, don't use NFS root mount as the network activity will effect the timing of your pipelines. See Setting Up A NFS Service for instructions on configuring your desktop machine.

HOST_IP=10.0.1.1
HOST_NFS_SHARE_DIR=/local/home/tfischer/work/sdk/fs/fs

mkdir /tmp/nfs
mount -t nfs -n -o nolock,rsize=1024,wsize=1024 $HOST_IP:$HOST_NFS_SHARE_DIR /tmp/nfs

Now you can copy files between host and target over the network. If you have a file like /tmp/debug-out.txt on the target, you can copy it to the $HOST_NFS_SHARE_DIR on the host using

cp /tmp/debug-out.txt /tmp/nfs

Look for memory leaks

Looking for memory leaks in programs is a very challenging problem. This is even more true for embedded systems, since some of the best memory analysis tools aren't always available, or are difficult to use on this environment.

An easy way to debug the memory leak of a GStreamer program that is running is by looking at the reference counters. Memory leaks on gstreamer happened because reference counters on one or more objects never go back to zero.

References:

[Linux Journal article - Memory Leak Detection in Embedded Systems]

GStreamer `GST_REFCOUNTING` Debug Variable

One of the debug variables is GST_REFCOUNTING. With that, you can get a log of all the refcount changes that happened inside a gst application.

Setting `GST_REFCOUNTING`

Add the following to your application's command line parameters:

--gst-debug=GST_REFCOUNTING:5

Another option is to set the GStreamer debugging shell variable, then run your program:

export GST_DEBUG=GST_REFCOUNTING:5

Filtering debug output

Here are several examples on how to filter the debug output.

You can see which objects disappear, meaning their reference count reaches zero:

gst-launch --gst-debug=GST_REFCOUNTING:5 videotestsrc num-buffers=3 ! fakesink 2>&1 | grep "\->0" | cut -d' ' -f 19

On useful applications there are always some objects that aren't freed until the program exists. With a simple gst-hello example all the objects are always freed. If you set the pipe to run, lets say 13 times, and in the log you see one object that wasn't freed and the count reached 13 or more, then you have a suspicious object or if you see that the number of non-freed objects increase as you increase the number of runs, then you have an indication of a problem.

Use remote syslog

If your application logs warnings and errors, use remote syslog to make the information more easily available.

If you application generates output, you can redirect it to syslog as follows:

MY_APP=gst-render

$MY_APP | logger -t "$MY_APP:"

Generated files debug

There are some tools that can be used for getting information from generated files by gstreamer:

libmp4v2

MP4BOX

mediainfo

With MP4Box you can use the -info in order to get more info from a generated mp4 file:

MP4Box -info test.mp4

GStreamer Performance Tuning

This is worthy of it own Embedded GStreamer Performance Tuning topic.

GStreamer Application Development

GStreamer Daemon

If you can use GStreamer Daemon as your streaming media server, then avoid writing a custom application in the first place!

Developing on host computer

If you have a pipeline where source pads and sink pads are getting connected and disconnected, then you will likely need to write a custom streaming media server. Most people find it easier to build and debug by running the application on the host before trying it on the target hardware.