How to Analyze GStreamer with Valgrind
Initial Preparation
Install Debugging Symbols
Debugging symbols map compiled instructions to the source code. These can be embedded into the same library/application binary or in a separate file. Tools like Valgrind won't be able to provide a thorough analysis unless debugging symbols for the application and its runtime dependencies are available. For Debian-based projects, debugging symbols may be installed using APT.
To run a GStreamer based application through Valgrind, you need to install GStreamer and GLib symbols:
GStreamer
sudo apt install gstreamer1.0-*dbg gstreamer1.0-tools
Glib
sudo apt install libglib2.0-0-dbg
If the package with debug symbols isn´t available in your system, you should follow these steps to enable it (More info at https://wiki.ubuntu.com/Debug%20Symbol%20Packages):
- Create an /etc/apt/sources.list.d/ddebs.list:
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \ sudo tee -a /etc/apt/sources.list.d/ddebs.list
- Import the debug symbol archive signing key from the Ubuntu server. On Ubuntu 18.04 LTS and newer:
sudo apt install ubuntu-dbgsym-keyring
- Update your package list
sudo apt update
- Install the GLIB packages with debug symbols:
sudo apt install libglib2.0-bin-dbgsym libglib2.0-0-dbgsym libglib2.0-dev-bin-dbgsym
Building your Application with Debug Symbols
Similarly, you need to build your application under analysis with debug symbols. Here are a couple of examples for Autotools and Meson:
Meson
# In the following line: # --prefix /usr # This is required when building GStreamer plug-ins. Other applications may ignore this # c_args # If using C # cpp_args # If using C++ meson --prefix /usr --buildtype debug build ninja -C build sudo ninja -C build install
Autotools
# In the following line: # --prefix /usr --libdir /usr/lib/aarch64-linux-gnu # This is required when building GStreamer plug-ins. Other applications may ignore this # Note that aarch64-linux-gnu is for AARCH64, change accordingly # CFLAGS # If using C # CXXFLAGS # If using C++ ./configure --prefix /usr --libdir /usr/lib/aarch64-linux-gnu CFLAGS="-g -O0" CXXFLAGS="-g -O0" make sudo make install
Suppressions File
The suppression file is a special file used by Valgrind/Memcheck to ignore certain leak errors that are either expected or not real. GStreamer provides a suppression file that must be used in order to produce a clean, real report.
Download and save the following file as gst.supp somewhere in your system:
https://gitlab.freedesktop.org/gstreamer/common/-/blob/master/gst.supp
Also, Glib provides a suppression file that must be used alongside the GStreamer suppression file. Download and save the following file as glib.supp somewhere in your system:
https://github.com/GNOME/glib/blob/master/glib.supp
https://github.com/GNOME/glib/blob/main/tools/glib.supp
Running Valgrind with GStreamer
This is the recommended recipe to run Valgrind with a GStreamer application:
G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind \ --leak-check=full --leak-resolution=high --num-callers=20 --trace-children=yes \ --suppressions=path/to/gst.supp \ --suppressions=path/to/glib.supp \ gst-launch-1.0 videotestsrc num-buffers=10 ! fakesink
From the command above, these are of special importance:
- G_SLICE=always-malloc
- Force GLib to use malloc instead of the internal slice allocator.
- G_DEBUG=gc-friendly
- Initialize all memory blocks in a way that helps memory analyzers.
- --suppressions=path/to/gst.supp
- Path to the GStreamer suppression file discussed above.
- --suppressions=path/to/glib.supp
- Path to the Glib suppression file discussed above.
The command above will produce the following output. Note the expected report of 0 bytes definitely lost.
==14217== Memcheck, a memory error detector ==14217== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al. ==14217== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info ==14217== Command: /opt/gstreamer-0.10/libexec/gstreamer-0.10/gst-plugin-scanner -l ==14217== GStreamer has detected that it is running inside valgrind. It might now take different code paths to ease debugging. Of course, this may also lead to different bugs. ==14217== ==14217== HEAP SUMMARY: ==14217== in use at exit: 74,096 bytes in 1,940 blocks ==14217== total heap usage: 3,437 allocs, 1,497 frees, 166,891 bytes allocated ==14217== ==14217== LEAK SUMMARY: ==14217== definitely lost: 0 bytes in 0 blocks ==14217== indirectly lost: 120 bytes in 10 blocks ==14217== possibly lost: 0 bytes in 0 blocks ==14217== still reachable: 10,158 bytes in 364 blocks ==14217== suppressed: 63,818 bytes in 1,566 blocks ==14217== Reachable blocks (those to which a pointer was found) are not shown. ==14217== To see them, rerun with: --leak-check=full --show-reachable=yes ==14217== ==14217== For counts of detected and suppressed errors, rerun with: -v ==14217== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 188 from 139) Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Got EOS from element "pipeline0". Execution ended after 19212166 ns. Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... ==14212== ==14212== HEAP SUMMARY: ==14212== in use at exit: 285,981 bytes in 3,333 blocks ==14212== total heap usage: 32,952 allocs, 29,619 frees, 1,766,634 bytes allocated ==14212== ==14212== LEAK SUMMARY: ==14212== definitely lost: 0 bytes in 0 blocks ==14212== indirectly lost: 120 bytes in 10 blocks ==14212== possibly lost: 0 bytes in 0 blocks ==14212== still reachable: 14,335 bytes in 480 blocks ==14212== suppressed: 271,526 bytes in 2,843 blocks ==14212== Reachable blocks (those to which a pointer was found) are not shown. ==14212== To see them, rerun with: --leak-check=full --show-reachable=yes ==14212== ==14212== For counts of detected and suppressed errors, rerun with: -v ==14212== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 196 from 145)
Valgrind on NVIDIA Jetson and Drive platforms
Based on our experience, we found that the Valgrind package installed with sudo apt install valgrind command reports a leak of definitely-lost 16384 bytes with the above baseline test example. This reported leak report is not reliable, since that basic pipeline should report 0 memory leak, which indicates that there is something wrong with the test setup. This problem was observed on NVIDIA Jetson and NVIDIA DRIVE platforms which run Ubuntu 18.04 with GStreamer 1.14.5.
To fix this problem is necessary to build and install Valgrind from the source code. Below you will find the required steps:
1) Uninstall Valgrind from APT sources:
#Execute the below commands only if you already have Valgrind installed in your system sudo apt-get remove valgrind sudo apt-get purge valgrind
2) Build and install Valgrind from source code:
git clone git://sourceware.org/git/valgrind.git cd valgrind ./autogen.sh ./configure make sudo make install
If you want a specific version download the tarball and follow the steps in the following examples for version 3.16.1:
wget https://sourceware.org/pub/valgrind/valgrind-3.16.1.tar.bz2 tar xvfj valgrind-3.16.1.tar.bz2 cd valgrind-3.16.1 ./configure make sudo make install
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.
ARTIFICIAL INTELLIGENCE
