How to Use gst-shark Measurements to Perform Comprehensive Analysis on GStreamer Pipeline

GstShark is an open-source project developed by RidgeRun that provides benchmarking and profiling tools for GStreamer versions 1.7.1 and above. It includes a variety of tracers for collecting detailed debugging information, as well as tools for visualizing and analyzing this data. GstShark simplifies the task of identifying performance bottlenecks by leveraging GStreamer's built-in tracing hooks and combining them with standard open-source tracing and plotting utilities. For more details, please refer to the Official GstShark wiki.

Although GstShark includes graphical tools for visualizing trace data, these plots can become difficult to interpret when there is significant variance or fluctuation in the measurements. For example, high variability in proctime values (the time each element spends processing data) can make it hard to identify patterns or draw conclusions directly from the graph.

To address this issue, we can compute statistical metrics such as the mean, median, and maximum, which provide clearer and more interpretable insights into pipeline performance. These calculations allow us to summarize key behaviors and make informed optimization decisions based on solid numerical evidence rather than visual approximation.

This guide assumes that:

GStreamer is already installed on your system.
You have cloned the source code of GstShark.
You are familiar with generating measurements and producing plots using the ./gstshark-plot script.

If you need help with any of these prerequisites, consult the GstShark documentation.

Performing Specific Calculations on gst-shark Trace Data

1. Prevent Trace Files from Being Deleted

First, open a terminal and navigate to the root of your cloned GstShark repository. Then, open the gst-shark/scripts/graphics/gstshark-plot script and comment out or delete the section at the bottom that removes intermediate files. We want to retain these files to perform our calculations.

The lines to be commented out or removed are:

# Remove files
#rm -f datastream.log

#for tracer in "${parser_group_list1[@]}"
#do
#    rm ${tracer}.log ${tracer}.mat -f
#done

#for tracer in "${parser_queuelevel[@]}"
#do
#    rm ${tracer}.log ${tracer}.tmp ${tracer}.mat -f
#done

#for tracer in "${parser_group_list2[@]}"
#do
#    rm ${tracer}.log
#    rm ${tracer}_fields.log ${tracer}_fields.mat -f
#    rm ${tracer}_values.log ${tracer}_values.mat -f
#done

#for tracer in "${parser_group_list3[@]}"
#do
#    rm ${tracer}.log ${tracer}.mat ${tracer}.tmp -f
#done

2. Modify the Octave Plotting Script

Each GstShark tracer has a corresponding plotting script written in Octave, a high-level language intended for numerical computations. These scripts are located in the gst-shark/scripts/graphics/ directory.

For this example, we will modify plot_proctime.m, which processes the proctime trace data. These scripts load measurement data into the following variables:

element_name_list: names of the GStreamer elements
timestamp_mat: timestamps of each measurement
time_mat: corresponding measurement values (e.g., processing time in nanoseconds)

Here is an example that calculates:

the mean processing time per element (in milliseconds),
the estimated FPS per element, and
the global FPS, based on the slowest element.

#! /usr/bin/octave -qf

[element_name_list, timestamp_mat, time_mat] = load_serie_timestamp_value('proctime.mat');

if ((1 == length(timestamp_mat)) && (0 == timestamp_mat))
    return
end

tracer.proctime.timestamp_mat = timestamp_mat;
tracer.proctime.time_mat = time_mat;
tracer.proctime.element_name_list = element_name_list;

fprintf('\n==== Mean proc_time per GStreamer element (in milliseconds) and estimated FPS ====\n');
num_elements = length(element_name_list);

element_fps_list = [];

for i = 1:num_elements
    element_name = element_name_list{i};
    
    values = time_mat(i, :);
    values = values(~isnan(values));  
    
    if ~isempty(values)
        med = mean(values) / 1e6;
        estimated_fps = 1000 / med;
        fprintf('%s: Mean proc_time = %.3f ms, Estimated FPS = %.2f\n', element_name, med, estimated_fps);
        
        element_fps_list = [element_fps_list; estimated_fps];
    end
end

global_estimated_fps = min(element_fps_list);

fprintf('\nGlobal max FPS (determined by slowest element): %.2f fps\n', global_estimated_fps);

3. Generate the Trace Files

Before executing your modified plotting script, you need to generate the required trace data. Follow the instructions here: GstShark - Generating trace files.

4. Execute the Plot Script

Once the trace files are generated, run the gstshark-plot tool and pass the path to the generated trace directory. For example:

./gstshark-plot /home/user/gstshark_YY-MM-DD_hh:mm:ss/ -s pdf

If you have already modified the specific plot script (e.g., plot_proctime.m), the results will be displayed automatically after this command is executed.

From this point onward, since the trace data is already saved, you can run only the specific plot script directly—modifying it as needed to calculate other metrics. Make sure the script is executable:

chmod +x plot_proctime.m

Then run it with:

./plot_proctime.m

Note: Every time you generate new trace data, you must re-run ./gstshark-plot to update the .mat files.

5. Example Output

Below is an example of the output produced by the modified plot_proctime.m script:

==== Mean proc_time per GStreamer element (in milliseconds) and estimated FPS ====
"source_queue_dec264",: Mean proc_time = 12.441 ms, Estimated FPS = 80.38
"h264parse0",: Mean proc_time = 0.111 ms, Estimated FPS = 9037.67
"avdec_h264-0",: Mean proc_time = 1.646 ms, Estimated FPS = 607.71
"source_scale_q",: Mean proc_time = 5.196 ms, Estimated FPS = 192.45
"source_videoscale",: Mean proc_time = 1.434 ms, Estimated FPS = 697.25
"source_convert",: Mean proc_time = 1.725 ms, Estimated FPS = 579.67
"capsfilter0",: Mean proc_time = 0.141 ms, Estimated FPS = 7070.44
"inference_videoscale",: Mean proc_time = 0.054 ms, Estimated FPS = 18422.75
"capsfilter1",: Mean proc_time = 0.037 ms, Estimated FPS = 27281.10
"inference_videoconvert",: Mean proc_time = 0.033 ms, Estimated FPS = 30590.42
"inference_hailonet",: Mean proc_time = 21.350 ms, Estimated FPS = 46.84
"inference_hailofilter",: Mean proc_time = 2.594 ms, Estimated FPS = 385.57
"identity_callback",: Mean proc_time = 0.141 ms, Estimated FPS = 7108.14

Global max FPS (determined by slowest element): 46.84 fps

With this method, you can gain deeper insight into the performance of your GStreamer pipeline by computing meaningful metrics over all collected data points. This enables data-driven decisions for optimization and debugging.

For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.

Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.

❯