GstExtractor: Difference between revisions

It is the responsibility of the user to implement an appropriate customized parser and add the parser to the GstExtractor.  The parser generates the binary SEI and KLV encoded data.

}}

== Getting the Code ==

When you purchase GstExtractor, you will get a git repository with the source code inside. You need to build the plug-in in your system (since you are adding a customized parser) then add the plug-in to your GStreamer catalogue. Clone the repository using something similar to:

git clone git://git@gitlab.com/RidgeRun/orders/${CUSTOMER_DIRECTORY}/gst-extractor.git

cd gst-extractor

GST_EXTRACTOR_DIR=${PWD}

Where <code>${CUSTOMER_DIRECTORY}</code> contains the name of your customer directory given by RidgeRun after purchase.

* The '''filepath''' property is optional in case you want to show a dummy image if a given condition is met. This image should be raw (not encoded) and of the same format and size of the normal stream.  The custom embedded data parser is responsible for deciding (based on the contents of the embedded data) if the actual frame, a dummy image or if no frame at all (dropping) should go downstream. An example use is if the embedded data includes a flag which indicates if the frame data is empty due to no video signal being received.  

}}

=== Custom Data Parsing Implementation ===

GstExtractor will be in charge of separating the embedded data from the video frame and handling the embedded data appropriately, such as sending the data as H264/5 SEI metadata and KLV metadata buffer.  GstExtractor includes helper functions to make it easy for the custom embedded parser to send SEI and KLV data downstream as shown in Figure 2. In order to make your custom embedded data parser, GstExtractor contains a special code section in the file ''gst/gstextractor.c'' for you to modify. This section currently has a basic example of how this could be done, but you must modify it according to your needs:

   {

     /* TODO: Add your custom parsing here. This is just an example that

     GST_OBJECT_UNLOCK (self);

   }

==== Example Data Parsing Implementation Walkthrough ====

1. First we declare and assign some variables for the parsing:

const gint bin_size = 5;

const gint bin_threshold = 50;

sei_size = 20;

klv_size = 14;

It's mandatory that you assign the '''sei_size''' and '''klv_size''' to their corresponding value, otherwise they will be set to 0 and the further buffer/metadata creation will fail. '''bin_size''' and '''bin_threshold''' on the other side, are local variables for the parsing to set the ''value'' property and to replace the input buffer with a dummy frame respectively, so they are optional. '''drop_probability''' is optional as well, and it's only to show how you could configure the parser also to drop a frame and/or meta.

2. "Parse" and assign the result to the provided pointers:

sei_data = (guint8 *) g_memdup (data, sei_size);

klv_data = (guint8 *) g_memdup (data + sei_size, klv_size);

In this example we are not really parsing anything, just copying the data into the '''sei_data''' and '''klv_data''' pointers, but this should be the last step after you parse/encode your data. Keep in mind that '''sei_data''' and '''klv_data''' are null pointers at that point, so you need to allocate them with '''sei_size''' and '''klv_size''' respectively (in this case the ''g_memdup'' function takes care of it).

3. Lock the object since we will modify data that can be accessed from another thread:

GST_OBJECT_LOCK (self);

4. Assign the appropriate binary data to the '''value''' property:

if (self->value) {

   g_byte_array_free (self->value, TRUE);

self->value->len = bin_size;

memcpy (self->value->data, klv_data + (klv_size - bin_size), bin_size);

If you don't need this step, just remove this code.

5. Determine if the frame and/or meta should be dropped

     drop_frame = ((gfloat) (1.0 * rand () / (RAND_MAX)) < drop_probability);

     drop_meta = drop_frame;

In this example we are just setting a probability for dropping the frame to show how the feature works but you can omit it if you don't need it.

6. Create a proper custom condition to know when to replace the input buffer with the dummy frame. You should only replace the ''(atoi ((char *) self->value->data) < bin_threshold)'' condition. Keep in mind that for this piece of code to work you need to assign the '''filepath''' property to extractor element.

if ((atoi ((char *) self->value->data) < bin_threshold) && self->filepath /* Replace with custom condition */

         && !drop_frame) {  

       inbuf = gst_buffer_make_writable (inbuf);

}

This custom condition includes the ''drop_frame'' flag verification so if you set it to TRUE, it won't use the dummy buffer since it would be useless. If you don't need this step, just remove this code.

7. Unlock the object.

GST_OBJECT_UNLOCK (self);

8. Follow the next section to install and test your changes.

In order to install the first three run the following commands:

# GStreamer

sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-plugins-base

sudo apt install python3 python3-pip python3-setuptools python3-wheel ninja-build

sudo -H pip3 install git+https://github.com/mesonbuild/meson.git

=== Compile and Install ===

Then, run:

cd $GST_EXTRACTOR_DIR # Path to your gst-extractor cloned repository

meson build --prefix /usr --libdir $LIBDIR

ninja -C build

sudo ninja -C build install

== Examples ==

gst-launch-1.0 filesrc location=${INPUT} ! videoparse width=1280 height=722 format=yuy2 framerate=30 ! extractor data-size=34 height-padding=2 name=ext ext.video ! videorate ! video/x-raw,framerate=30/1 ! queue ! videoconvert ! ximagesink sync=false ext.klv ! queue ! filesink location=meta.klv

</pre>

=== Advanced Examples ===

</pre>

# KLV meta

00000000 (0x7f893402e590): 73 6f 6d 65 20 6b 6c 76 20 30 31 34 39 00        some klv 0149.   

0:00:43.188620166 146210 0x55b6049b0f60 MEMDUMP          seiextract gstseiextract.c:356:gst_sei_extract_extract_h265_data:<seiextract0> 00000010: 3a 30 30 20                                      :00             

0:00:43.188650823 146210 0x55b6049b0f60 MEMDUMP          seiextract gstseiextract.c:356:gst_sei_extract_extract_h265_data:<seiextract0> ---------------------------------------------------------------------------

Where the injected metadata was '''some klv 0149''' and '''05-11-2021-15:43:00''' for KLV and SEI, respectively.

* '''stream-recovered''': Posted to the bus when it pushes a frame downstream after having dropped one or more.

"stream-recovered" : {

     }

* '''stream-lost''': Posted to the bus every time a video frame is dropped, it increases the '''drop-count''' accordingly.

"stream-lost" : {

         "drop-count" : 49

     }

So in the next example with the help of those messages we will query the amount of dropped frames and if it goes above a threshold we will switch the input stream to a dummy stream that will be playing a file on loop with a message of ''Signal Lost''.

Also, make sure you have the dummy video already created. This can be done with the following gst-launch-1.0 pipeline:

gst-launch-1.0 videotestsrc pattern=ball num-buffers=300 ! video/x-raw,width=1280,height=720,format=NV12 ! textoverlay text="Signal Lost" valignment=center ! nvvidconv ! nvv4l2h265enc ! mpegtsmux ! filesink location=dummy_video.ts

=====Full Example=====

#include <glib.h>

#include <signal.h>

   return ret;

}

=====Example Walkthrough=====

1. Include headers required for using signals, gstd and GLib functions.

#include <glib.h>

#include <signal.h>

#include "libgstc.h"

2. Define pipelines descriptions.

#define EXTRACTOR_NAME "pipe"

#define EXTRACTOR_PIPE "filesrc location=<path-to-yuy2-1280-722-file-with-metadata> ! \

#define DUMMY_PIPE "multifilesrc location=<path-to-your-dummy-video> loop=true \

   ! tsdemux ! mpegtsmux ! interpipesink name=dummy_video sync=true"

Remember to set the ''<path-to-yuy2-1280-722-file-with-metadata>'', ''<path-to-your-dummy-video>'', ''<client-ip>'' and ''<client-port>'' variables according to your environment.

3. Define some important global variables and macros.

#define DROP_STR_PARSE "\"drop-count\" : "

#define DROP_STR_LEN 15

#define RECOVER_STR_PARSE "\"stream-recovered\""

These are for parsing the bus messages that we are sending from the extractor element.

4. Create a signal mechanism to stop the program properly.

   

static int running = 1;

   running = 0;

}

This piece of code here will control the while thread in case of some ''Ctrl+C'' interruption to properly free all the resources.

5. Define the main function and some local variables we will be using.

gint

main (gint argc, gchar * argv[])

   const gint max_dropped = 5;

   const guint64 bus_timeout = 5000000000;

6. Create the Gstreamer Daemon client.

ret = gstc_client_new (address, port, wait_time, keep_open, &client);

   if (GSTC_OK != ret) {

     goto out;

   }

Keep in mind that this function will fail if you haven't started the gstd server.

7. Create the pipelines and set them to play.

   gstc_pipeline_create (client, EXTRACTOR_NAME, EXTRACTOR_PIPE);

   gstc_pipeline_create (client, DUMMY_NAME, DUMMY_PIPE);

   g_print ("Press ctrl+c to stop the pipeline...\n");

   signal (SIGINT, sig_handler);

8. Start the main loop to listen on bus messages of the extractor pipeline.

/* Loop for waiting on the stream-lost and stream-recovered messages. If no

     message is received in 5 seconds, exit the loop */

         &message);

     if (GSTC_OK == ret) {

Here as per simplicity and our gst-extractor custom parser code we expect frames to be dropped within less than 5 seconds so it's ok to have a limited timeout for waiting for the message, but you are free to handle this according to your use case.

9. Parse the received element message to see if it has the '''stream-lost''' message structure.

       str = g_strstr_len (message, -1, DROP_STR_PARSE);

       if (str) {

         continue;

       }

Here we are looking for the '''drop-count''' message part, and if it exists we query the number of frames that have been dropped consecutively. If this value is greater than the threshold and we are not already listening to the dummy source, then switch to the dummy source to let the streaming client know that something happened.

10. Parse the received message to see if it has the '''stream-recovered''' structure instead.

/* Parse the message to find out if the stream is recovered */

       str = g_strstr_len (message, -1, RECOVER_STR_PARSE);

       /* Reset the pointer */

       str = NULL;

If the message was not a stream-lost message then it can be a '''stream-recovered''' message, which leads us to switch back to the main source with the extractor element.

11. Exit the loop if no message was received during the given timeout.

     } else {

       g_printerr ("Unable to read from bus: %d\n", ret);

     }

   }

In our case this will only happen when the video file with embedded metadata has reached the end of the stream.

12. Get the total dropped frames count and free the resources.

   g_print ("Total dropped %d\n", total_drop_count);

   gstc_pipeline_stop (client, EXTRACTOR_NAME);

   return ret;

}

=====Testing the Example=====

1. Run Gstreamer Daemon server:

gstd

2. Run the receiving pipeline on the client side:

PORT=12345

GST_DEBUG=ERROR,*seiextract*:MEMDUMP gst-launch-1.0 udpsrc port={PORT} ! tsdemux name=demux demux. ! queue !  h265parse ! seiextract ! avdec_h265 ! autovideosink sync=false demux. ! queue ! 'meta/x-klv' ! metasink sync=false

3. In another terminal of the server side run the example:

./example

Now you should see on the screen of the receiving side that every time the extractor drops more than 5 frames it will switch to the dummy video stream, and if it recovers it will go back to the extractor pipeline stream.

1. First we start the TCP asynchronous metadata source:

TCP_SERVER_IP="127.0.0.1"

TCP_PORT="3001"

gst-launch-1.0 metasrc is-live=true period=1 metadata=test_async ! meta/x-klv ! tcpserversink host=${TCP_SERVER_IP} port=${TCP_PORT} -v

2. Then we start the final receiver pipeline which will receive the video with the SEI metadata and both types of KLV metadata through MPEG-TS.

FINAL_CLIENT_PORT="12345"

GST_DEBUG=2,*seiextract*:9 gst-launch-1.0 udpsrc port=${FINAL_CLIENT_PORT} ! tsdemux name=demux demux. ! queue !  h265parse ! seiextract ! avdec_h265 ! autovideosink sync=false demux. ! queue ! meta/x-klv,stream_type=6 ! metasink sync=false async=false demux. ! queue ! meta/x-klv,stream_type=21 ! metasink sync=false async=false

Note that here the '''stream_type''' part of the caps is used to identify what kind of KLV metadata we are expecting to receive from each pad. The '''stream_type=6''' is used for asynchronous metadata, while '''stream_type=21''' is for synchronous metadata. On the other side the debug of the seiextract element allows us to see in a quick way the contents of the SEI metadata of the video buffers.

3. Finally to see everything working, run the intermediate pipeline, which is the one having the extractor element.

TCP_SERVER_IP="127.0.0.1"

TCP_PORT="3001"

INPUT="<path-to-yuy2-1280-722-file-with-metadata>"

gst-launch-1.0 mpegtsmux name=mux alignment=7 ! udpsink host=${FINAL_CLIENT_IP} port=${FINAL_CLIENT_PORT} filesrc location=${INPUT} ! videoparse width=1280 height=722 format=yuy2 framerate=30 ! extractor name=ext data-size=34 height-padding=2 ! queue ! videorate ! nvvidconv ! nvv4l2h265enc name=encoder bitrate=2000000 iframeinterval=300 vbv-size=33333 insert-sps-pps=true control-rate=constant_bitrate profile=Main num-B-Frames=0 ratecontrol-enable=true preset-level=UltraFastPreset EnableTwopassCBR=false maxperf-enable=true ! seiinject ! h265parse ! queue ! mux. ext.klv ! meta/x-klv,stream_type=21 ! queue ! mux. tcpclientsrc host=${TCP_SERVER_IP} port=${TCP_PORT} ! queue ! meta/x-klv,stream_type=6 ! mux.meta_54 -v

Note that this pipeline is in charge of muxing all the streams into the MPEG-TS container and it requires the TCP metadata source to be available when starting.

Once this pipeline is running you will see on the receiver side the SEI metadata messages as part of the Gstreamer debug and the hexadecimal/ascii dump of the KLV messages on the standard output similar to this:

# Sync KLV

00000000 (0x7f335402f4d5): 73 6f 6d 65 20 6b 6c 76 20 30 31 32 30 00        some klv 0120.   

0:01:04.052910196  6320 0x55ef8bcbaf70 MEMDUMP          seiextract gstseiextract.c:357:gst_sei_extract_extract_h265_data:<seiextract0> 00000010: 3a 30 30 20                                      :00             

0:01:04.052913389  6320 0x55ef8bcbaf70 MEMDUMP          seiextract gstseiextract.c:357:gst_sei_extract_extract_h265_data:<seiextract0> ---------------------------------------------------------------------------

==Contact Us==