RidgeRun Metadata/RidgeRun Metadata Catalog/SEI
 | The RidgeRun Metadata documentation from RidgeRun is presently being developed. |
Introduction
GstSEIMetadata is a GStreamer plugin developed by RidgeRun that enables the insertion and extraction of metadata within H.264 and H.265 encoded video streams. It leverages Supplemental Enhancement Information (SEI) NAL units to embed metadata directly into the video bitstream, ensuring frame-level synchronization between the video content and its associated metadata, all while maintaining full compatibility with standard playback and decoders.
The main elements of the plugin are seiinject (for injecting metadata) and seiextract (for extracting it). Additional elements act as helpers for testing or for specific use cases, complementing the core functionality.
The plugin is composed of three elements, each with a specific role in the metadata workflow:
- seiinject: Inserts metadata into the H.264/H.265 stream. Metadata can be provided directly as a property, as binary data, or via
GstMeta. In a pipeline, the user can set theseiinjectproperty with the desired data or attach GstMeta messages directly to the buffers. Alternatively, theseimetatesterelement can be used to generate and attachGstMeta. Theseiinjectelement then embeds this metadata into the encoded stream. - seiextract: Extracts the metadata embedded in the H.264/H.265 bitstream and attaches it as
GstMetato the output buffer. Additionally, it emits a signal whenever metadata is received, allowing further processing. - seimetatimestamp: Inserts buffer timestamps as metadata, enabling accurate latency measurements and time correlation within the pipeline.
By combining these elements, GstSEIMetadata provides a flexible framework to synchronize and transport metadata alongside encoded video, making it particularly useful in applications such as analytics, monitoring, and real-time sensor data integration.
Supported Protocols and Containers
GstSEIMetadata is not tied to any specific container or transmission protocol and works with any solution that supports H.264/H.265 video. Since metadata is embedded directly into the H.264/H.265 video stream through SEI NAL units, it is carried transparently regardless of the surrounding container or transport.
This means the plugin can be used with a variety of containers such as MP4, MPEG-TS, and MKV, as well as common transport protocols including UDP and RTP. As long as the video codec is supported, the embedded metadata remains synchronized with the video and accessible for extraction.
This agnostic design provides maximum flexibility, allowing developers to integrate metadata into different streaming pipelines or storage formats without being tied to a single protocol or container.
Benefits and Limitations
| Category | Item | Description |
|---|---|---|
| Benefits | Frame-level synchronization | Metadata is embedded directly into the video stream, ensuring precise alignment with each frame. |
| Container and protocol agnostic | Works with any container or transport that carries H.264/H.265 video (e.g., MP4, TS, MKV, UDP, RTP). | |
| Standards-compliant | SEI messages are safely ignored by standard players and decoders that do not parse them, ensuring compatibility. | |
| Versatile use cases | Suitable for telemetry, analytics, closed captions, timecodes, sensor data, and other mission-critical metadata. | |
| Flexible workflows | Metadata remains intact whether the video is streamed or recorded, without requiring special handling. | |
| Non-intrusive | Does not break or affect playback for receivers or decoders that are unaware of the metadata. | |
| Limitations | No guaranteed visualization | Standard, unmodified players will not display the metadata without custom handling. |
| Pipeline control required | Metadata extraction is not supported by standard players and requires integration into a GStreamer pipeline using seiextract.
| |
| Codec-specific | Limited to H.264/H.265 streams; not applicable to VP8, VP9, or AV1. |
Examples
Here are some example pipelines demonstrating how to use the GstSEIMetadata plugin elements:
- seiinject – inserting metadata into an H.264 stream and saving to MP4:
gst-launch-1.0 videotestsrc num-buffers=10 ! x264enc ! seiinject metadata="Hello World" ! qtmux ! filesink location=Test.mp4 -e
- seimetatimestamp – adding buffer timestamps as metadata and transmitting over MPEG-TS via UDP:
gst-launch-1.0 videotestsrc is-live=true ! x264enc ! seimetatimestamp ! seiinject ! mpegtsmux ! udpsink host=127.0.0.1 port=5000
- seiextract – extracting SEI metadata from an H.264 stream and printing it to the console:
gst-launch-1.0 udpsrc port=5000 ! tsdemux ! h264parse ! seiextract ! fakesink silent=false -v
Documentation
