How to Inject Metadata into Video Streams: Choosing the Right Method

Table of Contents [Sticky]

To inject metadata into a video stream, choose a mechanism supported by both the sender and receiver, encode the payload, associate it with the appropriate media time, and insert it into the video bitstream or alongside video at the container or messaging layer. RidgeRun provides GStreamer solutions for H.264/H.265 Supplemental Enhancement Information (SEI), AV1 metadata Open Bitstream Units (OBUs), MPEG Transport Stream (MPEG-TS) metadata, and RTMP/FLV metadata. The right choice depends on the codec, delivery workflow, required frame association, and receiver's extraction capabilities.

This guide helps you select an approach and plan an integration. Follow the linked solution guides for installation, APIs, and sender/receiver pipelines. The RidgeRun Metadata Catalog provides the individual solution descriptions.

What is metadata in video streaming?

Metadata is information that describes content or adds context to it. In simple words, it is information about other information. A video's images show a scene; metadata can tell an application when the scene was captured, where the camera was located, or which objects were detected.

Illustrative examples include:

A recording's title, description, author, and copyright information.
Technical information such as codec, resolution, and frame rate.
GPS position and sensor readings associated with drone video.
Object labels, bounding boxes, and confidence scores associated with analyzed frames.
Timecode labels or event markers used to identify moments in a production workflow.

Metadata helps applications search recordings, correlate sensors with video, display contextual information, and automate processing. Its value depends on preserving the data and interpreting it correctly at the receiver.

Static metadata versus timed metadata

Static metadata describes a file, session, or track: for example, an asset title or a camera identifier. Timed metadata changes during the stream and must be associated with a frame, interval, or event: for example, a GPS position or detection result.

Descriptive MP4 tags and changing live telemetry address different tasks. Adding a title to an MP4 file does not create a stream of frame-associated sensor readings. For image-specific EXIF, IPTC, and XMP workflows, see Handling Image Metadata.

Embedded metadata versus a visible overlay

An overlay writes information into the displayed pixels. Embedded metadata carries information that software can extract separately. For example, an application can use embedded bounding-box coordinates to draw an overlay, store detection records, or trigger an action.

A standard player may display the video without exposing custom metadata. Plan the extraction and application behavior as part of the integration.

Where should metadata be injected?

The insertion point determines how metadata travels and how the receiver recovers it:

Inside encoded video: SEI carries supplemental data in H.264/H.265; metadata OBUs provide a mechanism within AV1. These approaches are useful when data must accompany encoded video frames.
Alongside video in MPEG-TS: A separate metadata stream is multiplexed with video and audio. The receiver demultiplexes it and correlates its timing with the media.
Through FLV metadata in an RTMP workflow: Metadata is handled at the FLV mux/demux layer and transported through the RTMP connection.

Codec, container, and transport are separate decisions. H.264/H.265 and AV1 are codecs; MPEG-TS and FLV are containers; RTMP is a transport protocol. A protocol name alone does not determine where metadata is stored or how an application extracts it.

In GStreamer, attaching GstMeta to a buffer is an application-side operation. An injection element must serialize that information into the output stream for it to reach a recording or remote receiver.

Which metadata injection method should you choose?

Start with the receiver's requirements, then select the mechanism that fits the existing media workflow. If the receiver expects KLV in MPEG-TS, that requirement may decide the approach. If you control both endpoints, you can evaluate bitstream embedding or FLV metadata against the application's timing and delivery needs.

Does your receiver determine the metadata format?

An existing receiver's requirements can determine the metadata injection method. Before comparing solutions, identify which formats the receiving application can extract and interpret, and whether you can modify that application.

If a receiver requires a specific Key-Length-Value (KLV) / Motion Imagery Standards Board (MISB) profile carried in MPEG-TS, evaluate the MPEG-TS metadata approach against that profile. Supporting MPEG-TS or KLV alone does not establish compatibility with every metadata schema. If you control both endpoints, you can instead evaluate SEI, AV1 OBU, or RTMP/FLV metadata against your application's requirements.

Use these questions to narrow the choice:

Does the receiver require a particular format or profile? Identify the required payload encoding, fields, timing conventions, and carriage mechanism.
Can you modify the receiving application? Custom extraction may be practical in a controlled application, while an existing third-party player may require a format it already supports.
Must each payload identify an exact video frame? For frame-associated data, evaluate how the injection and extraction paths preserve that relationship. For separate metadata streams, define an explicit timestamp-mapping policy and how missing or stale samples are handled.
Is the codec or container fixed? SEI applies to H.264/H.265, while metadata OBUs apply to AV1. MPEG-TS metadata uses a separate stream within the transport-stream container; supported video codecs depend on the installed muxer and receiver.
Does the delivery path preserve the chosen representation? Verify servers, recording, transcoding, and repackaging before assuming that metadata supported at the sender will remain available at the final consumer.

Choose a method that the complete workflow can preserve and consume. The ability to inject a payload is only one part of the integration; successful recovery, interpretation, and media association determine whether it meets the application's needs.

For background on receiver compatibility and frame mapping, see RidgeRun's SEI versus In-Band Metadata comparison.

Choosing a metadata injection method
Decision criterion	SEI — H.264/H.265	OBU — AV1	MPEG-TS metadata	RTMP/FLV metadata
Where data travels	Inside the encoded H.264/H.265 video bitstream	Inside the encoded AV1 video bitstream	As a separate metadata stream within MPEG-TS	Through FLV metadata handling in an RTMP workflow
Association with video	Frame-associated; define how each payload maps to a frame	Frame-associated; define how each payload maps to a frame	Timestamp-based correlation between metadata and video	Verify the application's timing requirements; FLV metadata handling differs from SEI carried over RTMP
Receiver requirements	SEI extraction and interpretation of the payload schema	OBU extraction and interpretation of the metadata type and payload schema	Demultiplexing, metadata parsing, and correlation with the media timeline	Compatible FLV metadata extraction and interpretation of the payload
Consider it when	Sending frame-specific detections, capture information, or telemetry with H.264/H.265 video	Sending embedded custom data in an AV1 workflow	Delivering telemetry to a receiver that expects KLV in MPEG-TS	Sending custom information to an application that consumes FLV metadata over RTMP
Verify before implementation	Codec support and metadata preservation through parsers, muxers, servers, and transcoders	AV1 carriage support, matching extraction, and metadata preservation throughout the workflow	Required KLV profile, timestamps, mux/demux capabilities, and receiver interoperability	RidgeRun mux/demux extensions, timing behavior, and preservation through servers or downstream repackaging
RidgeRun solution and documentation	GstSEIMetadata	GstObuMetadata	MPEG-TS In-Band Metadata	GstRTMPMetadata and RTMP integration guide

There is no universally best method. A mechanism that fits a controlled sender/receiver system may require additional integration when the stream passes through a commercial platform or transcoder.

When to use SEI or AV1 OBU

Evaluate SEI for H.264/H.265 and metadata OBUs for AV1 when the data belongs with encoded video frames. Typical requirements include detection results, capture information, or telemetry that the receiving application must associate with a particular image.

RidgeRun's GstSEIMetadata and GstObuMetadata provide injection and extraction elements for their respective codecs. Choose based on the actual codec and design the receiver to interpret the recovered payload. Bitstream embedding does not guarantee that every intermediate component will retain the metadata.

When to use MPEG-TS metadata

Evaluate MPEG-TS metadata when the receiving system expects metadata as a separate stream, particularly for Key-Length-Value (KLV) workflows. KLV describes an encoding structure; the application or required profile defines what the values mean. See SMPTE ST 336.

Video and metadata can have separate production rates. The application must define their time association and how to handle missing or stale samples. Carrying both streams in one container does not eliminate that requirement.

Use RidgeRun's MPEG-TS In-Band Metadata guide for the required components and synchronous/asynchronous handling. GStreamer's mpegtsmux documentation describes supported input caps; the receiver's required metadata profile may impose additional constraints.

What has RidgeRun tested about metadata injection performance?

RidgeRun publishes metadata performance tests with platform details and reproduction instructions. These results help assess an integration, but apply to the documented workloads; they are not universal limits or a direct comparison between metadata methods.

SEI injection at 1080p on Qualcomm Dragonwing 9075 EVK

RidgeRun tested a 1920×1080 NV12 camera pipeline configured for 30 FPS, using H.264 hardware encoding on Qualcomm Dragonwing 9075 EVK with Ubuntu 24.04. The published results compare a baseline pipeline with short and long metadata strings inserted through seiinject.

Reported SEI injection throughput
Test configuration	Mean frame rate
Without SEI injection	29.924 FPS
SEI injection with a short metadata string	29.970 FPS
SEI injection with a long metadata string	29.949 FPS

All three configurations remained near the configured 30 FPS. These throughput measurements do not establish zero processing overhead or measure end-to-end latency. See the complete SEI benchmark for the payloads, pipeline configurations, resource measurements, and reproduction instructions.

MPEG-TS metadata update-rate testing

On Qualcomm Dragonwing 9075 EVK with Ubuntu 24.04 LTS, RidgeRun's MPEG-TS performance documentation covers metadata insertion periods of 33, 20, 16, and 10 milliseconds, with video configurations of 30, 60, 90, and 120 FPS. Its results table reports a 10 ms minimum metadata period across these configurations, using a performance-error criterion below 5%.

The documented performance probe is on the metadata branch. Treat these results as metadata-production measurements in the described workloads, rather than proof of decoded-video throughput, frame-accurate synchronization, or delivery without loss. See the MPEG-TS performance tests for the measurement setup.

For your application, measure payload size, update rate, CPU and memory usage, video throughput, and extraction latency together. Also verify that the receiver recovers the expected data and associates it with the correct media time.

How does metadata injection work with RTMP?

An RTMP workflow can carry metadata through FLV metadata handling or inside the encoded video as SEI. These are different insertion and extraction paths, even when they use the same RTMP connection.

RTMP/FLV metadata injection

RTMP supports data messages, and FLV metadata commonly uses Action Message Format (AMF). RidgeRun's GstRTMPMetadata extends GStreamer's flvmux and flvdemux to insert and recover custom metadata. Its documented interfaces include string and binary payloads and GstFlvMeta integration.

Consider this approach when the application needs metadata handled at the FLV mux/demux layer. Verify whether the supported timing and association behavior meets the requirement for session information, events, or changing data. Do not assume that an arbitrary metadata message provides frame-accurate association.

SEI carried over RTMP

SEI is inserted into encoded video before the video enters the RTMP delivery path. At reception, the application extracts it from the encoded video. This differs from reading FLV metadata at the demuxer.

For a documented H.264 implementation, see RidgeRun's RTMP sender/receiver examples. Codec support depends on the complete RTMP implementation, including its muxer, server, and receiver; support for a codec in an SEI injector alone does not establish RTMP compatibility.

Will an RTMP platform preserve custom metadata?

Successful video ingest does not establish custom-metadata support. RidgeRun's RTMP metadata documentation states that commercial streaming-platform compatibility is not provided out of the box: platforms may modify, strip, or ignore custom fields. Evaluate the specific service and downstream playback path.

Repackaging an RTMP ingest stream for another delivery format may require metadata mapping or a different extraction mechanism. Confirm how the intended consumer obtains the data after every processing stage.

Step-by-step: plan a metadata injection integration

1. Define the payload and its meaning

List the fields the application needs, their units, encoding, schema version, and size limits. Specify whether a payload describes the session, a frame, an interval, or an event. A telemetry schema might include camera ID, capture time, position, and sensor values; a detection schema might include frame ID, coordinates, labels, and confidence.

Use the receiver's required standard or profile when one exists. Custom JSON or binary data needs an agreed interpretation at both ends.

2. Define timing and frame association

Distinguish capture time from presentation time and network arrival time. Specify the clock domain, reference, and association rule. For sensor samples produced at a different rate from video, define whether the application uses the nearest sample, interpolation, or another policy.

Timecode is a separate requirement. A custom timestamp string is not automatically an interoperable broadcast timecode. Identify the required representation and receiver behavior before choosing its carriage mechanism.

3. Identify the complete media path

Record the codec, container, transport, server, processing stages, recording format, and receiver. Identify where encoding, transcoding, repackaging, or frame-rate conversion occurs. This defines where metadata can be inserted and where it must survive or be mapped into a new representation.

4. Select and integrate the injection mechanism

Use the comparison above to select a candidate, then follow its implementation guide. Bitstream injection generally operates on encoded video; separate metadata streams join the media at the muxer. FLV metadata uses the relevant mux/demux interfaces.

Confirm installation, API, and version requirements in the chosen guide.

5. Integrate receiver extraction

Recover the payload with the matching bitstream extractor or demuxer interface, validate its schema, and associate it with the appropriate media sample. Decide what the application does with the data: render an overlay, log telemetry, trigger an event, or forward it to another system.

Schedule display actions against the media timeline. Buffering can separate the time a payload arrives from the time its associated frame appears.

6. Validate the complete workflow

Check actual payload values and frame associations at the final receiver. Include recording/playback, reconnects, seeking where applicable, frame drops, and every transcoding or repackaging stage. Confirm behavior when metadata is missing, delayed, or malformed.

Measure overhead and latency in the target environment. Payload size, injection frequency, buffering, and application processing determine the impact; selecting an embedding method alone does not establish performance.

Why can metadata be lost or associated with the wrong frame?

Successful injection is only one checkpoint. The complete workflow must preserve both the payload and its relationship to the video. A stream can remain playable while metadata is missing, inaccessible, or associated with the wrong image.

Frame association can be lost during batching

For multi-camera integrations, preserve explicit source identifiers and frame identifiers or timestamps with defined clock domains. Verify those associations after batching, splitting, and other buffer transformations.

Check preservation after each processing stage

Verify payload recovery after servers, recording, transcoding, and repackaging. Compare the recovered values and media identifiers with the sender's records to locate the stage where data or association changes. For commercial RTMP services, see the platform compatibility discussion above.

Validate payload preservation and frame association separately

Use a test payload containing a source identifier, frame identifier, and timestamp. At the receiver, check:

Payload preservation: Are the expected fields and values recovered?
Media association: Does each payload identify the correct source and frame or interval?
Timing behavior: Can the application distinguish capture time, presentation time, and arrival time?
Failure behavior: What happens when frames or metadata are dropped, delayed, duplicated, or reordered?

For timestamp-based instrumentation, see RidgeRun's SEI latency measurement example. It demonstrates timestamp insertion and extraction across an RTP/UDP path. The measured interval depends on where those instrumentation points are placed.

What happens after metadata extraction?

Metadata extraction turns embedded data into information an application can use. After recovering a payload, the receiver interprets its schema, associates it with the appropriate frame or event, and decides whether to display, store, process, or forward it.

The same payload can support several application functions. For example, detection coordinates can generate a video overlay, become searchable records, or trigger an event in another system.

Display contextual information with video

A receiving application can use extracted metadata to draw bounding boxes, show telemetry, or highlight events. The application must interpret the payload and schedule its display against the video timeline.

For detection overlays, define the coordinate system and account for resizing, cropping, or other image transformations. Data associated with the original image may need adjustment before it can be displayed correctly.

Store telemetry and analytics records

Extracted metadata can be stored for searches, reporting, or later analysis. Preserve enough information to reconnect each record with its source video: for example, a camera or stream identifier, recording identifier, frame identifier, and timestamp with a defined clock domain.

This enables applications to retrieve the video associated with a detection or sensor event. Saving the payload without its media association can make that connection difficult to reconstruct.

Forward metadata to MQTT or Kafka

Some applications need video and metadata together during capture and transmission, but separate structured records for downstream services. A gateway can extract the embedded payload and publish the relevant information through a messaging system.

RidgeRun's Metadata Suite overview describes MQTT and Kafka integration scenarios. See the MQTT and Kafka integration guides.

When forwarding data, retain its source identifier, timing information, and schema version. Define how downstream consumers handle delayed, duplicated, or missing messages. Publishing a payload to a messaging system does not automatically preserve synchronization with video delivered through another path.

Trigger application actions

An application can interpret extracted metadata to generate alerts, mark recording segments, or request additional processing. Define the conditions under which a payload remains valid and how the application behaves when metadata is unavailable or stale.

Plan the final consumer before selecting the injection method. An overlay needs frame association, an analytics database needs searchable identifiers, and a messaging integration needs a consistent schema and delivery policy. These requirements help determine both the metadata carriage mechanism and the receiver implementation.

Which approach fits common use cases?

These scenarios illustrate selection reasoning rather than customer deployments:

Drone telemetry: Start with MPEG-TS metadata if a ground station requires KLV. If both endpoints are controlled and the data should accompany H.264/H.265 frames, evaluate SEI instead.
Object detection: Evaluate SEI for H.264/H.265 or metadata OBU for AV1. Define coordinates, frame association, and how transformations such as resizing affect the payload.
RTMP ingest with custom information: Evaluate FLV metadata when the receiving application consumes that interface. Evaluate SEI when it parses embedded video data, and verify the server path in either case.
Recorded video with telemetry: Select a mechanism that the recording workflow preserves and the playback application can extract. A file's descriptive tags are insufficient for changing sensor readings.

Explore the RidgeRun Metadata hub for use cases, protocol guides, and messaging-system integration options.

Where can I find implementation commands and examples?

Follow the guide matching your selected approach:

SEI injection and extraction pipelines and SEI application integration.
AV1 OBU installation, user guide, and examples.
MPEG-TS metadata pipelines.
RTMP/FLV installation, interfaces, and examples.
SEI and FLV metadata over RTMP.
Streaming protocol guides for transport-specific integration.

Related Resources

FAQ

Can I inject GPS coordinates or detection results into a live stream?: Yes. Select a supported carriage mechanism, define the payload schema and timing, and integrate matching extraction at the receiver.
Is RTMP an alternative to SEI?: RTMP is a transport protocol; SEI embeds metadata in encoded video. An RTMP workflow can carry SEI or use FLV metadata handling, depending on the application.
Will custom metadata survive transcoding?: Do not assume it will. Verify preservation or explicitly extract, transform, and reinsert the data as required by the processing workflow.
Can a standard video player display embedded metadata?: A player may display video while ignoring custom metadata. Extracting and presenting that data requires compatible application support.
What is the best tool for metadata injection?: Choose by codec, container, timing requirements, and receiver support. RidgeRun provides SEI, AV1 OBU, MPEG-TS, and RTMP/FLV approaches for different workflows.

Get help selecting and integrating a metadata solution

To evaluate an approach with RidgeRun, provide:

The video codec, container, transport, and target platform.
A representative payload, its size, and expected update rate.
Required timing accuracy and frame-association behavior.
Servers, streaming services, and processing stages in the delivery path.
Receiver software and its extraction or display requirements.
Recording, playback, and standards-interoperability requirements.

Contact RidgeRun for solution evaluation and integration support. These details help identify the appropriate injection method and the components that need validation or customization.

Looking for Streaming Protocols and Metadata Solutions? Use our RidgeRun Protocol Advisor

Share This Page

Send Email Using