TensorRtInferenceOp

TensorRtInferenceOp executes a static single-input/single-output TensorRT network on CUDA tensors. It accepts either a serialized TensorRT engine or an ONNX model that is built for the selected GPU during startup.

TensorRT integration

Contact RidgeRun for help integrating your TensorRT model, preparing its inputs and processing its outputs.

Input arguments

Argument	Default and meaning
`model_path`	Required path to an `.onnx` model or serialized TensorRT engine.
`engine_cache_path`	Empty by default. For ONNX input, an optional serialized-engine cache path.
`cuda_device`	CUDA device used for engine building and execution; defaults to `0`.
`max_output_bytes`	`8 MiB` by default; upper bound declared for the runtime output Tensor.

When an ONNX model is used, the operator validates the model identity before reusing a cache. A stale cache is rebuilt instead of being silently paired with a different model file.

Ports and data contract

Port	Memory	Queue	Trigger	Contract
`input`	CUDA device	SDK default	`OnEach`	F32 Tensor. The loaded network must expose exactly one linear, non-vectorized device F32 input.
`output`	CUDA device, runtime pool	N/A	Same activation	F32 Tensor with shape discovered from the network and bounded by `max_output_bytes`.

The input shape must match the loaded engine. The normal device path binds TensorRT directly to CUDA Tensor memory.

V4L2UserPtrSourceOp -> RawBayerToYoloTensorOp -> TensorRtInferenceOp -> FakesinkOp

Build

Follow Holoscan RidgeRun Build and Installation for the module build and installation. The operator-specific include, target, and runtime requirements are listed below.

Include tensorrt_inference/tensorrt_inference.hpp and link holoscan::tensorrt_inference. The operator requires CUDA, TensorRT and the ONNX parser.

Example

The repository's headless YOLO application constructs the operator after the fused RG10 preprocessing stage:

auto inference = graph.op<TensorRtInferenceOp>(
    "inference", model_path, engine_cache_path, 0, 8U * 1024U * 1024U);
graph.add_flow(preprocess->output, inference->input);

compile_options.deployment.bind_tensor_output_device(
    holoscan::TensorOutputDevicePlacement{
        .operator_path = "inference",
        .output_port = "output",
        .device = holoscan::DeviceId{0}});

The model file may be a serialized TensorRT engine or an ONNX file. For ONNX, an optional cache is protected by a lock and accompanied by a model-identity record. A cache whose identity does not match the current model is ignored and rebuilt. The operator reports the discovered input and output names and byte sizes at startup.

Model requirements

The loaded network must have exactly one input and one output, both linear, non-vectorized, device-resident float32 tensors. Dynamic or zero dimensions are rejected by the current implementation. The output rank and byte span must fit the declared max_output_bytes bound. During compute, the input shape and byte size must match the engine exactly; the output Tensor is allocated from the runtime pool and bound directly to TensorRT with setTensorAddress.

The normal path does not stage tensors through host memory. A missing CUDA stream, invalid Tensor placement, failed engine/context creation or failed enqueueV3 returns an operator error. stop() destroys the execution context, engine and runtime resources after the graph has stopped.

No postprocessing

The output is the raw network Tensor. Detection decoding, box filtering, NMS, overlays and display must be implemented by later operators or the application.

Source reference

You can find the source code at TensorRtInferenceOp.

Related pages

RawBayerToYoloTensorOp prepares the YOLO input.
FakesinkOp is the headless benchmark sink.
InferenceOp is the general SDK-owned alternative.

❯