InferenceOp

InferenceOp loads ONNX models or pre-built TensorRT engines, executes them through HoloInfer, and exposes their external tensors as Holoscan ports. One instance can run independent models or connect models through internal tensor names.

The model defines each input's shape, numeric type, and interpretation; image tensors are one example. The module uses the TensorRT trt backend. Include <holoscan/operators/inference/inference.hpp> and link holoscan::op_inference.

Input arguments

Construct the operator with its final InferenceOpParams; there is no default constructor. The three maps establish model files and ordered tensor connections:

Map	Meaning
`model_path_map`	Model name → ONNX or engine path.
`input_map`	Model name → ordered input tensor names.
`output_map`	Model name → ordered output tensor names.

The constructor declares an external input for each unique consumed name not produced by any configured model, and an external output for each produced name not consumed by another configured model. An overlapping name is internal to HoloInfer, with no Holoscan port or queue. For example, backbone → features → head exposes the backbone's external input and the head's external output, while features stays inside the operator.

Use input_port(name) and output_port(name), checking their pointers before dereferencing. Port names are tensor names, not model names. Configure maps before compiling; their topology is fixed for an accepted plan.

Model configuration

The aggregate groups configuration as follows. Defaults are useful for a CUDA/TensorRT path, but must still match the model and deployment:

Fields	Defaults and Use
`backend`, `backend_map`	"trt" and empty map. Any per-model backend override must also be trt.
`trt_opt_profile`	Empty map. Supply ordered "min,opt,max" triples for dynamic dimensions in TensorRT input order; a model's empty vector uses HoloInfer's default profile.
`is_engine_path`	false: build/cache engines from ONNX. true: every path identifies a pre-built engine.
`input_on_cuda`	true: CUDA input port contract and HoloInfer input buffers. false: host input contract/buffers.
`output_on_cuda`, `transmit_on_cuda`	Both true. The first controls HoloInfer result buffers; the second controls emitted runtime-pool tensors.
`dynamic_input_dims`, `output_dynamic_dims`	Both false. The former enables changing input dimensions. The latter is reserved and not consumed; output dimensions are queried each compute.
`enable_fp16`, `enable_cuda_graphs`	false and true. FP16 requests engine-build precision; CUDA Graph capture is subject to fallback safety.
`dla_core`, `dla_core_map`, `dla_gpu_fallback`	−1, empty map, true. Per-model string indices override the global core; "−1" selects GPU.
`parallel_inference`	true: request parallel multi-model execution when eligible.
`max_batch_size`, `max_memory`	256 and 10,000,000,000 bytes. Engine-build batch/workspace ceilings; ignored for pre-built engines.
`cuda_stream_pool`, `cuda_green_context_pool`	Null by default. Optional dedicated model streams and GPU context/SM partitions.

The partial parameter schema covers selected scalar, bool and string fields; maps and resource pointers are outside its coverage. It does not make this aggregate a hot-reconfigurable model loader. Bind changes during authoring and compile a new plan when changing configuration.

from_yaml(path) and from_yaml_node(node, config_dir) load the same configuration. The three model/tensor map keys are required. Relative model paths resolve against the YAML directory or supplied configuration directory. Optional pool sub-maps construct real resource objects; they are not labels.

Ports and data contract

Port	Memory	Queue	Trigger	Contract
External model input	Host or CUDA device, selected by `input_on_cuda`	SDK default	`OnEach` for one input; `OnAligned` for multiple inputs	Model-compatible tensor. Multiple inputs are selected with matching capture times.
External model output	Host or CUDA device, selected by `transmit_on_cuda`	Runtime pool	Same activation	Model output tensors with their configured output names and shapes.

Model tensors must use supported scalar types and dimensions. Internal model tensors are not exposed as Holoscan ports. The input and output memory choices configure where the external tensors are available; they do not change the model's required layout.

Example

This fragment assumes an existing graph and an existing std::filesystem::path model_path pointing to the bundled identity_model.onnx. It uses the tensor names in the SDK's single-model configuration:

auto inference = graph.op<holoscan::ops::InferenceOp>(
    "inference", holoscan::ops::InferenceOpParams{
        .model_path_map = {{"model_a", model_path.string()}},
        .input_map = {{"model_a", {"tensor_in"}}},
        .output_map = {{"model_a", {"tensor_out"}}},
        .dynamic_input_dims = false,
        .enable_fp16 = false,
        .parallel_inference = false,
    });
auto* input = inference->input_port("tensor_in");
auto* output = inference->output_port("tensor_out");
if (input == nullptr || output == nullptr) {
  throw std::runtime_error("Expected identity-model ports were not declared");
}

Include <filesystem>, <stdexcept> and the inference header. Connect a compatible CUDA tensor source and a result consumer through the checked ports, then supply the graph's clock and deployment before compiling. C++20 designated fields follow declaration order: dynamic_input_dims precedes enable_fp16. The linked example supplies the complete source, sink, model paths and run control.

Build

Follow Build the SDK on AGX Orin for the SDK build and installation. Link the installed holoscan::op_inference target in the application.

Related examples and concepts

Examples

Inference Example runs the bundled identity model and compares single/multi-model configurations.
Vision Pipeline Example demonstrates tensor preparation and validation without loading a model.

Concepts

CUDA Stream Ordering explains readiness across streams.
Building Graphs and Queue Policies covers input buffering.
NormalizeOp and ReformatOp illustrate explicit preprocessing choices.

Source You can find the source code at InferenceOp.

Official documentation

NVIDIA Holoscan SDK User Guide provides the official operator and application reference.

❯