Inference Example

Run a TensorRT model through InferenceOp using synthetic device Tensors. The example starts with one model, then provides configurations for independent models, chained models, concurrent dispatch, CUDA Graphs, context isolation, and dynamic input shapes.

The application is a compact way to understand the configuration boundary around inference. It does not require a camera or display, but it does require CUDA, TensorRT, enough GPU memory to build an engine, and a writable model cache.

Prerequisites

Use the inference-enabled SDK installation from Build the SDK on AGX Orin. The build must include the inference examples. A working CUDA and TensorRT runtime is required.

What you will learn

How a YAML file supplies model, input, and output configuration.
How one inference operator can represent a single model or several model stages.
How device residency and staging flags affect the Tensor path.
How to treat concurrency and context requests as configuration choices that still need runtime evidence.

Build and run

Build the inference examples through the shared SDK build page. Then run the first configuration from the installed example directory:

WORKSPACE="$HOME/holoscan-5.0"
cd "$WORKSPACE/holoscan-sdk"
INSTALL_DIR="$(./run get_install_dir)"
RUN_BIN="/workspace/holoscan-sdk/$INSTALL_DIR/lib/holoscan/examples/inference/cpp/holoinfer_example"
CONFIG_DIR="/workspace/holoscan-sdk/$INSTALL_DIR/lib/holoscan/examples/inference/configs"
./run launch "$INSTALL_DIR" --run-cmd "$RUN_BIN --config $CONFIG_DIR/01_single_model.yaml --ticks 10"

The first run can take longer while TensorRT creates an engine cache.

Follow the workflow

Operators used

Operator	Role in this example
`TensorFeeder`	Creates the synthetic device Tensors used as model inputs.
InferenceOp	Loads the configured TensorRT model or model chain and produces inference outputs.
`TensorSink`	Receives the inference outputs and reports the run summary.

The YAML root is one inference parameter block. model_path_map names the model files, while input_map and output_map connect model names to Tensor names. The driver creates a feeder for the external inputs and a sink for the external outputs.

Use the configuration files as a set of focused variations:

Configuration group	What it explores
Single and independent models	One model or several models with no internal handoff.
Chained models	One model's output Tensor becomes the next model's input inside the inference stage.
Concurrent models	Independent models request parallel dispatch and can also request CUDA Graphs.
Context-isolated models	Concurrent configurations request separate CUDA resource partitions when the device supports them.
Dynamic-shape models	The configuration enables the dynamic input path used by the bundled model.

The supplied feeder uses device Tensors. The placement fields input_on_cuda, output_on_cuda, and transmit_on_cuda describe the internal input, internal output, and emitted output locations. Device residency does not by itself prove that an inference path has no copies; inspect the configured path and runtime evidence.

Expected result

A completed run prints a summary similar to:

Config: 01_single_model.yaml
Summary: 10 ticks in ... ms

The timing values depend on the target and include application observation effects. They are not a TensorRT kernel benchmark. A requested Green Context or CUDA Graph path can be unavailable or fall back, so check initialization messages before interpreting a configuration as isolated or concurrent.

Related examples and concepts

Examples

Vision Pipeline Example shows a multi-stage image Tensor path before model inference.
CUDA Green Context Example isolates the resource-planning behavior used by selected configurations.

Concepts

InferenceOp documents the operator contract and configuration.
Tensor Contracts and Memory Types explains representation and memory placement.
Automatic Partitioning and Plan Inspection explains resource planning.

Source The driver, configurations, and bundled models are in the Inference example. The driver source defines the graph and observations, while the configuration directory contains the supported variations.

Official documentation

NVIDIA Holoscan SDK official documentation provides the current inference and Tensor reference.

❯