Performance and Debugging

When a graph fails or misses a performance goal, identify the stage that owns the symptom before changing it. Check the build and declarations, inspect compilation, confirm startup, and then measure a short run. This sequence shows whether the next action belongs in the application, the graph plan, the runtime setup, or the data path.

The steps below cover a reduced reproduction, compiler diagnostics, runtime evidence, and source inspection. Keep the environment and measured results with each change so successive runs can be compared.

RidgeRun can help find the causes of dropped frames, growing memory use, or high latency and tune your pipeline to meet its performance goals. Contact RidgeRun for help profiling and optimizing your application.

Advanced diagnosis topic

Use this page after the application works on a small path. It helps investigate rejected plans, startup outcomes, data-path stalls, copies, waits, and measured performance.

Diagnose from compile to cleanup

Follow the first stage that explains the symptom:

Symptom	First evidence to inspect
Compilation is rejected	The execution plan status, diagnostics, and serialized plan
Startup does not reach a usable session	Session termination and diagnostic results
A compute path fails or stalls	Callback results, selected data, output loans, and data contracts
Memory or latency is unexpectedly high	Tensor ownership, queue retention, CUDA stream ordering, and copy evidence
The run finishes with an unexpected outcome	Final termination, cleanup diagnostics, and retained resources

The Debugging in the NVIDIA Holoscan SDK documentation provides the official failure-analysis introduction. The rest of this page adds the evidence needed for graph, Tensor, and performance investigations.

Related Concepts

Automatic Partitioning and Plan Inspection explains accepted execution boundaries.
Rates, Queue Sizing, and Derived Tensor Capacity explains the sizing evidence behind a queue or pool limit.
Copy Paths and Memory Conversion explains where actual payload movement can occur.

Reduce the reproduction before changing the graph

Start with the smallest application that still reproduces the symptom. Keep the input shape, dtype, memory kind, clocks, and queue policy the same while removing unrelated branches. Record the SDK revision, operator parameters, endpoint catalog where applicable, and the exact command. A short bounded run is easier to compare than a long session with changing inputs.

First distinguish a C++ build failure from a rejected graph compilation. Missing headers or an unavailable dependency belong to the build environment; a completed executable can still produce an invalid ExecutionPlan. Follow Build the SDK on AGX Orin for the supported build environment, and Writing a Minimal Operator for the application scaffold. Use a debuggable build such as RelWithDebInfo when investigating the application's code.

Read the compile diagnostic before running

After compile(), check plan.ok(). On rejection, inspect plan.diagnostics() and retain plan.json(). The structured Diagnostic includes a stable token, severity, source identity, and required/provided summaries where applicable. Follow the named port or operator back to its declaration; the final display stage may only be showing an earlier failure.

Diagnostic or observation	What to check next
`GRAPH_PARTITION_GRAPH_EMPTY`	Check for a selected embedded graph subtree with no executable operators. Use the compiler's source identity to find the partition declaration.
`GRAPH_PARTITION_AUTOMATIC_REMAINDER_EMPTY`	Add a nonempty exact partition anchor before requesting an automatic remainder, or remove the request to let the compiler partition the whole graph automatically.
`RATE_EXPECTATION_MISMATCH`	Compare the input expectation with the connected producer's authoritative rate. Correct the expectation instead of treating it as an override.
`RATE_QUEUE_DEPTH_BELOW_RECOMMENDATION`	Inspect the explicit depth's origin and the nominal recommendation. Decide whether the warning reflects an intentional bounded buffer; the compiler has kept the authored depth.
`RATE_QUEUE_DEPTH_BELOW_HARD_FLOOR`	Check the contract's required retention and burst bounds. Supply a sufficient depth or change the contract's semantics; advisory headroom cannot waive a hard floor.
`RATE_BOUNDED_EVIDENCE_REQUIRED`	Identify the missing finite rate, ceiling, or horizon required by the recipe. Add truthful bounded evidence instead of an arbitrary large queue.
`RATE_SUSTAINED_BACKPRESSURE` or `RATE_PREDICTED_REPLACEMENT`	Compare nominal arrivals with how many values the selector advances. Review producer pace, trigger, and selector together. A larger queue cannot solve a sustained nominal imbalance.
`PHYSICAL_TENSOR_CONVERSION_UNSUPPORTED`	Inspect the requested edge conversion and the admitted physical path. Current same-process lowering rejects explicit non-default conversion requests; insert a suitable converter operator and declare compatible producer/consumer layouts.

A warning is not automatically a rejected plan, and a nominal stability estimate is not a measured performance result. Once compilation succeeds, inspect partitions, queue recipes, Tensor bounds, and placement. Keep that plan with the run log so subsequent measurements refer to the same accepted graph.

Separate startup failure from a compute-path error

For asynchronous execution, a returned session does not prove that discovery, preparation, and arming all succeeded. From an external control thread, join with session.wait(), then inspect session.termination() and session.diagnostics(). Before terminal completion, the diagnostic prefix may still grow. A completion counter or a final emitted frame does not rule out a cleanup failure.

Inside an operator, check errors where they occur: input access, output loans, Tensor writer TensorOutputWriteGuard::commit(), publication through Output<Tensor>::emit(), imports, and CUDA calls. Propagate the error through the callback's result instead of continuing with an invalid object. For a stalled path, determine whether it is waiting for an input selection, output credit, retained history, or GPU completion before increasing a limit.

Use Managed Run Control Example to reproduce startup, cooperative completion, and failure outcomes. Neither wait() nor session destruction force-stops user code or a nonterminating GPU kernel; callbacks must cooperate with cancellation.

Measure the work and the waits separately

Once the short run completes correctly, establish a baseline with fixed inputs and parameters. Measure throughput and end-to-end latency over the same interval, and separate warm-up from steady-state work. Check capture timestamps and clock comparability before interpreting a capture-to-display interval.

For GPU work, launch on the admitted context.cuda_stream() and measure device execution with CUDA events or a profiler. A host timer around a kernel launch mainly measures submission; it does not establish how long the kernel ran. Keep kernel execution, explicit device-to-device copies, host transfers, and synchronization visible as separate costs.

For queue behavior, correlate the symptom with the selected samples and retained ownership. Fan-out or temporal history can keep Tensor slots alive after a producer has returned. A Tensor handle copy, a queue operation, or an event wait is not evidence that payload bytes moved. Trace backing ownership and instrument actual copy calls before describing a path as copy-free.

At an external boundary, inspect the graph-side path and standalone client separately. Record whether the admitted kCopyIfNeeded policy stages the payload, whether the receiver imports or copies backing, and whether the final consumer makes a host copy. Use matching endpoint catalogs and valid ownership/readiness evidence on both sides. Descriptor-Tensor External Topics and Standalone ChannelRuntime Clients explain those boundaries.

Make one bounded change and verify its effect

Choose a change that addresses the evidence: remove an unnecessary copy, shorten a retained loan, change a selector to match the intended sampling policy, or provide a justified queue or Tensor bound. Recompile, compare the accepted plan, and rerun the same reproduction. Check correctness, termination, latency, throughput, and memory together; a throughput improvement that silently drops required samples is a different behavior.

Keep the useful before/after measurements and the reason for the final declaration near the application. Tracing the failure remains the primary investigation.

Related Examples

Automatic Partitioning Example isolates plan inspection; its compile-only modes are not runtime benchmarks.
Rate-Aware Ports and Tensor Pool Capacity Example exposes rate, queue, and allocation evidence.
Managed Run Control Example exercises run outcomes and failure handling.
CUDA Green Context Example shows explicit placement and stream use.
Inference Example provides a focused inference and Tensor-path reproduction.
Descriptor-Tensor External Topic Example exposes metadata and payload handles at an external Tensor boundary.
HoloMQ Example runs the two transport sides as separate processes.

❯