User Guide-Custom Application Development
🚧 Documentation under development
The PVA ISP for NVIDIA Jetson guide is currently under active development. Some sections may be incomplete or change without notice.
Questions? Contact RidgeRun or email to support@ridgerun.com.
Custom Application Development
PVA ISP provides a C++ library that allows developers to build custom image processing pipelines accelerated by NVIDIA's Programmable Vision Accelerator (PVA).
Unlike the GStreamer plug-in, which focuses on multimedia pipeline integration, the library API provides direct access to the ISP processing graph and individual image processing stages.
This approach is recommended when applications require custom processing flows, proprietary software architectures, or fine-grained control over ISP behavior.
Complete API documentation, class references, and implementation details are available in the online documentation
PVA ISP Architecture
PVA ISP follows a graph-based processing model. Applications build a processing graph by connecting independent stages. Each stage performs one operation, such as decompanding, black level correction, white balance, demosaicing, tone mapping, or format conversion, and then forwards its output to the next stage.
At a high level, responsibilities are divided as follows:
- The application defines the graph structure.
- Each stage defines its supported inputs, outputs, and configurable properties.
- The pipeline negotiates formats, allocates resources, and executes the graph.
The library manages:
- Pipeline execution
- Frame negotiation
- Buffer allocation
- PVA resource management
- Runtime synchronization
Graph-Based Architecture
PVA ISP separates graph construction from graph execution.
Applications are responsible for building the processing graph by creating stages and defining the connections between them.
During graph construction, the application:
- Creates the pipeline object
- Creates stage objects
- Registers stages in the pipeline
- Creates frames that represent external inputs and outputs
- Connects stages and external frames through links
After the graph has been described, the library takes over the execution details. The Pipeline object becomes responsible for:
- Format negotiation
- Internal frame creation
- Resource allocation
- Stage configuration
- Runtime scheduling and processing
This separation allows applications to customize the ISP flow while reusing a common execution engine.
Core Concepts
Before integrating PVA ISP into an application, it is useful to understand the relationship between the main library concepts.
At a high level:
- A Pipeline owns and executes the graph.
- A Stage represents one processing block in that graph and expose pads.
8 A Pad' define how data enters and exits a stage within the pipeline.
- A Link connects pads between stages or between a stage and an external frame.
- A FrameSpec describes what a frame looks like.
- A Frame stores the actual runtime buffer that flows through the graph.
FrameSpec
A FrameSpec describes the structure of an image buffer, including its dimensions, pixel format, data type, bit depth, and memory layout. During pipeline creation, FrameSpec objects are mainly used during negotiation, when the pipeline determines which format each connection will use..
Frame
A Frame represents an image buffer at runtime. It contains the memory reference and the associated FrameSpec needed to transport image data or auxiliary metadata through the graph.
Applications typically create external input and output frames, for example:
- The Bayer RAW input frame
- The final RGB or NV12 output frame
Internal intermediate frames are usually created and managed by the pipeline.
Stage
A Stage represents an individual image processing operation. Each ISP algorithm is implemented as a stage for example, Decompanding, Black Level Correction, White Balance, or Demosaicing.
Each stage defines:
- Which input and output formats it supports
- Which pads it exposes
- Which configuration properties can be changed
- How the stage prepares and executes its internal PVA workload
Link
A Link represents a connection between two stage pads.
Links define how data flows through the processing graph and are created by the application during graph construction.
Each link stores the upstream and downstream stages, the connected pads, the negotiated FrameSpec, and the runtime frame buffer. Links are automatically negotiated by the Pipeline during initialization.
Pipeline
The Pipeline owns and executes the processing graph. It manages stage connections, negotiates frame formats, allocates resources, schedules execution, and collects runtime statistics.
A single application may create one or multiple pipelines depending on the system requirements.
Pads
Pads are the named connection points exposed by stages.
By default, every stage exposes one input pad (input) and one output pad (output). Some stages also expose additional pads to exchange auxiliary data such as gains or scene keys.
Examples include:
- BlackLevelStage → gains
- WhiteBalanceStage → gains
- DemosaicStage → scene_key
- GlobalToneMappingStage → scene_key
Pads allow stages to exchange metadata and auxiliary information in addition to image buffers.
Pipeline Lifecycle
A typical PVA ISP pipeline follows the lifecycle shown below:
Each phase has a specific responsibility:
- addFrame() creates external pipeline buffers.
- add() registers processing stages.
- link() defines graph connections.
- init() creates the execution context.
- negotiate() validates formats and creates internal graph buffers.
- allocate() allocates memory.
- configure() prepares each stage for execution.
- process() executes the graph.
From an application point of view, integration usually follows this model:
- Define the input and output buffers the application owns.
- Create a Pipeline object.
- Create the required Stage objects.
- Register frames and stages in the pipeline.
- Connect them using links.
- Initialize, negotiate, allocate, and configure the pipeline.
- Feed the input frame and process the graph.
- Read the results from the output frame.
Building ISP Pipelines
PVA ISP pipelines are constructed by connecting stages together in the desired order. <
A complete Bayer-to-NV12 pipeline may include:
Developers can also construct smaller pipelines containing only the algorithms required by their application.
Examples include:
- RAW → Black Level Correction
- RAW → White Balance
- RAW → Demosaicing
This flexibility allows applications to implement only the functionality they require.
Each ISP algorithm can be found in: PVA_ISP_for_NVIDIA_Jetson/User_Guide/Supported_ISP_algorithms
Side-Channel Data Flow
In addition to image buffers, stages may exchange auxiliary metadata through dedicated pads.
Examples include:
- Auto White Balance gains
- Scene Key metadata
These side channels are represented as graph frames and participate in the same negotiation process as image buffers.
This mechanism allows stages to exchange dynamic information without modifying the primary image stream.
Buffer Categories
Not every runtime allocation in PVA ISP is the same. For integration purposes, it is useful to group runtime data into four categories:
| Category | Description | Examples |
|---|---|---|
| Graph Frames | Buffers exchanged through links during normal pipeline execution. | Bayer images, RGB images, NV12 images, histogram outputs, AWB gains, scene key metadata |
| Stage Assets | Static stage-owned resources prepared during configuration. | Decompand LUTs, gamma LUTs, demosaic LUTs |
| Persistent State | Resources preserved across multiple processed frames. | Global Tone Mapping temporal state |
| Scratch Buffers | Temporary internal resources used by a stage while processing. | Implementation-specific temporary working buffers |
High-Level Integration Example
The following simplified example illustrates the relationship between the main objects:
isp::Pipeline pipeline;
// Application-owned external frames
isp::Frame input_frame(input_spec);
isp::Frame output_frame(output_spec);
// Stages
auto decompand = std::make_unique<isp::DecompandStage>("decompand0");
auto black_level = std::make_unique<isp::BlackLevelStage>("black_level0");
auto demosaic = std::make_unique<isp::DemosaicStage>("demosaic0");
// Register stages and frames
pipeline.addFrame("input", &input_frame);
pipeline.addFrame("output", &output_frame);
pipeline.add(std::move(decompand));
pipeline.add(std::move(black_level));
pipeline.add(std::move(demosaic));
// Connect the graph
pipeline.link("input", "decompand0", "input");
pipeline.link("decompand0", "output", "black_level0", "input");
pipeline.link("black_level0", "output", "demosaic0", "input");
pipeline.link("demosaic0", "output", "output");
// Prepare execution
pipeline.init();
pipeline.negotiate();
pipeline.allocate();
pipeline.configure();
// Process one frame
pipeline.process();
This example is intentionally simplified, but it shows the expected integration pattern:
- The application owns the external buffers
- The pipeline owns the graph execution
- The stages define the processing behavior
- The links define the data flow
Available Classes
The library exposes the following primary classes:
| Class | Purpose |
|---|---|
| Pipeline | Graph construction and execution |
| Frame | Runtime image buffers |
| FrameSpec | Frame metadata definition |
| Stage | Base stage interface |
| DecompandStage | RAW decompanding |
| BlackLevelStage | Black level correction |
| HistogramStage | Histogram computation |
| WhiteBalanceStage | White balance processing |
| DemosaicStage | Bayer reconstruction |
| GlobalToneMappingStage | Dynamic range compression |
| GammaToneMappingStage | Gamma correction |
| NV12ConversionStage | RGB to NV12 conversion |
