R2Inference - ONNXRT OpenVINO

Description

The ONNXRT OpenVINO backend is an extension of the ONNXRT backend. This backend is based on Intel's OpenVINO toolkit support available in ONNX Runtime. OpenVINO offers a boost in performance through optimizations for common computer vision and deep learning workloads on Intel's hardware (CPUs, Movidius USB sticks, MyriadX VPUs, and FPGAs).

Installation

The R2Inference ONNXRT backend depends on the C/C++ ONNX Runtime API. The following installation process is based on the ONNX Runtime source code.

x86

The ONNX Runtime C/C++ API can be installed from source code. This section presents instructions to build from the source files based on the official installation guide for OpenVINO execution provider.

The following commands are based on the instruction of the official guide presented before:

Build and install OpenVINO. Follow the instructions in the OpenVINO documentation, the instructions are detailed and will bring you to complete setup for the OpenVINO toolkit, it is highly recommended to test the sample demos available to make sure everything is set up correctly.

An important aspect of the OpenVINO installation is to make sure that you have run the following script:

source <openvino_install_directory>/bin/setupvars.sh

Install ONNX Runtime with OpenVINO support

#This step is needed if you require to use ONNX tools included in onnxruntime repo
pip3 install onnx

#Build onnx runtime
git clone --recursive https://github.com/Microsoft/onnxruntime -b v1.4.0
cd onnxruntime
./build.sh --config RelWithDebInfo --build_shared_lib --parallel --use_openvino CPU_FP32 #default hardware execution
cd build/Linux/RelWithDebInfo
sudo make install
sudo cp libonnxruntime.so.1.4.0 /usr/lib/aarch64-linux-gnu/libonnxruntime.so.1.4.0
sudo ln -s /usr/lib/aarch64-linux-gnu/libonnxruntime.so.1.4.0 /usr/lib/aarch64-linux-gnu/libonnxruntime.so

API

Some documentation of the C/C++ ONNX Runtime API can be found in onnxruntime_c_api.h and onnxruntime_cxx_api.h. The R2Inference uses the C++ API which is mostly a wrapper for the C API. R2Inference provides a high-level abstraction for loading the ONNX model, creating the ONNX Runtime session, and executing the inference of the model. We recommend looking at the examples available for this backend in the r2inference repository to get familiar with the R2Inference interface. R2Inference also abstracts many options available in the ONNX Runtime C++ API through the "IParameters class".

The parameters listed below are currently supported:

Property	C++ API Counterpart	Value	Operation	Description
logging-level	OrtLoggingLevel	Integer	R/W and Write before start	Level of log information of the ONNX Runtime session.
log-id	N/A	String	R/W and Write before start	String identification of the ONNX Runtime session.
intra-num-threads	SessionOptions::SetIntraOpNumThreads()	Integer	R/W and Write before start	Number of threads to parallelize execution within model nodes.
graph-optimization-level	GraphOptimizationLevel	Integer	R/W and Write before start	Graph optimization level of the ONNX Runtime session.
hardware-id	HardwareId	String	R/W and Write before start	OpenVINO hardware device id

OpenVINO setup for hardware devices

Neural Compute Stick 2

The following instructions were taken from the official OpenVINO documentation:

#Add the current Linux user to the users group:
sudo usermod -a -G users "$(whoami)"

#Log out and log in for it to take effect.

#To perform inference on Intel® Neural Compute Stick 2, install the USB rules as follows:
sudo cp /opt/intel/openvino/inference_engine/external/97-myriad-usbboot.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo ldconfig

Previous: Supported_backends/ONNXRT ACL

Index

Next: Examples

You can try to verify the setup by running Security Barrier Camera Demo with the MYRIAD hardware option.

cd /opt/intel/openvino/deployment_tools/demo
./demo_security_barrier_camera.sh -d MYRIAD

❯