R2Inference - ONNXRT ACL

Description

The ONNXRT ACL backend is an extension of the ONNXRT backend. It is based on the ACL Execution Provider available in ONNX Runtime. ACL stands for Arm Compute Library, which is an extensive library for optimized functions for the Arm Cortex-A family of CPU processors and the Arm Mali family of GPUs. The library allows accelerating common computer vision and machine learning operations through technologies like Arm Neon and OpenCL.

Installation

The R2Inference ONNXRT backend depends on the C/C++ ONNX Runtime API. The following installation process is based on the ONNX Runtime source code.

Jetson Xavier (native build)

The ONNX Runtime C/C++ API can be installed from source code. This section presents instructions to build from the source files based on the official installation guide for ACL execution provider.

The following commands are based on the instruction of the official guide presented before:

Build and install Arm Compute Library. Highly recommended to check build options at: https://arm-software.github.io/ComputeLibrary/latest/index.xhtml#S3_how_to_build for further information on how to configure the build system.

git clone https://github.com/Arm-software/ComputeLibrary.git ArmComputeLibrary
git checkout v19.05
cd ComputeLibrary
sudo apt install scons
sudo apt install g++-arm-linux-gnueabihf

#Jetson Xavier (be careful which arch option you select, it has to match the CPU complex of your system)
scons -j8 arch=arm64-v8.2-a build=native debug=0 asserts=0 os=linux examples=0 Werror=1 opencl=1 neon=1 cppthreads=1

export CPATH=~/ArmComputeLibrary/include/:~/ArmComputeLibrary/
export LD_LIBRARY_PATH=~/ArmComputeLibrary/build/

Install ONNX Runtime with ACL support

#This step is needed if you require to use ONNX tools included in onnxruntime repo
pip3 install onnx

#Update cmake
sudo apt install build-essential libssl-dev
wget https://github.com/Kitware/CMake/releases/download/v3.16.5/cmake-3.16.5.tar.gz
tar -zxvf cmake-3.16.5.tar.gz
cd cmake-3.16.5
./bootstrap
make 
sudo make install

#Build onnx runtime
git clone --recursive https://github.com/Microsoft/onnxruntime -b v1.4.0
cd onnxruntime
./build.sh --config RelWithDebInfo --update --build --parallel --build_shared_lib --use_acl ACL_1905
cd build/Linux/RelWithDebInfo
sudo make install
sudo cp libonnxruntime.so.1.4.0 /usr/lib/aarch64-linux-gnu/libonnxruntime.so.1.4.0
sudo ln -s /usr/lib/aarch64-linux-gnu/libonnxruntime.so.1.4.0 /usr/lib/aarch64-linux-gnu/libonnxruntime.so

API

Some documentation of the C/C++ ONNX Runtime API can be found in onnxruntime_c_api.h and onnxruntime_cxx_api.h. The R2Inference uses the C++ API which is mostly a wrapper for the C API. R2Inference provides a high-level abstraction for loading the ONNX model, creating the ONNX Runtime session, and executing the inference of the model. We recommend looking at the examples available for this backend in the r2inference repository to get familiar with the R2Inference interface. R2Inference also abstracts many options available in the ONNX Runtime C++ API through the "IParameters class".

The parameters listed below are currently supported:

Property	C++ API Counterpart	Value	Operation	Description
logging-level	OrtLoggingLevel	Integer	R/W and Write before start	Level of log information of the ONNX Runtime session.
log-id	N/A	String	R/W and Write before start	String identification of the ONNX Runtime session.
intra-num-threads	SessionOptions::SetIntraOpNumThreads()	Integer	R/W and Write before start	Number of threads to parallelize execution within model nodes.
graph-optimization-level	GraphOptimizationLevel	Integer	R/W and Write before start	Graph optimization level of the ONNX Runtime session.

Previous: Supported_backends/ONNXRT

Index

Next: Supported_backends/ONNXRT OpenVINO