212
edits
Tritonserver support for NVIDIA Jetson Platforms: Difference between revisions
Tritonserver support for NVIDIA Jetson Platforms (view source)
Revision as of 16:37, 26 November 2021
, 26 November 2021→Native support
Rgutierrez (talk | contribs) (Created page with "== Introduction == == Native support == == Yocto support == == Docker support == == Kubernetes support == A full guide on how to run Tritonserver with kubernetes is loca...") |
Rgutierrez (talk | contribs) m (→Native support) |
||
| Line 4: | Line 4: | ||
== Native support == | == Native support == | ||
Native support for JetPack is supported in the latest relases of tritonserver. Latest release at the moment of writing is 2.16, which supports JetPack 4.6, with the TensorFlow 2.6.0, TensorFlow 1.15.5, TensorRT 8.0.1.6 and OnnxRuntime 1.8.1, you can find more information here: https://github.com/triton-inference-server/server/releases/tag/v2.16.0#Jetson_Jetpack_Support. | |||
In order to use this release or any others, you can download the tarball with JetPack support from the release downloads: https://github.com/triton-inference-server/server/releases/download/v2.16.0/tritonserver2.16.0-jetpack4.6.tgz. This tarball includes the tritonserver as well as the client applications. | |||
=== Steps to use Triton server === | |||
1. Install the SDK components | |||
In order to use tritonserver with all it's capabilities for the models, you should first install all the SDK components available for the board on JetPack 4.6 | |||
2. Install Triton server dependencies | |||
<syntaxhighlight lang=bash> | |||
apt-get update | |||
apt-get install -y --no-install-recommends \ | |||
software-properties-common \ | |||
autoconf \ | |||
automake \ | |||
build-essential \ | |||
cmake \ | |||
git \ | |||
libb64-dev \ | |||
libre2-dev \ | |||
libssl-dev \ | |||
libtool \ | |||
libboost-dev \ | |||
libcurl4-openssl-dev \ | |||
rapidjson-dev \ | |||
patchelf \ | |||
zlib1g-dev | |||
</syntaxhighlight> | |||
3. Download a models repository | |||
This step is optional if you already have a models repository, if you don't have one for testing purposes, you can get some by using the step located in here: https://github.com/triton-inference-server/server/blob/main/docs/quickstart.md#create-a-model-repository | |||
For this you need to: | |||
<syntaxhighlight lang=bash> | |||
git clone https://github.com/triton-inference-server/server.git tritonserver-src | |||
cd tritonserver-src && cd docs/examples | |||
./fetch_models.sh | |||
export MODEL_PATH=$PWD/model_repository | |||
</syntaxhighlight> | |||
The models will then be located under the model_repository folder in the same directory as the executed script, exported in the MODEL_PATH variable. | |||
4. Download the triton server | |||
The tarball will include the executable and needed shared libraries, alongside the tritonserver backends | |||
<syntaxhighlight lang=bash> | |||
wget https://github.com/triton-inference-server/server/releases/download/v2.16.0/tritonserver2.16.0-jetpack4.6.tgz | |||
tar -xzf tritonserver2.16.0-jetpack4.6.tgz | |||
cd tritonserver2.16.0-jetpack4.6 | |||
export BACKEND_PATH=$PWD/backends | |||
</syntaxhighlight> | |||
5. Execute the server | |||
You can then execute the server this way: | |||
<syntaxhighlight lang=bash> | |||
./bin/tritonserver --model-repository=$MODEL_PATH --backend-directory=$BACKEND_PATH --backend-config=tensorflow,version=2 | |||
</syntaxhighlight> | |||
And you can check is up and ok by using the ready endpoint: | |||
<syntaxhighlight lang=bash> | |||
curl -v localhost:8000/v2/health/ready | |||
</syntaxhighlight> | |||
=== Steps to use Triton client === | |||
== Yocto support == | == Yocto support == | ||