View source for Tritonserver support for NVIDIA Jetson Platforms

<seo title="Tritonserver support | Triton | NVIDIA Jetson | RidgeRun" titlemode="replace" keywords="GStreamer, Linux SDK, Linux BSP,  Embedded Linux, Device Drivers, NVIDIA, Jetson, TX1, TX2, Jetson AGX Xavier, Jetson TX1, Jetson TX2, Embedded Linux driver development, Linux Software development, Embedded Linux SDK, Embedded Linux Application development, GStreamer Multimedia Framework, Xavier, AI, Deep Learning, Tritonserver, Triton, Tritonserver support, JetPack 4.6, Triton client, Tritonserver deployment, triton inference server, kubernetes, kubectl, kubelet, kubeadm, docker" description="This RidgeRun developer wiki guide is about Tritonserver, how to run and test the server on a JetPack."></seo>

== Introduction ==

From the official Triton documentation: The Triton Inference Server provides a cloud inferencing solution optimized for both CPUs and GPUs. The server provides an inference service via an HTTP or GRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. For edge deployments, Triton Server is also available as a shared library with an API that allows the full functionality of the server to be included directly in an application.

Here you will find how to run and test the server on a JetPack step by step. 

== Native support ==

Native support for JetPack is supported in the latest releases of tritonserver. Latest release at the moment of writing is 2.16, which supports JetPack 4.6, with the TensorFlow 2.6.0, TensorFlow 1.15.5, TensorRT 8.0.1.6, and OnnxRuntime 1.8.1, you can find more information here: https://github.com/triton-inference-server/server/releases/tag/v2.16.0#Jetson_Jetpack_Support. 

In order to use this release or any others, you can download the tarball with JetPack support from the release downloads: https://github.com/triton-inference-server/server/releases/download/v2.16.0/tritonserver2.16.0-jetpack4.6.tgz. This tarball includes the tritonserver as well as the client applications. 

=== Steps to use Triton server ===

1. Install the SDK components

In order to use tritonserver with all its capabilities for the models, you should first install all the SDK components available for the board on JetPack 4.6

2. Install Triton server dependencies

<syntaxhighlight lang=bash>
apt-get update
apt-get install -y --no-install-recommends \
    software-properties-common \
    autoconf \
    automake \
    build-essential \
    cmake \
    git \
    libb64-dev \
    libre2-dev \
    libssl-dev \
    libtool \
    libboost-dev \
    libcurl4-openssl-dev \
    rapidjson-dev \
    patchelf \
    zlib1g-dev
</syntaxhighlight>

3. Download a models repository

This step is optional if you already have a models repository, if you don't have one for testing purposes, you can get some by using the step located here: https://github.com/triton-inference-server/server/blob/main/docs/quickstart.md#create-a-model-repository

For this you need to:
<syntaxhighlight lang=bash>
git clone https://github.com/triton-inference-server/server.git tritonserver-src
cd tritonserver-src && cd docs/examples
./fetch_models.sh
export MODEL_PATH=$PWD/model_repository
</syntaxhighlight>

The models will then be located under the model_repository folder in the same directory as the executed script, exported in the MODEL_PATH variable. 

4. Download the triton server

The tarball will include the executable and needed shared libraries, alongside the tritonserver backends

<syntaxhighlight lang=bash>
wget https://github.com/triton-inference-server/server/releases/download/v2.16.0/tritonserver2.16.0-jetpack4.6.tgz
tar -xzf tritonserver2.16.0-jetpack4.6.tgz
cd tritonserver2.16.0-jetpack4.6
export BACKEND_PATH=$PWD/backends
</syntaxhighlight>

5. Execute the server

You can then execute the server this way:
<syntaxhighlight lang=bash>
./bin/tritonserver --model-repository=$MODEL_PATH --backend-directory=$BACKEND_PATH --backend-config=tensorflow,version=2
</syntaxhighlight>


And you can check is up and ok by using the ready endpoint:
<syntaxhighlight lang=bash>
curl -v localhost:8000/v2/health/ready
</syntaxhighlight>

=== Steps to use Triton client ===

1. Install all dependencies
<syntaxhighlight lang=bash>
apt-get install -y --no-install-recommends \
        curl \
        pkg-config \
        python3 \
        python3-pip \
        python3-dev

pip3 install --upgrade wheel setuptools cython && \
pip3 install --upgrade grpcio-tools numpy==1.19.4 future attrdict
</syntaxhighlight>

2. Install the python package

The python package is located inside the downloaded and uncompressed release.

You can install it by using:

<syntaxhighlight lang=bash>
python3 -m pip install --upgrade  clients/python/tritonclient-2.16.0-py3-none-any.whl[all]
</syntaxhighlight>

3. Using one of the examples 

There are some python examples located here: https://github.com/triton-inference-server/client/tree/main/src/python/examples

These examples are included inside the downloaded release too.

You can run for example to perform detection on an image:

<syntaxhighlight lang=bash>
python3 ./examples/image_client.py -m densenet_onnx -c 3 -s VGG /home/nvidia/client/src/python/cistus1.jpg --url localhost:8000 --protocol HTTP
</syntaxhighlight>

== Docker support ==

If you search the documentation, you will find that there are docker images on NGC for tritonserver, named <code>nvcr.io/nvidia/tritonserver:<xx.yy>-py3</code>. These images don't work for JetPack because are built for Windows or Ubuntu for PC, therefore you should need to create one by using the following Dockerfile. 

<syntaxhighlight lang=docker>

FROM nvcr.io/nvidia/l4t-ml:r32.6.1-py3

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        software-properties-common \
        autoconf \
        automake \
        build-essential \
        cmake \
        git \
        libb64-dev \
        libre2-dev \
        libssl-dev \
        libtool \
        libboost-dev \
        libcurl4-openssl-dev \
        rapidjson-dev \
        patchelf \
        zlib1g-dev && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /tritonserver

RUN wget https://github.com/triton-inference-server/server/releases/download/v2.16.0/tritonserver2.16.0-jetpack4.6.tgz && \
    tar -xzf tritonserver2.16.0-jetpack4.6.tgz && \
    rm tritonserver2.16.0-jetpack4.6.tgz

ENV LD_LIBRARY_PATH=/tritonserver/backends/tensorflow1:$LD_LIBRARY_PATH

# tritonserver looks in /opt/tritonserver/backends by default
RUN mkdir -p /opt/tritonserver/backends && cp -r ./backends/* /opt/tritonserver/backends/


ENTRYPOINT ["/tritonserver/bin/tritonserver"]
CMD ["--help"]
</syntaxhighlight>

You may do so by appending that into a Dockerfile file. And then:

<syntaxhighlight lang=bash>
sudo docker build . -t tritonserver
</syntaxhighlight>

Then you can run a container with that image

<syntaxhighlight lang=bash>
sudo docker run --gpus=1 -p8000:8000 -p8001:8001 -p8002:8002 -v/home/nvidia/server/docs/examples/model_repository:/models tritontest --model-repository=/models
</syntaxhighlight>

Then you can run any client as described above: https://developer.ridgerun.com/wiki/index.php?title=Tritonserver_support_for_NVIDIA_Jetson_Platforms#Steps_to_use_Triton_client

== Kubernetes support ==
A full guide on how to run Tritonserver with kubernetes is located in here: https://developer.nvidia.com/blog/deploying-nvidia-triton-at-scale-with-mig-and-kubernetes/

Here is a summary and more detailed steps on how to do that. 

The first thing that you should do is install kubernetes, in this case on Ubuntu 18.04.

=== Installing kubectl ===
The guide to install kubernetes Kubectl is located in here: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management

But overall, you should run:
<syntaxhighlight lang=bash>
# Install dependencies
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

# Add kubernetes to package registry
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Install kubectl
sudo apt-get update
sudo apt-get install -y kubectl
sudo apt-mark hold kubectl
</syntaxhighlight>

=== Installing kubeadmin ===
There are some tools necessary to run kubernetes, these are located here: https://kubernetes.io/docs/tasks/tools/

We should install kubelet and kubeadm, to work alongside kubectl. As stated here https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm, you should use the following commands: 

<syntaxhighlight lang=bash>
# Install kubelet and kubeadm
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
</syntaxhighlight>

After installing, you will need to configure multiple things:

==== Disable swap ====

<syntaxhighlight lang=bash>
sudo swapoff -a  
</syntaxhighlight>

==== Kubelet cgroup ====

You can find more information here: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configuring-a-cgroup-driver

The reason to make this modification is because the kubelet and the container engine (docker or containerd) need to be running under the same group. For this, you can configure the kubelet to run on the same group as docker for example, which runs under cgroupfs. To do this:

<syntaxhighlight lang=bash>
sudo vim /etc/default/kubelet
</syntaxhighlight>

And add 
<syntaxhighlight lang=bash>
KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs"
</syntaxhighlight>

==== Kubectl for non root user ====

You can find more information here:

First run once the kubeadm as root:
<syntaxhighlight lang=bash>
sudo kubeadm init
</syntaxhighlight>

Then create the configuration in your home directory:
<syntaxhighlight lang=bash>
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
</syntaxhighlight>

Then stop the kubeadm
<syntaxhighlight lang=bash>
sudo kubeadm reset -f
</syntaxhighlight>

==== Pod network add-on ====

You will need a pod network add-on in order for the master node to start correctly, more information here: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network 

In the link, you can find a list of add-ons available, but in this issue thread is one that worked: https://github.com/kubernetes/kubernetes/issues/48798

To install it and use it: 

<syntaxhighlight lang=bash>
sudo kubeadm init
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
</syntaxhighlight>

=== Tritonserver deployment ===


{{ContactUs}}

[[Category:GStreamer]][[Category:Jetson]][[Category:JetsonNano]][[Category:JetsonTX2]][[Category:NVIDIA Xavier]]