View source for NVIDIA Jetson Generative AI Lab RidgeRun Exploration

<seo title="NVIDIA Jetson Generative AI Lab | RidgeRun" titlemode="replace" metakeywords="GStreamer, Embedded Linux, Device Drivers, NVIDIA, Jetson, TX1, TX2, Jetson AGX Xavier, Jetson Nano, Jetson Orin, Orin Nano, Orin NX, Embedded Linux driver development, Linux Software development, Embedded Linux SDK, Embedded Linux Application development, GStreamer Multimedia Framework, Xavier, AI, Deep Learning, RidgeRun, camera drivers, ridgerun support, embedded systems, embedded, multimedia applications, computer vision, AI, deep learning, machine learning, artificial intelligence, jetpack, Generative AI, Generative AI Lab, NVIDIA Jetson Generative AI Lab, NVIDIA Jetson" metadescription="Read more about NVIDIA Jetson Generative AI Lab RidgeRun Exploration."></seo>

<br>
{{UnderConstruction}}
<br>

= Introduction =

This wiki explores the following NVIDIA Jetson Generative AI Lab: [https://www.jetson-ai-lab.com/tutorial_text-generation.html Text Generation], [https://www.jetson-ai-lab.com/tutorial_stable-diffusion.html Image Generation], [Text + Vision https://www.jetson-ai-lab.com/tutorial_llava.html] and [https://www.jetson-ai-lab.com/vit/tutorial_nanosam.html NanoSAM]. The idea is to guide the user and developer on utilizing the Jetson Generative AI Lab tool. Here we explain how to run some examples, what dependencies are needed, and how to use this powerful tool. 

= Install the dependencies =

The NVIDIA Jetson Generative AI Lab is only available to run on the following Jetson boards:

* Jetson AGX Orin 64GB Developer Kit
* Jetson AGX Orin 32GB Developer Kit
* Jetson Orin Nano Developer Kit

=== JetPack 5.1.2 NVIDIA Installation ===

1. Download and run the [https://docs.nvidia.com/sdk-manager/download-run-sdkm/index.html NVIDIA SDK Manager]

2. Install JetPack 5.1.2 on the Orin AGX DevKit board.
:*Disable the host installation (1).
:*Select the Jetson board (2).
:*Select JetPack 5.1.2 (3)
:*Continue the installation process.

<br>
[[File:Jetpack generative ai.jpg|center|600px]]
<br>

3. Select all NVIDIA packages.

<br>
[[File:Jetpack2 generative ai.jpg|center|600px]]
<br>

=== Install Jetson Generative AI Lab dependencies ===

1. Check if <code>docker</code> is already installed:

<syntaxhighlight lang=bash>
sudo docker version

Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~20.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/arm64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~20.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:        
 runc:
  Version:          1.1.7-0ubuntu1~20.04.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:
</syntaxhighlight>

If there is no output from the <code>sudo docker version</code> command, install docker by following the instructions in [https://docs.docker.com/engine/install/ubuntu/Install Docker Engine on Ubuntu]:

:1.1. Uninstall all conflicting packages:
<syntaxhighlight lang=bash>
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
</syntaxhighlight>

:1.2. Set up Docker's apt repository:
<syntaxhighlight lang=bash>
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
</syntaxhighlight>

:1.3. Install the latest Docker version:
<syntaxhighlight lang=bash>
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
</syntaxhighlight>

:1.4. Run the <code>hello-world</code> image to verify the Docker installation:
<syntaxhighlight lang=bash>
sudo docker run hello-world
</syntaxhighlight>

2. Check if the <code>nvidia-container</code> packages are already installed:

<syntaxhighlight lang=bash>
apt list | grep nvidia-container

libnvidia-container-tools/stable,now 1.10.0-1 arm64 [installed]
libnvidia-container0/stable,now 0.11.0+jetpack arm64 [installed]
libnvidia-container1/stable,now 1.10.0-1 arm64 [installed]
nvidia-container-runtime/stable,now 3.9.0-1 all [installed]
nvidia-container-toolkit/stable,now 1.11.0~rc.1-1 arm64 [installed]
</syntaxhighlight>

If there is no output from the <code>apt list | grep nvidia-container</code> command, install the following apt package:

<syntaxhighlight lang=bash>
sudo apt install nvidia-container-toolkit nvidia-container-runtime
</syntaxhighlight>

3. Check if <code>CUDA</code> is already installed

<syntaxhighlight lang=bash>
nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Sun_Oct_23_22:16:07_PDT_2022
Cuda compilation tools, release 11.4, V11.4.315
Build cuda_11.4.r11.4/compiler.31964100_0
</syntaxhighlight>

If there is no output, install CUDA by running the following command:

<syntaxhighlight lang=bash>
sudo apt install cuda-toolkit-11-4
</syntaxhighlight>

Modify the following environment variables:

<syntaxhighlight lang=bash>
PATH="/usr/local/cuda/bin:$PATH"
LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"
</syntaxhighlight>

The above environment variables changes need to be applied every time a terminal is opened.

3. Install the following apt dependencies packages:

<syntaxhighlight lang=bash>
sudo apt-get update && sudo apt install git python3-pip
</syntaxhighlight>

4. Clone the <code>jetson-containers</code> repository and install the requirements:

<syntaxhighlight lang=bash>
git clone --depth=1 https://github.com/dusty-nv/jetson-containers
cd jetson-containers
pip3 install -r requirements.txt
</syntaxhighlight>

5. Modify the <code>/etc/docker/daemon.json</code> file to enable the NVCC compiler and GPU:

<syntaxhighlight lang=bash>
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },

    "default-runtime": "nvidia"
}
</syntaxhighlight> 

Restart the Docker service:

<syntaxhighlight lang=bash>
sudo systemctl restart docker
</syntaxhighlight> 

Check if the change was successfully applied:

<syntaxhighlight lang=bash>
sudo docker info | grep 'Default Runtime'
Default Runtime: nvidia
</syntaxhighlight>

== Additional Docker set up steps ==

=== To Run As Non-Root User (No Sudo) ===

<syntaxhighlight lang=bash>
# Create the docker group
sudo groupadd docker

# Add yourself to the group
sudo usermod -aG docker $USER

# To avoid re-logging
newgrp docker

# Test proper installation
docker ps
</syntaxhighlight>

=== Move Docker Out Of Root Filesystem Partition ===

Docker can create many files that can quickly fill the root filesystem partition. To prevent this from happening follow the next steps. '''NOTE''': The following steps only have significant effects when there is another disc, for example, an NVME in the Jetson. If there is no other disc, these steps are not needed.

1. Verify if the <code>/var/lib/docker/</code> directory exists:

<syntaxhighlight lang=bash>
ls /var/lib/docker/
</syntaxhighlight>

If not, create it by running the following command:

<syntaxhighlight lang=bash>
docker run hello-world
</syntaxhighlight>

2. Copy the Docker directory to another path, for example <code>/home/nvidia</code> :

<syntaxhighlight lang=bash>
sudo cp -r /var/lib/docker/ <path>
</syntaxhighlight>

3. Remove the original Docker directory:

<syntaxhighlight lang=bash>
sudo rm -rf /var/lib/docker/
</syntaxhighlight>

4. Create a symbolic link at the original root location to point at the new home location:

<syntaxhighlight lang=bash>
cd /var/lib/
sudo ln -s <path>/docker/ docker
</syntaxhighlight>

5. Restart the Docker service:

<syntaxhighlight lang=bash>
systemctl restart docker
</syntaxhighlight>

= Run text-generation-webui tutorial =

In order to run the <code>text-generation-webui</code> example, follow the next instructions:

1. Pull or build and run the <code>text-generation-webui</code> compatible docker image

<syntaxhighlight lang=bash>
cd jetson-containers
./run.sh $(./autotag text-generation-webui)
</syntaxhighlight>

The following GIF shows the terminal output the first time the above is run:

<br>
[[File:Run text generation webui.gif| | center]]
<br>

The <code>run.sh</code> and <code>autotag</code> are in charge of finding this compatible docker image based on the example that the user wants to run. In this case, the compatible image is <code>dustynv/text-generation-webui:r35.4.1</code>. Also, they are going to run the docker container as:

<syntaxhighlight lang=bash>
docker run --runtime nvidia -it --rm --network host --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /home/nvidia/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb dustynv/text-generation-webui:r35.2.1
</syntaxhighlight>

2. Open the browser and access <code>http://<IP_ADDRESS>:7860</code>. For example:

<syntaxhighlight lang=bash>
http://192.168.0.5:7860
</syntaxhighlight>

3. Now, a model has to be downloaded and loaded into the generative example. 
There are two ways to download the model(s): from within the web UI, or using <code>download-model.py</code> script inside the docker container.

== Download a model from within the web UI ==

1. On the web UI, click on the <code>Model</code> tab and go to the <code>Download model or LoRA</code> subsection.

2. Look for a text generation model in [https://huggingface.co/models?pipeline_tag=text-generation&sort=trending Hugging Face Hub] and enter <code>username/model</code> path. The path can be copied from the Hub into the clipboard. The [http://www.jetson-ai-lab.com/tutorial_text-generation.html#fn:1 Tutorial - text-generation-webui] already used some GGUF (GPT (Generative pre-trained transformers)-Generated Unified Format) models. According to the tutorial, the fastest <code>oobabooga</code> model to  to use the [https://github.com/dusty-nv/jetson-containers/blob/dev/packages/llm/llama_cpp llama.cpp] with 4-bit quantized GGUF models, which are:

{| class="wikitable"
|+ GGUF models for text generation
|-
! Model !! Quantization !! Memory (MB)
|-
| [https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF TheBloke/Llama-2-7b-Chat-GGUF] || llama-2-7b-chat.Q4_K_M.gguf || 5,268
|-
| [https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF TheBloke/Llama-2-13B-chat-GGUF] || llama-2-13b-chat.Q4_K_M.gguf || 8,609
|-
| [https://huggingface.co/TheBloke/LLaMA-30b-GGUF TheBloke/LLaMA-30b-GGUF] || llama-30b.Q4_K_S.gguf || 19,045
|-
| [https://huggingface.co/TheBloke/Llama-2-70b-Chat-GGUF TheBloke/Llama-2-70B-chat-GGUF] || llama-2-70b-chat.Q4_K_M.gguf || 37,655
|}

To use the models in the above table, copy the model <code>username/model</code> path, click on <code>Get file list</code>, select the model file for a particular quantization, input the file name and hit <code>Download</code> button. To check the model size, check [https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF#provided-files TheBloke/Llama-2-7B-Chat-GGUF/provided-files] section, for example.

The following GIF shows an example of how to download the model:

<br>
[[File:Download model text generation webui resize.gif| | center]]
<br>

== Load a model ==

When the model has finished downloading, click on the refresh button to list the model, and select the preferred model to use.

For a GGUF model:

* Set <code>n-gpu-layers</code> to <code>128</code>
* Set <code>n_gqa</code> to <code>8</code> if <code>Llama-2-70B</code> are used (on Jetson AGX Orin 64GB)

Then click the Load button.

The following GIF shows an example of how to download the model:

<br>
[[File:Load model text generation webui.gif| | center]]
<br>

== Chat with the AI model ==

The <code>Instruction Template</code> sub-tab on the <code>Parameters tab</code> has one parameter that needs to be changed when a Llama model fine-tuned for chat is used, such as the models specified in the [https://developer.ridgerun.com/wiki/index.php/NVIDIA_Jetson_Generative_AI_Lab_RidgeRun_Exploration#Download_a_model_from_within_the_web_UI Download a model from within the web UI] section. To perform that change:

1. Click on the <code>Parameters tab</code>.

2. Click on the <code>Instruction Template</code> sub-tab.

3. Select <code>Llama-v2</code> from the <code>Instruction Template</code> drop-down (or Vicuna, Guanaco, etc., if one of those models is used).

4. On the <code>Generation</code> tab, some settings can be changed such as the <code>max_new_tokens</code> which is related to the maximum length the AI should output per reply or randomness control parameters like [https://medium.com/@dixnjakindah/top-p-temperature-and-other-parameters-1a53d2f8d7d7 temperature and top_p].

5. Click on the <code>Chat tab</code> and under the <codeMode</code> section select the <code>Instruct</code> option.

Now, the user can chat with the AI model. [https://modal.com/docs/examples/vllm_inference#run-the-model Here] are some questions that the user can ask the AI model. Here is an example on how to run the above steps:

<br>
[[File:Text generation webui chat.gif | center]]
<br>

= Run Image generation tutorial =

To run the <code>Stable Diffusion (Image Generation)</code> example, follow the next instructions:

1. Pull or build and run the <code>stable-diffusion-webui</code> compatible docker image

<syntaxhighlight lang=bash>
cd jetson-containers
./run.sh $(./autotag stable-diffusion-webui)
</syntaxhighlight>

The following GIF shows the terminal output the first time the above is run: 

<br>
[[File:Start image generation tutorial.gif | center]]
<br>

The compatible pull image is <code>dustynv/stable-diffusion-webui:r36.2.0</code> and the container is run as follows:

<syntaxhighlight lang=bash>
docker run --runtime nvidia -it --rm --network host --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /home/nvidia/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb dustynv/stable-diffusion-webui:r36.2.0
</syntaxhighlight>

Notice in the above gif that the model is loaded after running the image generation container. The model used is [v1-5-pruned-emaonly.safetensors https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/v1-5-pruned-emaonly.safetensors]. So, the tutorial is ready to use.

== Create images ==

1. Open the browser and access <code>http://<IP_ADDRESS>:7860</code>. For example:

<syntaxhighlight lang=bash>
http://192.168.0.5:7860
</syntaxhighlight>

2. In the prompt box write the image description that would you like to create. For example:

''Futuristic city with sunset, high quality, 4K image.''

3. Click in the <code>Generate</code> button and check the result. The following gif summarizes the image creation process:

<br>
[[File:Futuristic city.gif | center]]
<br>

The image created is the following:

[[File:Futuristic city image.png|center|600px]]

4. The image generated can be saved into your system. Just first click on the save button and then on the file size save it. The next image shows how to do it:

[[File:Save image generated.jpg|center|600px]]

= Run Text Plus Vision tutorial =

The [http://www.jetson-ai-lab.com/tutorial_llava.html Text + Vision] NVIDIA tutorial, cover some methods that allow to run the Text Plus Vision tutorial, which are:

1. [http://www.jetson-ai-lab.com/tutorial_llava.html#1-chat-with-llava-using-text-generation-webui Chat with Llava using text-generation-webui]

2. [http://www.jetson-ai-lab.com/tutorial_llava.html#2-run-from-the-terminal-with-llavaservecli Run from the terminal with llava.serve.cli]

3. [http://www.jetson-ai-lab.com/tutorial_llava.html#3-quantized-gguf-models-with-llamacpp Quantized GGUF models with llama.cpp]

4. [http://www.jetson-ai-lab.com/tutorial_llava.html#4-optimized-multimodal-pipeline-with-local_llm Optimized Multimodal Pipeline with local_llm]

The one that is going to be explored in this wiki is the ''Chat with Llava using text-generation-webui'' tutorial.

== Chat with Llava using text-generation-webui ==
to run the text plus vision example, follow the next instructions: 

1. Update to the latest container:

<syntaxhighlight lang=bash>
docker pull $(./autotag text-generation-webui)
</syntaxhighlight>

2. Download the text plus vision model:

<syntaxhighlight lang=bash>
./run.sh --workdir=/opt/text-generation-webui $(./autotag text-generation-webui) \
  python3 download-model.py --output=/data/models/text-generation-webui \
    TheBloke/llava-v1.5-13B-GPTQ
</syntaxhighlight>

Notice that the above command is going to download a model to be used. The model is [https://huggingface.co/TheBloke/llava-v1.5-13B-GPTQ TheBloke/llava-v1.5-13B-GPTQ]

3. Open the browser, access <code>http://<IP_ADDRESS>:7860</code>, drop an image in the Chat tab and get the answer from the IA:

[[File:Text plus vision result.gif| | center]]

The image to test is a free-to-use image taken from https://www.pexels.com/photo/people-walking-in-market-439818/

= NanoSAM (Segment Anything Model) tutorial =

<syntaxhighlight lang=bash>
cd jetson-containers
./run.sh $(./autotag nanosam)
</syntaxhighlight>

The NVIDIA [https://www.jetson-ai-lab.com/vit/tutorial_nanosam.html NanoSAM] tutorial uses the [https://github.com/NVIDIA-AI-IOT/nanosam#example-1---segment-with-bounding-box Example 1 - Segment with bounding box] to show how to generate a segmentation output using the bounding-box specified in the example code. In this case, the example is going to segment the dog that is enclosed in the rectangle created by the <code>(100,100), (850,759)</code> points.

1. Move to the <code>nanosam</code> directory:
<syntaxhighlight lang=bash>
cd /opt/nanosam
</syntaxhighlight>

2. Run the example:
<syntaxhighlight lang=bash>
python3 examples/basic_usage.py \
    --image_encoder="data/resnet18_image_encoder.engine" \
    --mask_decoder="data/mobile_sam_mask_decoder.engine"
</syntaxhighlight>

Note that this example is going to use the ResNet18 model as the image encoder and the [https://huggingface.co/dhkim2810/MobileSAM MobileSAM] as the mask decoder.

3. The output is going to be saved in the <code>/opt/nanosam/data/</code> under the <code>basic_usage_out.jpg</code> image. To check in your host machine, follow the next commands:

:3.1. Copy the image into the <code>/data/</code> directory that is mounted from the Jetson host:
<syntaxhighlight lang=bash>
cp data/basic_usage_out.jpg /data/ 
</syntaxhighlight>

:3.2. Copy the image into your computer using <code>scp</code>:

<syntaxhighlight lang=bash>
scp data/basic_usage_out.jpg <host_user>@<host_IP>:<working_directory_path</code>
</syntaxhighlight>

The output generated should look like the following:

[[File:Basic usage out.jpg|center|600px]]

{{ContactUs}}

[[Category:Jetson]][[Category:NVIDIA Xavier]][[Category:Jetson V4L2 Drivers]][[Category:JetsonNano]][[Category:JetsonTX2]][[Category:NVIDIA Xavier]][[Category:NVIDIA Jetson Orin]][[Category:NVIDIA Jetson Orin Nano‎]][[Category:NVIDIA Jetson Orin NX‎]][[Category:NVIDIA Jetson Orin NX‎]]