Metropolis Microservices/RidgeRun Services/AI Agent: Difference between revisions

Metropolis Microservices/RidgeRun Services/AI Agent (view source)

Revision as of 12:25, 1 July 2024

8,145 bytes added , 1 July

no edit summary

Mmontero

866

edits

@@ Line 4: / Line 4: @@
+RidgeRun's AI Agent Microservice llows a natural communication between the user and other microservices. This service
+uses the Hugging Face LLM  Trelis/Llama-2-7b-chat-hf-function-calling-v3 to convert text commands into API
+calls, processes the LLM result and calls the corresponding API request.
-<noinclude>
+[[File:aiagent_microservice.png|600px|center|thumb| Ridgerun's AI Agent Microservice General Idea]]
-{{Metropolis Microservices/Foot||}}
-</noinclude>
+* The service requires a prompt description defining the available functions and the behavior of the LLM and a
+configuration file that defines the API calls and its mapping with the prompt function. See below for more details.
+* The microservice receives text commands queries and uses the LLM model to convert the commands to API calls.
+* As an output the service actually makes the call to the API obtained from the LLM. The LLM output is consistent to parse and easily match with a given API call.
+* The agent provides a simple web page that allows sending text commands from a web browser.
+The commands that can understand the LLM and the API calls depends of the configuration provided, check the demo for a use case example.
+=== Prompt Format ===
+In order to achieve the desired behavior from the model we need it to respond to input commands in a consistent and parseable way that allows us to easily match the model output with a given API call or command.  For this, we will make use of prompting to let the LLM know the format we want it to use for the answers to our commands.
+Prompt should follow the model prompt format, please check model documentation at https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v3
+=== API Configuration ===
+The API configuration is a JSON file with the following structure:
+<syntaxhighlight lang="JSON">
+{
+    "function_name": {
+        "ip": "api_ip",
+        "port": "api_port",
+        "path": "api_request_path",
+        "method": "request_method",
+        "properties": {
+            "prop1": "prop1 value",
+            "prop2": "prop2 value"
+        },
+        "body": {
+            "arg1": "value",
+            "arg2": "value"
+        }
+    }
+}
+</syntaxhighlight>
+The JSON should have a function object per function described in the prompt, since it will be used to map the LLM reply function with the microservice API call. So the function_name should match one of the functions defined in the prompt.
+The arguments port, path and method are required. Port is the port of the microservice to call, path is the route of the specific API request and method is the method for the request can be GET, POST, PUT.
+The argument ip is optional, it defines the IP of the microservice to call. If not defined localhost 127.0.0.0 will be used.
+properties object define the parameters of the API request. It is optional, add it only if the API request use parameters. The value of each property will be obtained from the LLM reply, so the string in the value should match the argument name defined in the corresponding prompt function.
+body object define the API request content. It is optional, add it only if the API request need body description. The value of each argument in the body will be obtained from the LLM reply, so the string in the value should match the argument name defined in the corresponding prompt function.
+Check the following example:
+<syntaxhighlight lang="JSON">
+{
+  "search_object": {
+    "ip": "192.168.86.25",
+    "port": 30080,
+    "path": "genai/prompt",
+    "method": "GET",
+    "properties": {
+      "objects": "input",
+      "thresholds": 0.2
+    }
+  },
+  "move_camera": {
+    "port": 1234,
+    "path": "position",
+    "method": "PUT",
+    "body": {
+      "pan": "pan_angle",
+      "tilt": "tilt_angle"
+    }
+  }
+}
+</syntaxhighlight>
+== Running the Service ==
+=== Launch Services ===
+You must launch the services defined in the API configuration file before sending any text command.
+=== Using Docker ===
+You can obtain or build a docker image for the ai agent microservice, check below the method of your preference. The image include a base tensorrt image and the dependencies to run ai-agent microservice application. The image was developed and testing for NVIDIA JetPack 6.0. Once you get your image with either method proceed to [[Launch the container]]
+==== Pre-build Image ====
+You can obtain a pre-build image of the detection service from Docker Hub:
+<syntaxhighlight lang="bash">
+docker pull ridgerun/ai-agent-service
+</syntaxhighlight>
+==== Build Image ====
+You can build the detection microservice image using the Dockerfile in the docker directory.  First we need to prepare the context directory for this build, you need to create a directory and include this repository and the rrms-utils project. The Dockerfile will look for both packages in the context directory and copy them to the container.
+<syntaxhighlight lang="bash">
+ai-agent-context/
+├── ai-agent
+└── rrms-utils
+</syntaxhighlight>
+Then build the container image with the following command:
+<syntaxhighlight lang="bash">
+DOCKER_BUILDKIT=0 docker build --network=host --tag ridgerun/ai-agent-service --file detection-context/ai-agent/docker/Dockerfile ai-agent-context/
+</syntaxhighlight>
+Change ai-agent-context/ to your context's path and the tag to the name you want to give to your image.
+====  Launch the container ====
+You can ensure the images have started successfully by running
+<syntaxhighlight lang="bash">
+docker image
+</syntaxhighlight>
+You should get an entry showing the ridgerun/detection-service image
+<syntaxhighlight lang="bash">
+nvidia@ubuntu:~$ docker images
+REPOSITORY                                  TAG                    IMAGE ID       CREATED        SIZE
+ridgerun/ai-agent-service                   latest                 3874b9429f9a   2 days ago     18.1GB
+</syntaxhighlight>
+The container can be launched by running the following command:
+<syntaxhighlight lang="bash">
+docker run --runtime nvidia -it  --network host --volume /home/nvidia/config:/ai-agent-config --name agent-service  ridgerun/ai-agent-service:latest ai-agent --system_prompt ai-agent-config/prompt.txt --api_map ai-agent-config/api_mapping.json
+</syntaxhighlight>
+Here we are creating a container called agent-service. Notice we are mounting the directory /home/nvidia/config into /ai-agent-config, this contains the prompt and api configuration files, you can change it to point to your configuration directory or any place where you have the required configs. Also we are defining the ai-agent microservice application as entry point with its corresponding parameters.
+You can verify the container is running with:
+<syntaxhighlight lang="bash">
+docker ps
+</syntaxhighlight>
+You should see an entry with the detection-service container:
+<syntaxhighlight lang="bash">
+CONTAINER ID   IMAGE                                            COMMAND                  CREATED        STATUS          PORTS     NAMES
+dd40d212e360   ridgerun/ai-agent                                "ai-agent --system_p…"   25 hours ago   Up 53 seconds             agent-service
+</syntaxhighlight>
+=== Using Standalone Application ===
+The project is configured (via setup.py) and install the service application called ai-agent. Just need to run:
+<syntaxhighlight lang="bash">
+pip install .
+</syntaxhighlight>
+Then you will have the service with the following options:
+<syntaxhighlight lang="bash">
+usage: ai-agent [-h] [--port PORT] --system_prompt SYSTEM_PROMPT --api_map API_MAP
+options:
+  -h, --help            show this help message and exit
+  --port PORT           Port for server
+  --system_prompt SYSTEM_PROMPT
+                        String with system prompt or path to a txt file with the prompt
+  --api_map API_MAP     Path to a JSON file with API mapping configuration
+usage: ai-agent [-h] [--port PORT] [--system_prompt SYSTEM_PROMPT] [--api_map API_MAP]
+</syntaxhighlight>
+Notice that the system_prompt and api_map are required, so to run it use the following command:
+<syntaxhighlight lang="bash">
+ai-agent --prompt PROMPT --api_map API_MAP
+</syntaxhighlight>
+This will start the service in address 127.0.0.0 and port 5010. If you want to use a different port, use the __--port__ options.