RidgeRun AI Agent Microservice
RidgeRun's AI Agent Microservice llows a natural communication between the user and other microservices. This service uses the Hugging Face LLM Trelis/Llama-2-7b-chat-hf-function-calling-v3 to convert text commands into API calls, processes the LLM result and calls the corresponding API request.
- The service requires a prompt description defining the available functions and the behavior of the LLM and a
configuration file that defines the API calls and its mapping with the prompt function. See below for more details.
- The microservice receives text commands queries and uses the LLM model to convert the commands to API calls.
- As an output the service actually makes the call to the API obtained from the LLM. The LLM output is consistent to parse and easily match with a given API call.
- The agent provides a simple web page that allows sending text commands from a web browser.
The commands that can understand the LLM and the API calls depends of the configuration provided, check the demo for a use case example.
Prompt Format
In order to achieve the desired behavior from the model we need it to respond to input commands in a consistent and parseable way that allows us to easily match the model output with a given API call or command. For this, we will make use of prompting to let the LLM know the format we want it to use for the answers to our commands.
Prompt should follow the model prompt format, please check model documentation at https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v3
API Configuration
The API configuration is a JSON file with the following structure:
{
"function_name": {
"ip": "api_ip",
"port": "api_port",
"path": "api_request_path",
"method": "request_method",
"properties": {
"prop1": "prop1 value",
"prop2": "prop2 value"
},
"body": {
"arg1": "value",
"arg2": "value"
}
}
}
The JSON should have a function object per function described in the prompt, since it will be used to map the LLM reply function with the microservice API call. So the function_name should match one of the functions defined in the prompt.
The arguments port, path and method are required. Port is the port of the microservice to call, path is the route of the specific API request and method is the method for the request can be GET, POST, PUT.
The argument ip is optional, it defines the IP of the microservice to call. If not defined localhost 127.0.0.0 will be used.
properties object define the parameters of the API request. It is optional, add it only if the API request use parameters. The value of each property will be obtained from the LLM reply, so the string in the value should match the argument name defined in the corresponding prompt function.
body object define the API request content. It is optional, add it only if the API request need body description. The value of each argument in the body will be obtained from the LLM reply, so the string in the value should match the argument name defined in the corresponding prompt function.
Check the following example:
{
"search_object": {
"ip": "192.168.86.25",
"port": 30080,
"path": "genai/prompt",
"method": "GET",
"properties": {
"objects": "input",
"thresholds": 0.2
}
},
"move_camera": {
"port": 1234,
"path": "position",
"method": "PUT",
"body": {
"pan": "pan_angle",
"tilt": "tilt_angle"
}
}
}
Running the Service
Launch Services
You must launch the services defined in the API configuration file before sending any text command.
Using Docker
You can obtain or build a docker image for the ai agent microservice, check below the method of your preference. The image include a base tensorrt image and the dependencies to run ai-agent microservice application. The image was developed and testing for NVIDIA JetPack 6.0. Once you get your image with either method proceed to Launch the container
Pre-build Image
You can obtain a pre-build image of the detection service from Docker Hub:
docker pull ridgerun/ai-agent-service
Build Image
You can build the detection microservice image using the Dockerfile in the docker directory. First we need to prepare the context directory for this build, you need to create a directory and include this repository and the rrms-utils project. The Dockerfile will look for both packages in the context directory and copy them to the container.
mkdir ai-agent-context cd ai-agent-context git clone https://github.com/RidgeRun/ai-agent-service.git git clone https://github.com/RidgeRun/rrms-utils.git
After this, your context directory should look like this:
ai-agent-context/ ├── ai-agent └── rrms-utils
Then build the container image with the following command:
DOCKER_BUILDKIT=0 docker build --network=host --tag ridgerun/ai-agent-service --file detection-context/ai-agent/docker/Dockerfile ai-agent-context/
Change ai-agent-context/ to your context's path and the tag to the name you want to give to your image.
Launch the container
You can ensure the images have started successfully by running
docker image
You should get an entry showing the ridgerun/detection-service image
nvidia@ubuntu:~$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE ridgerun/ai-agent-service latest 3874b9429f9a 2 days ago 18.1GB
The container can be launched by running the following command:
docker run --runtime nvidia -it --network host --volume /home/nvidia/config:/ai-agent-config --name agent-service ridgerun/ai-agent-service:latest ai-agent --system_prompt ai-agent-config/prompt.txt --api_map ai-agent-config/api_mapping.json
Here we are creating a container called agent-service. Notice we are mounting the directory /home/nvidia/config into /ai-agent-config, this contains the prompt and api configuration files, you can change it to point to your configuration directory or any place where you have the required configs. Also we are defining the ai-agent microservice application as entry point with its corresponding parameters.
You can verify the container is running with:
docker ps
You should see an entry with the detection-service container:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dd40d212e360 ridgerun/ai-agent "ai-agent --system_p…" 25 hours ago Up 53 seconds agent-service
Using Standalone Application
The project is configured (via setup.py) and install the service application called ai-agent. Just need to run:
pip install .
Then you will have the service with the following options:
usage: ai-agent [-h] [--port PORT] --system_prompt SYSTEM_PROMPT --api_map API_MAP
options:
-h, --help show this help message and exit
--port PORT Port for server
--system_prompt SYSTEM_PROMPT
String with system prompt or path to a txt file with the prompt
--api_map API_MAP Path to a JSON file with API mapping configuration
usage: ai-agent [-h] [--port PORT] [--system_prompt SYSTEM_PROMPT] [--api_map API_MAP]
Notice that the system_prompt and api_map are required, so to run it use the following command:
ai-agent --prompt PROMPT --api_map API_MAP
This will start the service in address 127.0.0.0 and port 5010. If you want to use a different port, use the --port options.
AI Agent Web Page
The agent microservice provides a web page to make easier the communication with the agent. You can access it through ingress if configured as in RidgeRun's smart seek demo or directly to the board ip address and service port.
Video below shows an example of how to use it. You just need to write a request to the agent, the agent will process it and return success if the request is valid according to defined prompt and API mapping or failure if the request is invalid in the context.
{{Metropolis Microservices/Foot
|previous=https://developer.ridgerun.com/wiki/index.php/Metropolis_Microservices/RidgeRun Services/Detection
|next=https://developer.ridgerun.com/wiki/index.php/Metropolis_Microservices/RidgeRun Services/Analytics