RidgeRun GStreamer Analytics Example:Troubleshooting
Follow Us On
NO System Metrics in Grafana Dashboard
If you installed everything and you don't see any system data in the dashboard, it means that there is not data flowing from the agent to the server. Things to check:
- Make sure the services are up and running in the agent side. Output of docker ps in the agent should show the following services running:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 678da6d18650 rranalytics-system-metrics:latest "rr-system-metrics -…" 2 hours ago Up 2 hours rranalytics-system-metrics-service dec26763adb2 gst-rranalytics-alloy "/bin/alloy run /etc…" 2 hours ago Up 2 hours alloy d1f4cebd0e2f rranalytics-prometheus-exporter:latest "rr-prometheus-expor…" 2 hours ago Up 2 hours 0.0.0.0:6000->6000/tcp, :::6000->6000/tcp rranalytics-prometheus-exporter-service 63fb4895b569 rranalytics-logger-agent:latest "rr-logger-agent --n…" 2 hours ago Up 2 hours 0.0.0.0:6001->6001/tcp, :::6001->6001/tcp rranalytics-logger-agent-service
- Make sure the services are up and running in the server side. Output of docker ps in the server should show the following services running:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8e5d4ef78aab grafana/grafana-enterprise:12.1.0 "/run.sh" 5 days ago Up 5 days 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp rranalytics-grafana 917460955b78 prom/prometheus:v3.5.0 "/bin/prometheus --c…" 5 days ago Up 5 days 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp rranalytics-prometheus ac57012d5436 hashicorp/consul:1.21.4 "docker-entrypoint.s…" 5 days ago Up 5 days 8300-8302/tcp, 8301-8302/udp, 8600/tcp, 8600/udp, 0.0.0.0:8500->8500/tcp, :::8500->8500/tcp rranalytics-consul 42952fa36e0a gst-rranalytics-loki "/usr/bin/loki -conf…" 5 days ago Up 5 days 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp rranalytics-loki
The system needs the server to be running before the agents start. If the agent services start before the server services, no data will be shown in the dashboard. You can verify this is the case by checking consul:
- In the server side, in your browser, go to <IP_ADDRESS>:8500 to see the Consul dashboard. You should see something like this:
- If you don't see prometheus-exporter you need to restart the agent services. Go to the agent and run:
nox -s agent -- down nox -s agent -- up
- Check Consul again in the server <IP_ADDRESS>:8500 address, you should see prometheus-exporter now.
- Reload Grafana Dashboard again, you should see system metrics now.
It is very important to start server services before agent services.