Qualcomm Robotics RB5 - Drone Demo
Introduction
This demo consists of a simulated drone system on a Qualcomm Robotics RB5 that performs live capture, image processing, sensor readings, metadata, encoding and network transmission. Some of the key components it explores include:
- PTZR for dynamic control of the imager
- Accelerometer readings embedded as metadata in the stream
- H265 encoding
- RTSP streaming
- A media server for media management
The following image summaries the overall functionality.
Interested in the Full Demo?
Send us a message through the Contact Us form with the following details:
- Platform: i.e. Qualcomm RB5 Devkit, Lantronix, Thundercomm
- OS: i.e. LU Ubuntu 20.04
- Brief Description of your use case
Important: Our demo is available only for the RB5 with Ubuntu 20.04
Dependencies
In order to accomplish the desired behaviour, we use some of our products with out-of-the-box functionalities that significantly reduce development time. For this demo, we will be using:
- GstPTZR: for digital control of the imager
- GstSEIMetadata: for sensor reading and metadata
- Gstd: for media server management
- RTSPSink: for media transmission.
All these products are contained in the demo in their evaluation versions. If a professional version is required, please, Contact Us.
Installation
Please, install all the dependencies using:
bash pre-install.sh
After that, use the meson build system and install the demos and dependencies:
meson builddir --prefix=/usr ninja -C builddir ninja -C builddir install
Run the demo
Once you get the code, you should see the following files:
├── requirements.txt ├── drone_demo.py ├── rrdronedemo │ ├── __init__.py │ ├── argumenthandler.py │ ├── imu.py │ ├── pipelineentity.py │ └── tcp_server.py
Environment
Please, enable the HDMI with the following instructions: Qualcomm_Robotics_RB5/Development_in_the_Board/Device_Setup/HDMI_Setup and set the following environment variables (if you are remotely connected to the RB5):
export WAYLAND_DISPLAY=wayland-0 export XDG_RUNTIME_DIR=/usr/bin/weston_socket
Running
The demo can be easily run with the following command:
python3 drone_demo.py
Application parameters
For a flexible configuration, we exposed the following arguments which can be used to tune the application. This includes:
| Option | Type | Description |
|---|---|---|
| --rotate_level | float | Rotation (in degrees) about the depth axis with origin on the center of the capture region in the range [-180.0, 180.0] |
| --tilt_level | float | Translation about the y axis starting from the center of the input image in the range [-image height, image height] |
| --pan_level | float | Translation about the x axis starting from the center of the input image in the range [-image width, image width] |
| --zoom_level | float | Zoom level |
| --resolution | string | Camera input resolution {720p,1080p,4k} |
| --control_rate | int | Bitrate control method: (0) Disable (1) Constant (2) CBR-VFR (3) VBR-CFR (4) VBR-VFR (5) CQ |
| --bitrate | unsigned int | Target bitrate in bits per second (0 is no specific bitrate) |
| --idr_interval | unsigned int | IDR frame interval (0 means no specific IDR) |
| --rtsp-port | unsigned int | RTSP port |
| --mapping | string | RTSP mappings |
| --auth | string | Authentication and authorization. Format: user1:password1 |
| --tcp-port | unsigned int | TCP server port for receiving commands |
Control through TCP
This demo allows the user to control the media server through a TCP connection, which can be established using the following command:
nc <IP_ADDRESS> <TCP_PORT>
Please, execute this command from your computer or another machine. The IP_ADDRESS is the code of the RB5 and the default TCP_PORT is 9999 by default. It can by changed through command arguments when executing the demo.
The supported commands include:
For PTZR control:
- zoom
- rotate
- tilt
- pan
For the media server:
- play
- pause
- stop
Sample usage:
nc 192.168.23.180 9999 zoom=5 rotate=1 tilt=3 pan=4 play pause stop
Capture metadata
Finally, if you want to inspect the received metadata, you can use the seiextract element as follows:
GST_DEBUG=*seiextract*:MEMDUMP gst-launch-1.0 rtspsrc location=rtsp://$IP_ADDRESS:49152/stream1 ! rtph265depay ! video/x-h265 ! seiextract ! h265parse ! avdec_h265 ! videoconvert ! autovideosink
This code is executed on another machine with a display attached to it and connected to the same network as the RB5. If you don't have the seiextract, you can execute the following:
gst-launch-1.0 rtspsrc location=rtsp://$IP_ADDRESS:49152/stream1 ! rtph265depay ! video/x-h265 ! h265parse ! avdec_h265 ! videoconvert ! autovideosink
Please, note that the RTSP port is by default 49152 and the IP_ADDRESS must contain the IP address of the RB5
Looks good, right?
Are you interested in getting through this demo and more solutions like this? Feel free to Contact Us.
Known Issues
1. The demo shows a segfault when running it
This is due to a misconfiguration on the display. Make sure of connecting a display, enabling the HDMI and setting the environment variables.
2. There is an error associated to GLX context
This is due to a misconfiguration in the display. Please, send stop and play again.