Qualcomm Robotics RB5/Demos/Drone Demo
Introduction
This demo consists of a simulated drone system on a Qualcomm Robotics RB5 that performs live capture, image processing, sensor readings, metadata, encoding and network transmission. Some of the key components it explores include:
- PTZR for dynamic control of the imager
- Accelerometer readings embedded as metadata in the stream
- H265 encoding
- RTSP streaming
- A media server for media management
The following image summarizes the overall functionality.
Dependencies
In order to accomplish the desired behavior, we use some of our products with out-of-the-box functionalities that significantly reduce development time. For this demo, we will be using:
- GstPTZR: for digital control of the imager
- GstSEIMetadata: for sensor reading and metadata
- Gstd: for media server management
- RTSPSink: for media transmission.
Run the demo
Once you get the code, you should see the following files:
├── requirements.txt ├── drone_demo.py ├── rrdronedemo │ ├── __init__.py │ ├── argumenthandler.py │ ├── imu.py │ ├── pipelineentity.py │ └── tcp_server.py
The demo can be easily run with the following command:
python3 drone_demo.py
Application parameters
For a flexible configuration, we exposed the following arguments which can be used to tune the application. This includes:
Option | Type | Description |
---|---|---|
--rotate_level | float | Rotation (in degrees) about the depth axis with origin on the center of the capture region in the range [-180.0, 180.0] |
--tilt_level | float | Translation about the y axis starting from the center of the input image in the range [-image height, image height] |
--pan_level | float | Translation about the x axis starting from the center of the input image in the range [-image width, image width] |
--zoom_level | float | Zoom level |
--resolution | string | Camera input resolution {720p,1080p,4k} |
--control_rate | int | Bitrate control method: (0) Disable (1) Constant (2) CBR-VFR (3) VBR-CFR (4) VBR-VFR (5) CQ |
--bitrate | unsigned int | Target bitrate in bits per second (0 is no specific bitrate) |
--idr_interval | unsigned int | IDR frame interval (0 means no specific IDR) |
--rtsp-port | unsigned int | RTSP port |
--mapping | string | RTSP mappings |
--auth | string | Authentication and authorization. Format: user1:password1 |
--tcp-port | unsigned int | TCP server port for receiving commands |
Control through TCP
This demo allows the user to control the media server through a TCP connection, which can be established using the following command:
nc <IP_ADDRESS> <TCP_PORT>
The commands supported include:
For PTZR control:
- zoom
- rotate
- tilt
- pan_level
For the media server:
- play
- pause
- stop
Sample usage:
nc 192.168.23.180 9999 zoom=5 rotate=1 tilt=3 pan_level=4 play pause stop
Capture metadata
Finally, if you want to inspect the received metadata, you can use the seiextract element as follows:
GST_DEBUG=*seiextract*:MEMDUMP gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:5000/stream1 ! rtph265depay ! video/x-h265 ! seiextract ! h265parse ! avdec_h265 ! videoconvert ! autovideosink