Introduction

This demo consists of a simulated drone system on a Qualcomm Robotics RB5 that performs live capture, image processing, sensor readings, metadata, encoding and network transmission. Some of the key components it explores include:

PTZR for dynamic control of the imager
Accelerometer readings embedded as metadata in the stream
H265 encoding
RTSP streaming
A media server for media management

The following image summarizes the overall functionality.

Dependencies

In order to accomplish the desired behavior, we use some of our products with out-of-the-box functionalities that significantly reduce development time. For this demo, we will be using:

GstPTZR: for digital control of the imager
GstSEIMetadata: for sensor reading and metadata
Gstd: for media server management
RTSPSink: for media transmission.

Run the demo

The demo can be easily run with the following command:

python3 drone_demo.py

Application parameters

For a flexible configuration, we exposed the following arguments which can be used to tune the application. This includes:

Resolution:
- 1280x720
- 1920x1080
- 4k

Encoding:
- Control rate
- Bitrate
- IDR interval

PTZR:
- Rotate level
- Tilt level
- Pan level
- Zoom level

RTSP:
- RTSP port
- Mapping
- Authentication

TCP server:
- TCP port

Further information can be obtained from the application.

usage: drone_demo.py [-h] [--rotate_level FLOAT] [--tilt_level FLOAT] [--pan_level FLOAT] [--zoom_level FLOAT]
                     [--resolution {720p,1080p,4k}] [--control_rate CONTROL_RATE] [--bitrate BITRATE] [--idr_interval IDR_INTERVAL]
                     [--rtsp-port RTSP_PORT] [--mapping MAPPING] [--auth AUTH] [--tcp-port TCP_PORT]

RB5 Drone Demo Help

optional arguments:
  -h, --help            show this help message and exit
  --rotate_level FLOAT  Rotation (in degrees) about the depth axis with origin on the center of the capture region in the range
                        [-180.0, 180.0]
  --tilt_level FLOAT    Translation about the y axis starting from the center of the input image in the range [-image height, image
                        height]
  --pan_level FLOAT     Translation about the x axis starting from the center of the input image in the range [-image width, image
                        width]
  --zoom_level FLOAT    Zoom level
  --resolution {720p,1080p,4k}
                        Camera input resolution
  --control_rate CONTROL_RATE
                        Bitrate control method: (0) Disable (1) Constant (2) CBR-VFR (3) VBR-CFR (4) VBR-VFR (5) CQ
  --bitrate BITRATE     Target bitrate in bits per second (0 is no specific bitrate)
  --idr_interval IDR_INTERVAL
                        IDR frame interval (0 means no specific IDR)
  --rtsp-port RTSP_PORT
                        RTSP port
  --mapping MAPPING     RTSP mappings
  --auth AUTH           Authentication and authorization. Format: user1:password1
  --tcp-port TCP_PORT   TCP server port for receiving commands

Control through TCP

This demo allows the user to control the media server through a TCP connection, which can be established using the following command:

nc <IP_ADDRESS> <TCP_PORT>

The commands supported include:

For PTZR control:

zoom
rotate
tilt
pan_level

For the media server:

play
pause
stop

Sample usage:

nc 192.168.23.180 9999
zoom=5
rotate=1
tilt=3
pan_level=4
play
pause
stop