Qualcomm Robotics RB5/Demos/Drone Demo

Introduction

This demo consists of a simulated drone system on a Qualcomm Robotics RB5 that performs live capture, image processing, sensor readings, metadata, encoding and network transmission. Some of the key components it explores include:

• PTZR for dynamic control of the imager
• Accelerometer readings embedded as metadata in the stream
• H265 encoding
• RTSP streaming
• A media server for media management

The following image summarizes the overall functionality.

Dependencies

In order to accomplish the desired behavior, we use some of our products with out-of-the-box functionalities that significantly reduce development time. For this demo, we will be using:

• GstPTZR: for digital control of the imager
• GstSEIMetadata: for sensor reading and metadata
• Gstd: for media server management
• RTSPSink: for media transmission.

Run the demo

Once you get the code, you should see the following files:

├── requirements.txt
├── drone_demo.py 
├── rrdronedemo
│   ├── __init__.py
│   ├── argumenthandler.py
│   ├── imu.py
│   ├── pipelineentity.py
│   └── tcp_server.py

The demo can be easily run with the following command:

python3 drone_demo.py

Application parameters

For a flexible configuration, we exposed the following arguments which can be used to tune the application. This includes:

Option	Type	Description
--rotate_level	float	Rotation (in degrees) about the depth axis with origin on the center of the capture region in the range [-180.0, 180.0]
--tilt_level	float	Translation about the y axis starting from the center of the input image in the range [-image height, image height]
--pan_level	float	Translation about the x axis starting from the center of the input image in the range [-image width, image width]
--zoom_level	float	Zoom level
--resolution	string	Camera input resolution {720p,1080p,4k}
--control_rate	int	Bitrate control method: (0) Disable (1) Constant (2) CBR-VFR (3) VBR-CFR (4) VBR-VFR (5) CQ
--bitrate	unsigned int	Target bitrate in bits per second (0 is no specific bitrate)
--idr_interval	unsigned int	IDR frame interval (0 means no specific IDR)
--rtsp-port	unsigned int	RTSP port
--mapping	string	RTSP mappings
--auth	string	Authentication and authorization. Format: user1:password1
--tcp-port	unsigned int	TCP server port for receiving commands

Control through TCP

This demo allows the user to control the media server through a TCP connection, which can be established using the following command:

nc <IP_ADDRESS> <TCP_PORT>

The commands supported include:

For PTZR control:

• zoom
• rotate
• tilt
• pan_level

For the media server:

• play
• pause
• stop

Sample usage:

nc 192.168.23.180 9999
zoom=5
rotate=1
tilt=3
pan_level=4
play
pause
stop

Capture metadata

Finally, if you want to inspect the received metadata, you can use the seiextract element as follows:

GST_DEBUG=*seiextract*:MEMDUMP gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:5000/stream1 ! rtph265depay ! video/x-h265 ! seiextract !  h265parse ! avdec_h265 ! videoconvert ! autovideosink