Qualcomm Robotics RB5 - AI hardware acceleration - Neural Processing SDK - Running Detection

Qualcomm Robotics RB5/RB6
Introduction
SoM Overview Why the RB5/RB6? Carrier Boards Qualcomm Robotics RB5 developer kit Qualcomm Robotics RB6 platform
Software Development Support
Boot Devices Thundercomm SDK Downloading SDK Installing SDK Generating Image and flashing Linaro Getting Started Flashing Bootloader Flashing Boot Image and Rootfs Special / Debug Modes Setting up board in EDL Mode Setting up board in Fastboot Mode
Development in the Board
Getting into the Board Using ADB Using Serial Console Using SSH Device Setup WiFi setup HDMI setup Update Ubuntu Packages
Build from source
Base Image Customize Image
Capture Subsystem
Hardware Capture Components Software Capture Components Building your own camera sensor driver
Image Processing and Acceleration Software
GPU Profiling Fast CV Installing Hexagon SDK Installing Fast CV Building your own Fast CV application
GStreamer Pipelines
Capture and Display Video Encoding Video Decoding
AI Acceleration
Neural Processing SDK Downloading Requirements Setup SDK Environment Install qtimlesnpe Running Inception v3 Running MobileNetV2 + SSD TensorFlow Example pipeline TensorFlow to DLC Conversion
Sensors
IMU
Demos
Drone Demo Smart Camera
Contact Us

In this section, we are going to see how to work with a detection model and the SNPE plugin for GStreamer. In this case, we are going to use a pre-trained MobileNetv2+SSD model for the detection, inspecting and locating of all files required for this task. To continue with these steps, the previous sections should be completed. If not, please check our sections Downloading Requirements, Setup SDK Environment and Install qtimlesnpe.

Get All the Files

Please, make sure of downloading and creating the following files:

MobileNetv2 + SSD Model

You can download it from GitHub or this Mirror Site.

Once downloaded, move the file to /data/misc/camera with the name mobilenet-SSD.dlc

Labels

You can download it from GitHub or this Mirror Site.

Once downloaded, move the file to /data/misc/camera with the name coco_labels.txt

Configuration File

Create a file in /data/misc/camera/mle_snpe.config with the following contents:

org.codeaurora.mle.snpe
input_format = 3
BlueMean = 128.0
GreenMean = 128.0
RedMean = 128.0
BlueSigma = 128.0
GreenSigma = 128.0
RedSigma = 128.0
UseNorm = true
preprocess_type = 1
confidence_threshold = 0.6
output_layers = < "add_6", "Postprocessor/BatchMultiClassNonMaxSuppression" >
runtime = 1
model = "/data/misc/camera/mobilenet-SSD.dlc"
labels = "/data/misc/camera/coco_labels.txt"

From here, you can change the runtime by:

• 0: CPU
• 1: DSP
• 2: GPU

Run the Example on GStreamer

Once the files are in place and the prerequisites are covered, you should be able to execute the following pipeline:

gst-launch-1.0 qtiqmmfsrc ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1, camera=0 ! qtimlesnpe config=/data/misc/camera/mle_snpe.config postprocessing=detection ! queue ! qtioverlay ! queue ! qtic2venc ! h264parse ! qtmux ! filesink location=detection.mp4 -evvv

The detection metadata is passed on top of the buffer as buffer metadata.

Detection Demo

Demo

We have worked on a demo to facilitate getting familiar with the RB5/RB6 and AI. Please, have a look at this wiki: Demos/Smart Camera

Known Issues

Sometimes, depending on the environment, it claims that version 1.43 or lower should be used. You can change the runtime to get it working and determine if the libraries are too new for the model.

Moreover, this pipeline was tested on a Qualcomm RB5 running Linux Ubuntu 20.04.