MediaTek Genio 700 AI Development Guide: NeuroPilot, DLA Conversion, Yocto Build and NPU Inference

Introduction

SBC-MB710 [1]

MediaTek’s hardware platforms, such as mobile System-on-Chip (SoC) and ultra-low power embedded devices, span different levels of compute density. MediaTek is deeply invested in creating an AI ecosystem with efficient yet powerful AI-processors in its devices, ranging from smartphones to smart homes, wearables, Internet-of-Things (IoT), and connected cars.

The MediaTek Genio 700 (MT8390) is an edge AI SoC platform that integrates a dedicated Neural Processing Unit (NPU) to accelerate deep learning workloads directly on-device. It is important to note that models must be converted into hardware-specific format known as Deep Learning Archive (DLA) to execute neural networks on this hardware.

DLA files are optimized binaries that can be executed efficiently by the MediaTek NPU, enabling low-latency and power-efficient inference on edge devices.

The MediaTek Genio 700 development kit, also known as SBC-MB710, is a single-board computer built by Amobile Solutions around the MediaTek Genio 700 SoC. It provides power delivery, connectivity and a wide range of external interfaces required for development and debugging.

For this guide, we use the SBC-MB710 development kit. If you are using a different board with the MediaTek Genio 700 SoC, some flash steps may vary depending on the hardware configuration.

This guide walks through the on the MediaTek Genio 700 complete workflow, according to next sections:

• How to set up build environment: Create a yocto image and flash tool installation.
• How to flash image on board: Steps to flash an image in the board.
• UART Debug: Explain how to use UART in the board.
• NeuroPilot: Explain about the required tools to convert a model to DLA an its steps.
• How to execute DLA inferences: Steps to run an inference with a DLA model.

How to set up build environment

System requirements

Before building Yocto images, flashing the device, or using the MediaTek Genio development tools, ensure that your host system meets the following requirements.

• Ubuntu 22.04 (LTS)
• Git 1.8 or later
• Python 3.8 or later
• pip 20.3 or later
• Fastboot 28.0.2 or later
• Administrator privileges
• Internet connection
• At least 200 GiB of free disk space

#Use this command to verify your available space
df -h /

• At least 16 GiB of system memory

#Use this command to verify your system memory size
free -h

Yocto Image set up

Install packages

Use this command to install the necessary dependencies:

sudo apt-get install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libelf-dev libsdl1.2-dev lz4 pylint xterm python3-subunit mesa-common-dev libstdc++-12-dev

Install repo tool

Install the latest repo tool

mkdir -p ~/bin
PATH="${HOME}/bin:${PATH}"
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+rx ~/bin/repo

Get Yocto Sources

Before building a Yocto image, you must obtain the Yocto source package corresponding to your platform.

As you can see in Amobile Build from Source Code, you need to contact Amobile Solutions to get access to the sources for your SBC-MB710 board or any amobile board which provides specific sources for their boards.

For other MediaTek-based platforms please contact with MediaTek according to Mediatek Build From Source Code to obtain access to the sources you need.

Create Yocto Image

Once the Yocto sources are available, the build environment must be initialized before generating an image.

Use the following steps:

• Configure the build directory
• Load the Yocto environment
• Compile the Rity demo image.

First, navigate to the Yocto source directory and initialize the build environment:

#Enter to yocto directory
cd amobile_yocto_kirkstone_v240

#Set environment variables
export PROJ_ROOT=$(pwd)
export BUILD_DIR=$(pwd)/sd700_build/
TEMPLATECONF=$PROJ_ROOT/src/meta-amobile-sd700/conf

#Init build
source src/poky/oe-init-build-env "sd700_build"

bitbake rity-demo-image

(Optional) Before executing bitbake, it is recommended to configure an URL mirror to reduce fetch failures in bibake. So add the next lines in amobile_yocto_kirkstone_v240/sd700_build/conf/local.conf :

INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "https://downloads.yoctoproject.org/mirror/sources/"
BB_FETCH_PREMIRRORONLY = "0"

Then execute bitbake:

#Bitbake Image
bitbake rity-demo-image

You can verify the generated image files using:

ls amobile_yocto_kirkstone_v240/sd700_build/tmp/deploy/images/sd700

Example output:

bl2.img
fip.bin
fitImage
rity-demo-image-sd700.aiotflash.tar
rity-demo-image-sd700.wic.img
sd700.dtb
u-boot.bin
...

The exact files may vary depending on the SDK and build version. The generated images and flashing artifacts are located in this directory and will be used in the following sections to flash the board using Genio Tools.

Genio Tools set up

What is Genio Tools?

Genio Tools is a collection of command-line utilities provided by MediaTek for configuring, controlling, and flashing Genio development boards from a host PC.

These tools are mainly used during:

• Board bring-up
• Firmware and image flashing
• System deployment
• Device configuration
• Development and debugging workflows

Genio Tools are designed to work together with MediaTek Genio Yocto SDKs and evaluation kits (EVKs). The toolkit relies on standard Android communication utilities such as adb and fastboot to communicate with target devices.

The most important utilities included in Genio Tools are:

• genio-flash: flash firmware and system images into Genio devices
• genio-board: control board reset and power signals
• genio-config: configure USB access and host communication settings

Recent versions of Genio Tools also support advanced workflows such as daemon mode and multi-device flashing for manufacturing and validation environments.

Refer to the official Genio Tools Documentation for more details.

Device Tree Overlays (DTBO)

A DTBO (Device Tree Overlay) is an additional hardware configuration file used to dynamically enable or configure specific peripherals and drivers in the Linux kernel without replacing the base Device Tree.

MediaTek Genio platforms use DTBO files to enable optional hardware components such as:

• HDMI or DisplayPort outputs
• Camera sensors
• Video codec drivers
• GPU acceleration
• Audio subsystems
• Wi-Fi or Bluetooth modules

DTBO files are loaded during boot through U-Boot environment variables and can be selected during the flashing process using genio-flash.

To list the DTBO files supported by the current image, use:

genio-flash --list-dtbo

Example output:

List of available DTBO:
    - display-dp.dtbo
    - gpu-mali.dtbo
    - video.dtbo

Some DTBO files are automatically loaded by default, while others must be explicitly enabled depending on the hardware configuration and desired features.

As example, to flash an image with a specific DTBO enabled, you can use:

genio-flash --load-dtbo video.dtbo

You can add many .dtbo as your preference.

Refer to: Genio Tools Manual, Device Tree Overlay and Video CODEC Device Tree Overlay

Install packages

Install Git and other necessary packages:

sudo apt update
sudo apt-get install git python3 python3-pip android-tools-adb android-tools-fastboot

USB device rules

Create a udev rule for USB access without root privileges:

echo -n 'SUBSYSTEM=="usb", ATTR{idVendor}=="0e8d", ATTR{idProduct}=="201c", MODE="0660", TAG+="uaccess"
SUBSYSTEM=="usb", ATTR{idVendor}=="0e8d", ATTR{idProduct}=="0003", MODE="0660", TAG+="uaccess"
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", MODE="0660", TAG+="uaccess"
SUBSYSTEM=="gpio", MODE="0660", TAG+="uaccess"
' | sudo tee /etc/udev/rules.d/72-aiot.rules

sudo udevadm control --reload-rules
sudo udevadm trigger

For adb access, add a rule and include your user in the plugdev group:

echo 'SUBSYSTEM=="usb", ATTR{idVendor}=="0e8d", ATTR{idProduct}=="201c", MODE="0660", GROUP="plugdev"' | sudo tee -a /etc/udev/rules.d/96-rity.rules

sudo udevadm control --reload-rules
sudo udevadm trigger
sudo usermod -a -G plugdev $USER

UART setup

Add your user to the group dialout

sudo usermod -a -G dialout $USER

Ensure your user is in the dialout group to connect to the device:

groups

Expected output:

user dialout cdrom floppy sudo audio dip video plugdev netdev bluetooth

If "dialout" doesn't appear in the output, log out and log back in to apply changes or you can restart your PC to ensure the change takes place.

Install a Serial Terminal Emulator

A serial terminal emulator is required to read serial data. For example, you can use minicom, which can be installed with:

sudo apt install minicom

Or you can use any terminal emulator as your preference.

Install genio-tools

pip3 install -U genio-tools

To check that genio-tools is correctly configured:

genio-config

Expected output:

fastboot: OK
udev rules: OK (md5: a3b2767b42ee01d7c62bf394400528ae)
Serial device write access: OK

Install fastboot

If fastboot is not shown as OK, install using:

sudo apt update
sudo apt-get install android-tools-adb android-tools-fastboot

Or download from Android Platform Tools, then follow the next steps:

1. Download the Linux version
2. Extract the .tar.gz file
3. Add the platform-tools directory to your PATH

cd /EXTRACTED_FILES_PATH/platform-tools-latest-linux/platform-tools
echo "export PATH=\$PATH:$(pwd)" >> ~/.bashrc
source ~/.bashrc

How to flash image on board

Navigate to the Image Directory:

If you have built your image from a yocto source, locate it in:

cd ../BUILD_DIR/tmp/deploy/images/YOUR_IMAGE

If you download a prebuilt image you may have to extract the file and then:

cd ../EXTRACTED_DIR/KIRKSTONE_IMAGE/PREBUILT_IMAGE

Note: The Genio 700 - SBC-MB710 only supports Yocto Kirkstone images.

Execute genio-flash:

#Default flashing
genio-flash

# Flashing with specific DTBO overlays (e.g. RAM size, peripherals).
genio-flash --load-dtbo gpu-mali.dtbo --load-dtbo apusys.dtbo --load-dtbo video.dtbo --load-dtbo display-hdmi.dtbo --load-dtbo camera-hi846.dtbo --load-dtbo ram-8gb.dtbo

You will see an output similar to the following:

Genio Tools: v1.7.0
Yocto Image:
	name:     Rity Demo Image (rity-demo-image)
	distro:   Rity Demo Layer 24.0-release (rity-demo)
	codename: kirkstone
	machine:  sd700
	overlays: ['gpu-mali.dtbo', 'video.dtbo', 'apusys.dtbo', 'display-hdmi.dtbo', 'camera-hi846.dtbo', 'ram-8gb.dtbo']

WARNING:aiot:No 'ftdi-cbus' device found
WARNING:aiot:Unable to find and reset the board. Possible causes are:
1. This is not a Genio 350/700 EVK, nor a Pumpkin board.
2. The board port UART0 is not connected.
3. The UART0 port is being opened by another tool, such as TeraTerm on Windows.
You can now manually reset the board into DOWNLOAD mode.

INFO:aiot:Continue flashing...

Then you should keep shorting the DOWNLOAD pin with GND as shown in Flash Image to Board. Another option to shorting DOWNLOAD pin with GND is to use an Alligator clamp to USB-C shell with the pin #37 of expansion connector, as pin shown in the image below.

Expansion Connector Download Pin

Then connect the USB-C power input port of the Genio 700 to your PC. The port is highlighted in red as shown in the image below.

SBC-MB710 Power Input [2]

You must use a USB-C to USB-C cable to connect the PC to the SBC-MB710. A USB-A to USB-C cable will not work. If your PC does not have a USB-C port, you can use a USB-C adapter.

After connecting the board, genio-flash will display an output like to the shown in Mediatek Genio 700 - Flash Log

When the process completes, your image will be successfully flashed.

UART Debug

To debug over UART, connect the micro-USB port of the SBC-MB710 to your PC.

The serial output includes different stages, each with its own baud rate, as shown below:

Console	Baud Rate	Purpose
Download Agent	921600	Download progress log from download agent (lk.bin) when using genio-flash
Boot ROM	115200	SoC boot ROM logs and messages
Bootloader Console	921600	U-Boot console for debugging and boot configurations
Linux Console	921600	kernel messages and Linux console

You can read the serial output using the appropriate serial device:

#select the appropriate device
DEVICE=/dev/ttyUSB0

Then read the serial console:

# To read and use the device across
sudo minicom -D $DEVICE -b 921600

Example output of linux console after board boot:

sd700 login: root
root@sd700:~#

The same command can also be used to monitor board messages during the flashing process. For a full flash log, refer to Mediatek Genio 700 Flash Log

The same serial connection can be used before connecting the board to capture boot messages. Refer to Mediatek Genio 700 Boot Log for a full boot log.

Also you can read the Boot ROM messages using the following command before connecting the board:

# To read boot ROM
sudo minicom -D $DEVICE -b 115200

Example output:

F0: 102B 0000
F1: 1004 0000
F1: 1004 0000 [0200]
F3: 1006 0033 [0200]
F3: 4001 00E0 [0200]
F3: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0C00 0241 [0000]
G0: 1190 0000
EC: 0000 0000 [0000]
T0: 0000 01A7 [1010]
Jump to BL

Note: The device name may vary (e.g., /dev/ttyUSB0 or /dev/ttyACM0). You can verify it by running:

dmesg | grep tty

NeuroPilot

NeuroPilot (NP) is a collection of software tools and APIs which are at the center of MediaTek's AI ecosystem. These tools are designed to fulfill the goal of “Edge AI”, which means that AI processing is performed locally on the device rather than remotely on a server. With NP, users can develop and deploy AI applications on edge devices with extremely high efficiency. This makes a wide variety of AI applications run faster, while also keeping data private.

The NeuroPilot ecosystem is distributed in three access tiers: Public, Basic and Premium, each providing different levels of documentation, tools, and platform support. Refer to: NeuroPilot Versions to learn more about it.

In addition, NeuroPilot evolves across multiple SDK generations. For example, the Genio 700 platform is associated with NeuroPilot 6, while NeuroPilot 8 (NP8) provides newer tooling and maintains backward compatibility with previous versions. For this reason, NP8 is typically recommended even when working with platforms originally designed for earlier releases.

Also refer to Neuropilot Basic documentation to learn about each specific version.

NeuroPilot Tools

Tools Deployment [3]

NeuroPilot provides a complete toolchain for converting, compiling, and executing AI models on MediaTek hardware accelerators such as the MDLA (NPU). The most relevant components are:

• Neuron SDK (NeuronSDK): The Neuron SDK is the core deployment package for MediaTek platforms. It contains the tools for model compilation and execution. The SDK is split into host and target components:
  • The host-side tools (e.g., ncc-tflite) are used on a PC to compile models.
  • The target-side runtime (e.g., neuronrt) is used on the device to execute them.
• Neuron Compiler (ncc-tflite): A tool included as part of the Neuron SDK. It performs offline compilation, converting TensorFlow Lite models into hardware-specific binaries known as DLA (Deep Learning Archive) files. This tool is intended to run on the host, although it can also be executed on the device in certain scenarios.
• Neuron Runtime (neuronrt): Also part of the Neuron SDK, this runtime is deployed on the target device. It is responsible for loading and executing .dla models on the NPU, and is typically used for inference, validation, and benchmarking.
• MTK Converter (mtk_converter): A separate tool that is used to convert models from frameworks such as PyTorch, TensorFlow, or ONNX into TensorFlow Lite format, with support for quantization and compatibility targeting specific NeuroPilot versions.
• Neuron Delegate (TFLite Delegate): A runtime component that integrates with TensorFlow Lite to enable on-the-fly execution of .tflite models on the NPU. Instead of requiring offline compilation to .dla, it dynamically offloads supported operations during inference. This provides flexibility, but with less control and predictability compared to the offline compilation workflow.

Offline vs Online Compilation

NeuroPilot supports two execution paradigms depending on how models are prepared for the NPU: offline compilation and online execution.

In the offline compilation workflow, models are first converted to TensorFlow Lite and then compiled into hardware-specific binaries (.dla) using the Neuron Compiler (ncc-tflite). This process is typically performed on a host machine and produces a fully optimized model tailored for the target hardware. The resulting .dla file is then deployed to the device and executed using the Neuron Runtime. This approach provides maximum control, predictability, and compatibility with the MDLA, and is the recommended workflow for production deployments.

In contrast, the online execution workflow uses the TensorFlow Lite Delegate to execute .tflite models directly on the device. In this mode, supported operations are dynamically offloaded to the NPU at runtime, while unsupported ones may fall back to the CPU. This approach is more flexible and easier to prototype with, but offers less control over performance and hardware utilization compared to offline compilation.

This guide focuses on the offline compilation workflow, as it provides the most reliable and optimized path for deploying models on MediaTek NPUs.

NeuroPilot Tools Installation

Before installing NeuroPilot tools, it is highly recommended to create a dedicated Python virtual environment.

Installing the packages directly into the system Python may introduce dependency conflicts and can potentially break existing Python applications or tools installed on your system.

Python virtual environments provide an isolated workspace where NeuroPilot dependencies can be installed safely without affecting the global Python installation.

Example using Python venv:

# Create a virtual environment
python3 -m venv neuropilot_env

# Activate the environment
source neuropilot_env/bin/activate

Once the environment is activated, continue with the installation of the NeuroPilot tools.

Ways to Obtain the Installation Files

NeuroPilot is distributed in multiple SDK generations and access tiers.

The available versions are:

• NeuroPilot 4 (NP4)
• NeuroPilot 6 (NP6)
• NeuroPilot 8 (NP8) (latest version)
• NeuroPilot Public (public NP8 MTK Converter)

Each version provides different releases of the available tools and supports different MediaTek platforms and MDLA generations. Choosing the correct version is important because it determines the compatibility and capabilities available for your target hardware.

The most important tools to consider for this guide are:

• MTK Converter: used for model conversion and quantization
• Neuron SDK: used for model compilation and runtime execution on MediaTek NPUs

In most cases, NP8 tools are recommended because they provide backward compatibility with previous NeuroPilot generations. For example, NP8 converter tools can still target NP6 platforms using compatibility settings.

MTK Converter

Obtaining the MTK Converter

It is recommended to use the latest NP8 converter version whenever possible and avoid older converter releases unless your platform specifically requires them.

There are two options to obtain the NP8 MTK Converter:

1. Download version 8.13.0 directly from the Public Downloads Section.

You may see a zip file like mtk_converter-8.13.0_packages.zip .

2. Obtain version 8.16.0 from the NeuroPilot SDK All-In-One Bundle available in the NeuroPilot 8 Downloads Section.

This bundle includes newer converter release together with additional tools such as Neuron SDK and quantization utilities.

MTK Converter is located in a zip file:

neuropilot-sdk-basic-8.0.11-build20260211/offline_tool/mtk_converter-8.16.0_packages.zip

Once you have downloaded the desired MTK Converter package, follow these steps to install it:

1. Unzip the file and enter to directory

2. Depending of your Python version you need to use the following command to install it:

# For Python 3.X
$ pip3 install mtk_converter-<version>+release-cp3X-cp3Xm-manylinux_2_5_x86_64.manylinux1_x86_64.whl

Note: Replace the <version> with the version of the corresponding file and the cp3X with your python version.

Example:

# For Python 3.10 and MTK Converter 8.13.0 version
$ pip3 install mtk_converter-8.13.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

To check if installation was successful, use:

python3 -c 'import mtk_converter; print(mtk_converter.__version__)'

Expected output:

8.13.0

Refer to the Converter Tool Installation Guide if you need more information.

Neuron SDK

Obtaining the Neuron SDK

You may need to select your Neuron SDK version according to the platform device that you are using.

MediaTek Genio 700 is corresponds to the MT8390 SoC and uses NeuroPilot 6. To find the correct Neuron SDK, enter to NP 6 Downloads page and locate the Neuron SDK package compatible with your platform.

For your respective platform you can search your MT SoC platform using the following options:

• Respective SoC datasheet.
• Where-to-download-the-correct-neuropilot-sdk-for-iot-platforms
• NeuroPilot Documentation in platform section or in Download Section of each NP version.

For example with the MediaTek Genio 700 (MT8390), it is necessary to download from NP6 Downloads the file:

20240108_Neuron_SDK_v1.2402.01_neuron-6-0-release.tar.gz

Once you have downloaded your corresponding neuronsdk , such as MT8390, extract the zip file and you will see a directory with this structure:

20240108_Neuron_SDK_v1.2402.01_neuron-6-0-release$ tree -L 2
.
├── host
│   ├── bin
│   ├── include
│   └── lib
├── mt2715 -> mt8195
├── mt6983
│   ├── bin
│   ├── include
│   └── lib
├── mt8188
│   ├── bin
│   ├── include
│   └── lib
├── mt8195
│   ├── bin
│   ├── include
│   └── lib
├── mt8390 -> mt8188
├── mt8395 -> mt8195
└── mt8798 -> mt6983

Here you can find specific versions of tools for your platform, but the most important directory is host, which contains Neuron Compiler (ncc-tflite) and it is required to compile TensorFlow Lite models into DLA format on the host PC.

Before exporting a model to DLA, you will need to set the following variable:

export LD_LIBRARY_PATH=20240108_Neuron_SDK_v1.2402.01_neuron-6-0-release/host/lib

It is needed to execute ncc-tflite.

Workflow to Convert a Model to DLA

The following section demonstrates the complete offline compilation workflow required to convert a neural network model into a MediaTek DLA model.

This example starts from a yolo11n-pose.pt model, which is first converted to ONNX format and then compiled into a DLA model compatible with the MediaTek NPU.

The workflow consists of the following steps:

1. Environment setup
2. Convert the model to ONNX
3. Convert the ONNX model to quantized TensorFlow Lite
4. Compile the TensorFlow Lite model into DLA format

Convert model from ONNX to DLA

For this example, we are converting a model yolo11n-pose.pt converted to ONNX model to DLA

You can refer to mediatek yolov8s to see a convertion by torchscript and Body Pose Estimation Example to see an conversion example from ONNX to TFLite.

Environment setup

The first step is to create your work directory

mkdir work_dir
cd work_dir

Create Python env

Use the following command:

python -m venv np_venv
source np_venv/bin/activate

Copy NeuroPilot files

For Genio 700 you have to use 20240108_Neuron_SDK_v1.2402.01_neuron-6-0-release as your neuron_sdk and it is recommended to use the lastest version of mtk_converter, which is included as part of NeuroPilot SDK All-In-One Bundle/offline_tool/mtk_converter-8.16.0_packages.zip, but feel free to use the public version of mtk_converter (8.13 at this moment).

Copy your neuron_sdk version and mtk_converter directories to your work_dir, also add your model into a directory. You may have the following files:

ls
20240108_Neuron_SDK_v1.2402.01_neuron-6-0-release  Models  mtk_converter-8.16.0_packages.zip

ls Models
yolo11n-pose.pt

Install dependencies

In your python env install the nex dependencies:

pip install ultralytics==8.4.22 onnx==1.13.1 onnxruntime==1.23.0 onnxsim==0.4.36 onnxslim==0.1.91 numpy==1.23.5 ml-dtypes==0.5.4 colorama==0.4.6

Also install your MTK Converter acording to your version:

unzip mtk_converter-8.16.0_packages.zip -d mtk_converter
pip install mtk_converter/mtk_converter-8.16.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Convert your model to ONNX:

yolo export model=Models/yolo11n-pose.pt format=onnx imgsz=640 opset=12 simplify=True dynamic=False nms=False batch=1

Convert your Onnx model to TFLite quantized

Create a Python script onnx_to_TFLite.py

import os
import argparse
import onnx
import numpy as np
import mtk_converter

class ONNXToTFLiteConverter:
    """Converter class to transform ONNX models to TensorFlow Lite format."""
    
    def __init__(
        self,
        input_shape=None,
        layout="nhwc",
        ir_version=7,
        export_spec="builtin_ignore_version",
        quantize=False,
        calib_dir="dataset/calibration_dataset",
        input_names=None,
        input_shapes=None,
    ):
        self.input_shape = input_shape or [1, 640, 640, 3]

        self.layout = layout
        self.ir_version = ir_version
        self.export_spec = export_spec
        self.quantize = quantize
        self.calib_dir = calib_dir

        self.input_names = (
            input_names.split(",") if isinstance(input_names, str) else input_names
        )

        self.input_shapes = input_shapes

    
    def _add_transpose_nhwc(self, model_proto):
        """Add a transpose operation to convert layout."""
        input_nhwc = onnx.helper.make_tensor_value_info(
            "input_nhwc",
            onnx.TensorProto.FLOAT,
            self.input_shape
        )

        input_transpose = onnx.helper.make_node(
            "Transpose",
            ["input_nhwc"],
            [model_proto.graph.input[0].name],
            perm=[0, 3, 1, 2]
        )

        model_proto.graph.node.insert(0, input_transpose)
        model_proto.graph.input.pop(0)
        model_proto.graph.input.insert(0, input_nhwc)

        onnx.checker.check_model(model_proto)
        return model_proto

    def _prepare_model(self, input_path):
        """Load and prepare the ONNX model for conversion."""
        model = onnx.load(input_path)
        model.ir_version = self.ir_version

        if self.layout == "nhwc":
            model = self._add_transpose_nhwc(model)

        output_path = "prepared_model.onnx"
        onnx.save(model, output_path)
        return output_path

    def _create_converter(self, model_path):
        """Create and configure an ONNX converter for the given model."""
        if self.input_names or self.input_shapes:
            converter = mtk_converter.OnnxConverter.from_model_proto_file(
                model_path,
                self.input_names,
                self.input_shapes
            )
        else:
            converter = mtk_converter.OnnxConverter.from_model_proto_file(model_path)

        converter.decompose_silu_ops = True
        converter.decompose_grouped_conv2d_ops = True
        converter.decompose_log_softmax_ops = True

        return converter

    def _get_calibration_data(self):
        """Generate calibration data from numpy files in the calibration directory."""
        for fn in sorted(os.listdir(self.calib_dir)):
            if not fn.endswith(".npy"):
                continue

            path = os.path.join(self.calib_dir, fn)

            try:
                arr = np.load(path, allow_pickle=False)
                yield [arr.astype(np.float32)]
            except Exception as e:
                print(f"[WARN] Skipping {path}: {e}")

    def _apply_quantization(self, converter):
        """Apply quantization settings to the converter."""
        converter.quantize = True
        converter.calibration_method = "Percentile"
        converter.calibration_percentile_value = 99.9
        converter.calibration_data_gen = self._get_calibration_data
        converter.precision_proportion = {"16W16A": 1.0}
        return converter

    def convert(self, input_onnx_path, output_tflite_path):
        """Convert an ONNX model to TensorFlow Lite format."""
        prepared_model_path = self._prepare_model(input_onnx_path)

        converter = self._create_converter(prepared_model_path)

        if self.quantize:
            converter = self._apply_quantization(converter)

        converter.convert_to_tflite(
            output_file=output_tflite_path,
            tflite_op_export_spec=self.export_spec
        )

def parse_shapes(shapes_str):
    """Parse shape strings into a list of integer shape tuples."""
    if shapes_str is None:
        return None
    return [list(map(int, s.split(','))) for s in shapes_str.split(';')]

if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Convert ONNX model to TFLite"
    )

    parser.add_argument("input_model", type=str)
    parser.add_argument("output_model", type=str)

    parser.add_argument("--input_names", type=str, default=None)
    parser.add_argument("--input_shapes", type=str, default=None)

    parser.add_argument("--quantize", action="store_true")

    parser.add_argument("--tflite_op_export_spec",
                        default="builtin_ignore_version")

    parser.add_argument("--dataset_dir", default="dataset/calibration_dataset")

    parser.add_argument("--layout",
                        default="nhwc",
                        choices=["nhwc", "nchw"])

    parser.add_argument("--ir_version", type=int, default=7)

    args = parser.parse_args()

    input_shapes = parse_shapes(args.input_shapes)

    converter = ONNXToTFLiteConverter(
        input_shape=input_shapes[0] if input_shapes else None, 
        layout=args.layout,
        ir_version=args.ir_version,
        export_spec=args.tflite_op_export_spec,
        quantize=args.quantize,
        calib_dir=args.dataset_dir,
        input_names=args.input_names,
        input_shapes=input_shapes,
    )

    converter.convert(
        args.input_model,
        args.output_model
    )

Conversion script explanation

This script implements the pipeline to convert an ONNX model into a TensorFlow Lite model compatible with the MediaTek toolchain.

The conversion process consists of several stages:

1. The ONNX model is loaded and adapted to meet the converter requirements. This includes adjusting the IR version and modifying the input layout if needed.
2. The model layout is explicitly transformed to NHWC format. By default, ONNX models use NCHW layout, while TensorFlow Lite expects NHWC. The converter automatically inserts transpose operations to handle this difference, which can introduce runtime overhead. To avoid this, the script modifies the model beforehand to align with the expected layout.
3. A converter object is created using the MediaTek Converter Tool. This object is responsible for translating the ONNX model into TensorFlow Lite format and includes configuration options to ensure compatibility with the target NPU.
4. If quantization is enabled, a calibration dataset is used to estimate value ranges within the model. This step reduces model size and improves execution efficiency on the NPU.
5. Finally, the model is converted into TensorFlow Lite format, producing a .tflite file that can be further compiled into a DLA model.

Model Preparation

    def _add_transpose_nhwc(self, model_proto):
        """Add a transpose operation to convert layout."""
        input_nhwc = onnx.helper.make_tensor_value_info(
            "input_nhwc",
            onnx.TensorProto.FLOAT,
            self.input_shape
        )

        input_transpose = onnx.helper.make_node(
            "Transpose",
            ["input_nhwc"],
            [model_proto.graph.input[0].name],
            perm=[0, 3, 1, 2]
        )

        model_proto.graph.node.insert(0, input_transpose)
        model_proto.graph.input.pop(0)
        model_proto.graph.input.insert(0, input_nhwc)

        onnx.checker.check_model(model_proto)
        return model_proto

    def _prepare_model(self, input_path):
        """Load and prepare the ONNX model for conversion."""
        model = onnx.load(input_path)
        model.ir_version = self.ir_version

        if self.layout == "nhwc":
            model = self._add_transpose_nhwc(model)

        output_path = "prepared_model.onnx"
        onnx.save(model, output_path)
        return output_path

The first step in the conversion process is to prepare the ONNX model to ensure compatibility with the MediaTek Converter Tool.

This includes adjusting the ONNX IR version and modifying the input layout when necessary. Most ONNX models use the NCHW layout (channels-first), while TensorFlow Lite expects NHWC (channels-last).

By default, the converter automatically inserts transpose operations to handle this layout difference. However, this may introduce unnecessary overhead during inference. To avoid this, the script explicitly modifies the model to use NHWC layout before conversion.

At the end of this stage, the model is structurally aligned with the expected format for the conversion process.

Converter Object Creation

    def _create_converter(self, model_path):
        """Create and configure an ONNX converter for the given model."""
        if self.input_names or self.input_shapes:
            converter = mtk_converter.OnnxConverter.from_model_proto_file(
                model_path,
                self.input_names,
                self.input_shapes
            )
        else:
            converter = mtk_converter.OnnxConverter.from_model_proto_file(model_path)

        converter.decompose_silu_ops = True
        converter.decompose_grouped_conv2d_ops = True
        converter.decompose_log_softmax_ops = True

        return converter

Once the model is prepared, a converter object is created using the MediaTek Converter Tool.

This object is responsible for translating the ONNX model into TensorFlow Lite format. During initialization, the converter can be configured with input names and shapes if required.

Additionally, several compatibility options are enabled to ensure that unsupported operations are properly handled. Some operators that are not directly supported by the target NPU are decomposed into equivalent operations that can be executed efficiently.

This step ensures that the model is compatible with the MediaTek hardware before conversion.

Post-Training Quantization (PTQ):

    def _get_calibration_data(self):
        """Generate calibration data from numpy files in the calibration directory."""
        for fn in sorted(os.listdir(self.calib_dir)):
            if not fn.endswith(".npy"):
                continue

            path = os.path.join(self.calib_dir, fn)

            try:
                arr = np.load(path, allow_pickle=False)
                yield [arr.astype(np.float32)]
            except Exception as e:
                print(f"[WARN] Skipping {path}: {e}")

    def _apply_quantization(self, converter):
        """Apply quantization settings to the converter."""
        converter.quantize = True
        converter.calibration_method = "Percentile"
        converter.calibration_percentile_value = 99.9
        converter.calibration_data_gen = self._get_calibration_data
        converter.precision_proportion = {"16W16A": 1.0}
        return converter

Quantization is an optional but highly recommended step in the conversion process, as it reduces model size and improves performance on the NPU.

When enabled, the converter uses a calibration dataset to estimate the numerical range of activations within the model. This allows the model to be converted from floating-point representation to lower precision formats such as 16-bit or 8-bit.

The calibration dataset typically consists of representative input samples stored as numpy arrays. These samples are passed through the model to determine appropriate scaling factors.

Proper quantization is critical for achieving optimal performance and efficiency on edge devices, especially when deploying models to dedicated accelerators.

Model Conversion

    def convert(self, input_onnx_path, output_tflite_path):
        """Convert an ONNX model to TensorFlow Lite format."""
        prepared_model_path = self._prepare_model(input_onnx_path)

        converter = self._create_converter(prepared_model_path)

        if self.quantize:
            converter = self._apply_quantization(converter)

        converter.convert_to_tflite(
            output_file=output_tflite_path,
            tflite_op_export_spec=self.export_spec
        )

In the final step, the configured converter is used to generate the TensorFlow Lite model.

During this stage, all previous configurations are applied, including layout adjustments, operator decomposition, and optional quantization. The converter processes the ONNX model and produces a .tflite file as output.

This TensorFlow Lite model is fully compatible with the MediaTek toolchain and can be used as input for the next stage of the workflow, where it is compiled into a DLA file using the Neuron SDK.

Convert ONNX model to TFLite

Run the script to generate a TFLite model:

#Use --quantize to generate a quantized model
python3 onnx_to_tflite.py models/yolo11n-pose.onnx models/yolo11n-pose.tflite --quantize

Converter Configuration

The previous script provides a basic example of the ONNX to TensorFlow Lite conversion process. However, the MediaTek Converter Tool allows further customization of the generated model. In this section, we describe the most important configuration options.

Model Precision

The converter allows generating models with different numerical precision levels, which directly impact performance and accuracy. The precision is configured using the precision_proportion attribute.

#To create a fp32 model use:
converter.precision_proportion = {'FP': 1.0}

#To create an int16 model use:
converter.precision_proportion = {'16W16A': 1.0}

#To create an int8 model use:
converter.precision_proportion = {'8W8A': 1.0}

# To create a mixed model you can use combinations like
converter.precision_proportion = {'8W16A': 1.0} # input int8, output int16
converter.precision_proportion = {'16W8A': 1.0} # input int16, output int8

Target SDK Compatibility

The converter can generate TensorFlow Lite models targeting different versions of the NeuroPilot SDK.

This is controlled by the `TFLite_op_export_spec` parameter, which determines the operator set and compatibility level.

#To convert model to tflite, you should use: 
converter.convert_to_tflite(
output_file = output_model_path,
tflite_op_export_spec = 'builtin_ignore_version')

Some available options include:

- npsdk_v5, npsdk_v6, npsdk_v7: target specific SDK versions.

- builtin_first: default behavior, prioritizes newer operator definitions.

- custom_first: prioritizes MediaTek custom operators (MTKEXT).

- builtin_ignore_version: avoids introducing custom operators (MTKEXT) and improves compatibility.

Operator Compatibility

Some neural network operations are not directly supported by the MediaTek NPU.

To ensure compatibility, the converter provides options to decompose these operations into supported ones.

#This operations can not be supported by Genio 700 mdla, so you can decompose them
converter.decompose_silu_ops = True
converter.decompose_grouped_conv2d_ops = True
converter.decompose_log_softmax_ops = True

(Optional) Model to TFLite conversion without MTK converter

MediaTek recommends using MTK Converter to generate TensorFlow Lite models because it provides compatibility optimizations, quantization utilities, and model transformations specifically designed for NeuroPilot workflows.

However, it is also possible to generate TensorFlow Lite models using external conversion tools, provided that the resulting model only contains operations supported by the target NeuroPilot version and the Neuron Compiler ncc-tflite.

This means that some TensorFlow Lite models generated without MTK Converter may still be successfully compiled into DLA format.

Note:

• Compatibility is not guaranteed when using external conversion workflows.

• Even if a model is successfully converted to TensorFlow Lite, the compilation step may fail if unsupported operators remain in the graph.

For this reason, MTK Converter remains the recommended workflow for production deployments.

As an example, the following command was used to export a YOLO Pose model directly into an INT8 TensorFlow Lite model:

yolo export \
    model=yolov8s-pose.pt \
    format=tflite \
    imgsz=640 \
    opset=12 \
    simplify=True \
    dynamic=False \
    nms=False \
    batch=1 \
    int8=True

The generated TensorFlow Lite model was compatible with the Neuron Compiler and could be successfully compiled into DLA format.

However, similar export settings applied to other models such as yolo11n-pose resulted in unsupported operations during compilation.

This shows that compatibility depends on the final operator graph generated by the export process rather than the original framework model itself.

To know about what model operations support your board use:

ncc-tflite --show-builtin-ops

Convert your TFLite model to DLA

Export the path of neuron_sdk dependencies

#If you are using a target for NP 8 use:
export LD_LIBRARY_PATH=/neuron_sdk/host/lib

#else use the host path of Neuron_sdk of your target device
export LD_LIBRARY_PATH=20240108_Neuron_SDK_v1.2402.01_neuron-6-0-release/host/lib

Convert the TFLite model to DLA using ncc-tflite:

20240108_Neuron_SDK_v1.2402.01_neuron-6-0-release/host/bin/ncc-tflite --arch=mdla3.0 yolo11n-pose-quant-int16.tflite

Refer to: How does the `–opt-accuracy` flag affect ncc-tflite (TFLite-to-DLA) conversion results and performance? to know more about this. Finally you can see the following model files, including your dla model:

work_dir$ ls models/
yolo11n-pose.pt         yolo11n-pose-quant-int16.tflite         
yolo11n-pose.onnx         yolo11n-pose-quant-int16.dla

Now you can use your dla file in your MediaTek Genio 700.

How to execute DLA inferences

Once a model has been compiled into DLA format, it can be executed on the target MediaTek platform using different runtime approaches.

The two most common methods are:

• Neuron Runtime API
  • Low-level C/C++ runtime API
  • Direct execution of DLA models
  • Greater control over tensor management and execution flow

• NNStreamer
  • GStreamer-based inference pipeline
  • Recommended for multimedia and streaming applications
  • Easier integration with cameras, video and image pipelines

This guide focuses on the NNStreamer workflow because it provides a simpler integration path for video and AI applications running on MediaTek Genio platforms.

Get model parameters

Before running inferences, it is important to determine the expected tensor layout and quantization parameters of the model.

These parameters define how the input tensor should be interpreted and processed by the neural network.

Parameters to get

Scale and zero point

Quantized models also require scale and zero point parameters.

These values are used to convert between floating-point values and quantized integer representations.

• Scale is the multiplication factor used to convert quantized integer values into floating-point values.
• Zero point defines the integer value representing zero

Together, these parameters allow the model to reconstruct approximate floating-point values during inference.

Tensor layout

The input layout defines how tensor dimensions are organized in memory.

The most common layouts are:

• NCHW: Batch x Channels x Height x Width
• NHWC: Batch x Height x Width x Channels

Using an incorrect layout may produce invalid inference results even if the pipeline executes successfully.

Inspecting parameters of TFLite with Netron

The easiest way to inspect these parameters is by opening the generated TensorFlow Lite model in Netron.

To inspect the model:

1. Open the .tflite file in Netron
2. Select the input tensor
3. Inspect the tensor metadata

Example of tensor metadata displayed in Netron::

Netron Parameters of tflite model quantized

The tensor metadata includes:

• Tensor dimensions: [1, 640, 640, 3] input, [1, 56, 8400] output
• Tensor layout: NHWC, determined by the tensor dimensions. Output doesn't follow the same layout, it is the specific yolo-pose layout
• Tensor type: int16 input an output
• Quantization scale: 0.0000305 for the input tensor and 0.01985441 for the output tensor.
• Quantization zero point: 0 for both input and output tensors.

Quantization in TensorFlow Lite uses a linear transformation:

$RealValue=Scale*(q-ZeroPoint)$

where:

• q is the quantized integer value
• scale is the conversion factor
• zero_point defines which integer value represents the real value zero

In Netron, these values can be identified directly from the quantization equation shown in the tensor metadata.

-1.00003 <= 0.0000305 × q + 0 <= 1

In this example:

• 0.0000305 is the quantization scale
• q represents the quantized integer tensor values
• Since no offset is applied, the zero point is 0

The quantization parameters used in the .tflite model are preserved during the DLA compilation process.

Inspecting parameters of DLA with Neuron Runtime

It is also possible to inspect the input and output tensor dimensions directly from the DLA model using:

neuronrt -a model.dla -d

Example output:

neuronrt -a yolo11n-pose-quant.dla -d

INFO: dlopen libneuronusdk_runtime.mtk.so
Input :
	Handle = 0, <1 x 640 x 640 x 3>, size = 2457600 bytes
Output :
	Handle = 0, <1 x 1 x 56 x 8400>, size = 940800 bytes
ERROR: The number of input files should be 1
ERROR: Inference fail

The runtime may display an inference error because no input tensor was provided. This can be ignored when only inspecting model information.

You can identify the tensor layout with this command, but it doesn't provide any scale or zero point. It is a better option to use your TFLite model in Netron to get this parameter.

Inspecting parameters of TFLite with Neuron Compiler

The Neuron Compiler also provides a command-line method to inspect the input and output tensor information directly from a TensorFlow Lite model.

Use the following command:

ncc-tflite --show-io-info model.tflite

Example output:

subgraph[0]
# of input tensors: 1
[0]: input_nhwc
 ├ Type: kTfLiteInt16
 ├ AllocType: kTfLiteArenaRw
 ├ Shape: {1,640,640,3}
 ├ Scale: 3.05185e-05
 ├ ZeroPoint: 0
 └ Bytes: 2457600

# of output tensors: 1
[0]: output0
 ├ Type: kTfLiteInt16
 ├ AllocType: kTfLiteArenaRw
 ├ Shape: {1,56,8400}
 ├ Scale: 0.0198544
 ├ ZeroPoint: 0
 └ Bytes: 940800

This command displays the same input and output tensor information required to perform inference, including:

• Tensor shape
• Tensor data type
• Quantization scale
• Quantization zero point
• Tensor memory size

Run a simple inference

To run a simple inference create an input binary file using dd command with de number of bytes expected by the input tensor of your model.

As example create a dummy input:

dd if=/dev/zero of=input.bin bs=1 count=2457600

Expected output:

2457600+0 records in
2457600+0 records out
2457600 bytes (2.5 MB, 2.3 MiB) copied, 3.16202 s, 777 kB/s

Then use it as input for your model:

neuronrt -m hw -a yolo11n-pose-quant.dla -i input.bin -o output.bin -c 10

Expected output:

INFO: dlopen libneuronusdk_runtime.mtk.so
Inference repeats for 10 times.
Total inference time = 252 ms

That means your model can run succesfully inferences on NPU hardware.

How to run DLA with NNStreamer

What is NNstreamer

NNStreamer is a GStreamer-based framework that enables neural network inference within multimedia pipelines. It operates by converting video frames into tensors, applying transformations, and passing them to a neural network backend.

Elements of NNStreamer

tensor_converter

The tensor_converter element transforms multimedia buffers into tensor streams.

For video inputs, it converts video/x-raw frames into other/tensor buffers that can be processed by neural network elements such as tensor_filter.

The generated tensor preserves the original memory organization of the incoming video frame. This means that tensor_converter does not perform:

• Memory rearrangement
• Layout transformations
• Channel reordering

For example, an RGB frame stored in memory as RGBRGBRGB... will remain in the same channel order after conversion.

Internally, NNStreamer represents video tensors using dimensions in the form:

Channels:Width:Height:Batch

Example

3:640:640:1

This representation describes the raw tensor dimensions generated from the video frame:

• 3 channels (RGB)
• 640 width
• 640 height
• 1 frame per tensor

However, these dimensions do not necessarily represent the semantic tensor layout expected by the neural network model (NHWC or NCHW).

Although tensor_transform can be used to modify tensor dimensions or apply transpose operations, these operations may reinterpret or reorganize tensor data in memory.

Because tensor_converter preserves the original video frame memory organization, applying additional layout transformations may produce invalid inference results if the model was not designed for those transformations.

You can refer to NNStreamer Tensor Converter to learn more about it.

tensor_transform

This element applies operations such as

• Type casting (typecast)
• Arithmetic (add, mul, div)
• Dimension changes (dimchg)
• Transpose (transpose)

Note: Arithmetic and typecast are generally safe, but transpose and dimchg can change how data is interpreted and may break the pipeline if used incorrectly

Use example:

... ! tensor_transform mode=arithmetic option=typecast:float32,div:255.0,div:{MODULE_INPUT_SCALE} !
tensor_transform mode=typecast option={DATA_MODEL_TYPE} ! ...

In this example you can see , it takes an other/tensor and applies a typecast of the data to float32, later applies a division

Avoid concatenate multiple typecast in the same element, it may cause an error.

You can refer to NNStreamer Tensor Transform to learn more about it.

tensor filter

This is the main element of the whole NNStreamer project. This connects gstreamer data stream with neural network frameworks (NNFW) such as Tensorflow, also you can use this element to execute on Neuron Framework for .dla files.

The main tensor_filter properties related to tensor type and dimension are

• inputtype: Type of each element of the input tensor.
• inputlayout: Channel-first (NCHW), channel-last (NHWC), or none for input data.
• input: Input tensor dimension, up to 4 dimensions.
• outputtype: Type of each element of the output tensor.
• outputlayout: Channel-first (NCHW), channel-last (NHWC), or none for output data.
• output: Output tensor dimension, up to 4 dimensions.

Use example

... !
tensor_filter framework=neuronsdk model=/path/to/dla/file inputtype=uint8 input=3:640:640:1 outputtype=uint8 output=1001:1 !
...

In this example you can supose it is a model with a uint8 input and a uint8 output.

You can refer to NNStreamer Tensor Filter to learn more about it.

tensor_mux

The tensor_mux element combines multiple tensor streams into a single other/tensors stream.

This element is useful when a neural network requires multiple input tensors, or when several tensor streams need to be synchronized and processed together inside the same pipeline.

tensor_sink

The tensor_sink element is a sink plugin used to receive tensor data from an NNStreamer pipeline. It is used to extract tensors from the pipeline and make them available to an application.

When tensors arrive from upstream elements, tensor_sink emits signals that allow applications to access the tensor buffers directly.

This element is commonly used for:

• Debugging tensor data
• Accessing inference results from applications
• Custom post-processing
• Monitoring intermediate pipeline outputs

The element supports both:

• other/tensor
• other/tensors

It exposes tensor buffers to the application layer.

One important signal provided by tensor_sink is:

• new-data: Triggered whenever a new tensor buffer is received

Example usage:

videotestsrc ! tensor_converter ! tensor_sink

In this example:

1. Video frames are generated
2. Frames are converted into tensors
3. The tensors are delivered to the application through tensor_sink

You can refer to the NNStreamer tensor_sink documentation to learn more about it.

How to use NNStreamer

Take the following pipeline as example

    pipeline_tensor_str = f"""
    t. ! queue !
    tensor_converter !
    tensor_transform mode=arithmetic option=typecast:float32,div:255.0,div:{INPUT_SCALE} !
    tensor_transform mode=typecast option={DTYPE} !
    tensor_filter framework=neuronsdk model={MODEL_FILE} 
    input=3:{IMG_SIZE}:{IMG_SIZE}:1 inputtype={DTYPE} inputlayout=None 
    output=1:56:8400 outputtype={DTYPE} !
    tensor_sink name=tensor_sink
    """

This example is used for a quantized yolo11n-pose model,

Understanding inputlayout

In tensor_filter, inputlayout tells the element how to interpret the buffer. It does not modify input data.

This means that if you are using an NHWC model and tensor_converter produces a tensor like

 3:{IMG_SIZE}:{IMG_SIZE}

But if your model expects

 1:{IMG_SIZE}:{IMG_SIZE}:3

You will have a mismatch.

For example:

    pipeline_tensor_str = f"""
    tensor_converter !
    ...
    tensor_filter ... input=1:{IMG_SIZE}:{IMG_SIZE}:3 inputlayout=NHWC !
    ...
    """

This will produce an error like:

The given gstcap is other/tensor, framerate=(fraction)0/1, dimension=(string)3:640:640:1:1:1:1:1, type=(string)int16, which is not compatible: 0 : int16 [3:640:640:1:1:1:1:1] | int16 [1:640:640:3:1:1:1:1] Not equal

How to solve error with inputlayout

To bypass this issue, you can use:

 tensor_filter 3:{IMG_SIZE}:{IMG_SIZE} inputlayout=none

This means that tensor_filter will take the input data as-is and will not try to reinterpret it as another layout. So you should make sure that the input values ​​are sorted in memory according to what your model expects.

This behaviour is because according to NNStreamer Component Description "the NNStreamer elements and data-pipeline, being agnostic to such semantics, should treat every other/tensor and other/tensors as multi-dimensional arrays of general numbers."

How to post-process the inferenced data

How to get a video and inference layout with tensor_sink

You can use a tensor_sink to get the video and the inference result.

To do this you can convert the input video to a tensor and separate it with a tee, use one of the tee sink pad connected to the input to tensor filter and procces it, later with a tensor_mux you can make an union of the original video tensor and the result of inference, take the following pipeline as example:

 ... ! video/x-raw,width=640,height=640,format=RGB,framerate=30/1 !
tensor_converter !
tee name=t

t. ! queue ! mux.sink_0

t. ! queue !
tensor_transform mode=arithmetic option=typecast:float32,div:255.0,div:0.00003051850 !
tensor_transform mode=typecast option=int16 !
tensor_filter framework=neuronsdk model=yolo11n-pose.dla
input=3:640:640:1 inputtype=int16 inputlayout=None 
output=1:56:8400 outputtype=int16 !
mux.sink_1

tensor_mux name=mux sync-mode=nosync !
tensor_sink name=tensor_sink

Later you can read each tensor in your callback function:

 def on_new_tensor(self, sink, buffer):
    """Callback for new tensor data from tensor_sink"""

    num_tensors = buffer.n_memory()
    raw_tensors = []
    model_dtype = np.int16

    for i in range(num_tensors):

        mem = buffer.peek_memory(i)
        success, map_info = mem.map(Gst.MapFlags.READ)
        if not success:
            continue

        # Tensor 0 = RGB frame
        if i == 0:
            np_dtype = np.uint8

        # Tensor 1 = inference output
        else:
            np_dtype = model_dtype

        data = np.frombuffer(map_info.data, dtype=np_dtype).copy()
        raw_tensors.append(data)
        mem.unmap(map_info)

You can use the function peek_memory to iterate over each tensor, but ensure you map your tensor with the correct data type, in this example you can see that frame format is uint8, but the model used uses int16. Put attention to this to avoid map incorrectly your frame or your inference tensor results.

This is only the raw data for each tensor, it is necessary to reshape te tensor according to the layout expected, layout of each tensor, for example, to get the formatted frame according to the original image (640x640 in RGB format):

        # Reconstruct frame
        frame = raw_tensors[0].reshape((640, 640, 3))

Also for the inference, the yolo11n-pose bring us an inference result with: 1:56:8400, which contains the result of the inference, this include scores, bounding boxes positions and keypoints detected. The 56 is divided into first 4 elements meaning the bounding box (x center, y center, width, heigh) the 5th element represents the confidence of the detection and the rest 51 elements represents the keypoints of the people detected, it is divided into 17 keypoints, each one with 3 vlaues: x, y and point confidence, each keypoint represents a part of the body (eyes, nose, wrists, anklees, etc.), this the meaning of the 56, the 8400 means that yolo produces 8400 possible detections. So you can process the output like:

         raw_output = raw_tensors[1]

        decoded = raw_output.reshape(1, 56, 8400).astype(np.float32)
        decoded = (decoded - model_zero_point) * model_scale
        pred = decoded[0].T

        # YOLO uses [0,3] xywh, [4] conf, [5:56] kpts
        boxes = pred[:, :4]
        scores = pred[:, 4]
        kpts = pred[:, 5:].reshape(-1, 17, 3)

Note that it has a Transpose operation, this produces the order mentioned before, if we don't use it the order will be: "All x box, all y box, all widths, all heigts, al confidence, etc.", which is more difficult to understand. (But it can be usefull if you want process this with a GPU)

With this information we can determine the result of the inference, to show in an image

First, if you need to convert the boxes from xywh to xyxy, use the following function:

     def decode_boxes(self, boxes):
        """ Convert from [x_center, y_center, width, height]
            to [x1, y1, x2, y2] """
        boxes_xyxy = np.zeros_like(boxes)

        x = boxes[:, 0]
        y = boxes[:, 1]
        w = boxes[:, 2]
        h = boxes[:, 3]

        x1 = x - w / 2
        y1 = y - h / 2
        x2 = x + w / 2
        y2 = y + h / 2

        boxes_xyxy = np.stack([x1, y1, x2, y2], axis=1)
        return boxes_xyxy

Also the model was converted with nms=false for compatibility with DLA conversion, Non-Maximun Supression (NMS) is a post-processing technique in computer vision, primarily used in object detection to eliminate redundant, overlapping bounding boxes. Yolo uses NMS in their intern final operations, but without NMS means we need to apply it manually, so we have to use the following function:

     def nms(self, boxes, scores):
        """ Non-Maximum Suppression to filter overlapping boxes """
        idxs = scores.argsort()[::-1]
        keep = []

        while len(idxs) > 0:
            i = idxs[0]
            keep.append(i)
            if len(idxs) == 1:
                break

            rest = idxs[1:]

            x1 = np.maximum(boxes[i, 0], boxes[rest, 0])
            y1 = np.maximum(boxes[i, 1], boxes[rest, 1])
            x2 = np.minimum(boxes[i, 2], boxes[rest, 2])
            y2 = np.minimum(boxes[i, 3], boxes[rest, 3])
            
            inter = np.maximum(0, x2 - x1) * np.maximum(0, y2 - y1)

            # Calculate areas for IoU computation
            area1 = (boxes[i, 2] - boxes[i, 0]) * (boxes[i, 3] - boxes[i, 1])
            area2 = (boxes[rest, 2] - boxes[rest, 0]) * \
                (boxes[rest, 3] - boxes[rest, 1])

            # Avoid division by zero
            iou = inter / (area1 + area2 - inter + 1e-6)

            # Keep boxes with IoU less than threshold
            idxs = idxs[1:][iou < self.iou_thres]

        return keep

This returns the boxes to keep, you can get the final result with:

         # Get absolute box coordinates
        boxes = self.decode_boxes(boxes)

        # Filter by confidence threshold
        mask = scores > self.conf_thres
        boxes = boxes[mask]
        scores = scores[mask]
        kpts = kpts[mask]

        # Apply NMS to remove overlapping boxes
        keep = self.nms(boxes, scores)

        return boxes[keep], scores[keep], kpts[keep]

Then you can draw this results over your frame to visualize it.