Top SDK TFLite LLM Speech Voice Tokenizer Tracker JA

QNN Backend Guide

How to run fast inference on the NPU of Qualcomm SoCs using the QNN backend of ailia SDK.

Overview

QNN is a backend API provided by Qualcomm, equivalent to NVIDIA's cuDNN. By using the QNN backend of ailia, you can use the NPU on Qualcomm SoCs.

Supported Platforms

Note: FP16 computation on the HMX of QNN requires Hexagon v69 or later. Therefore, a device with Snapdragon 8 Gen 1 or later is required.

Usage

QNN is planned to become available experimentally from ailia SDK 1.7 onwards.

Using on Android

In addition to libailia.so, bundle libailia_qnn.so with your application.

You need to declare the execution permission for libcdsprpc.so in AndroidManifest.xml:

<uses-native-library android:name="libcdsprpc.so" android:required="false"/>

You also need to add the following to build.gradle:

implementation 'com.qualcomm.qti:qnn-runtime:2.47.0'

Once added to build.gradle, QNN is automatically bundled into the apk.

A sample is available in ailia-models-kotlin.

Using on Windows

After running pip3 install ailia, download QNN and copy the following files from the zip to site-packages/ailia/windows/arm64, based on the path shown by pip3 show ailia.

https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk

Note: QNN has version dependencies, so please use the following:
v2.47.0.260601.zip
Note: The hexagon-v* directory must be selected according to the Hexagon architecture of your device. For Snapdragon X Plus, copy hexagon-v73. In addition to the so files, the cat files such as qairt\2.47.0.260601\lib\hexagon-v73\unsigned\libsnpehtpv73.cat must also be copied.
Note: The so files are not Windows libraries — they are binaries executed on the NPU (Hexagon). The DLLs run on Windows and dispatch processing to the NPU via FastRPC, while libQnnHtpV73.so is transferred to the NPU via FastRPC and loaded onto the Hexagon processor to execute the actual computation. Since Hexagon binaries use the ELF format (.so), the extension remains .so even when the host OS is Windows.

Architecture

ailia SDK takes ONNX as input, parses the graph, instantiates the layers, and performs inference using QNN in each layer. QNN (Qualcomm AI Engine Direct) offloads processing to the Hexagon NPU via libcdsprpc.so (FastRPC), and computation is executed on the HVX (vector unit) and HMX (matrix unit) inside the NPU. Operators not supported by QNN are executed by ailia_cpu, so the NPU and CPU operate in parallel. Computation is performed in FP16.

ONNX
Model file
↓
ailia
Graph management
↙ ↘
ailia_qnn
Layer management
↓
QNN
Inference runtime (Qualcomm AI Engine Direct / HTP backend)
↓
libcdsprpc.so
FastRPC (RPC interface between CPU and NPU)
↓
Hexagon NPU
HVX (vector unit) / HMX (matrix unit)
ailia_cpu
CPU layer processing (offload for operators not supported by QNN)
↓
CPU
Arm CPU

Performance

The NPU is approximately 5–10x faster than CPU inference. ailia automatically offloads operators not supported by QNN to the CPU.

A demo app for Android can be downloaded from the following link:

https://github.com/ailia-ai/ailia-models-kotlin/wiki/QNN

A demo app for Windows can be downloaded from the following link:

https://github.com/ailia-ai/ailia-models-flutter/wiki/QNN

Limitations

QNN only supports static graphs. In ailia SDK, if a dynamic graph is provided as input, inference is performed on the CPU.