ailia Speech NPU (QNN) Guide

Overview

ailia Speech 1.6.0 and later support Whisper inference on the NPU (Hexagon) of Qualcomm SoCs. Using the NPU lets you run speech recognition quickly and with low power consumption.

QNN (Qualcomm AI Engine Direct) is the backend API provided by Qualcomm. ailia Speech uses Whisper encoder and decoder models converted into an SoC-specific QNN Model (.qnn). A single .qnn contains both the encoder and the decoder, so the device needs only the .qnn and no ONNX files.

Supported Platforms

• Android (Snapdragon 8+ Gen 1 / sm8475, Snapdragon 7s Gen 3 / sm7635)
• Windows on Arm (Snapdragon X Plus / sc8380xp)
• QCS6490 (qcs6490)

A separate QNN Model is provided for each SoC. Use the .qnn that matches the SoC of your device. Models built for a different SoC cannot be reused.

Supported Models

Whisper Large V3 Turbo and Whisper Medium are currently supported. Both are quantized to Int16.

Model Download

Converted Whisper QNN Models can be downloaded below. Use the file that matches the SoC of your target device.

All files are built for ailia Speech 1.6.0 (QAIRT 2.47.0.260601).

Project Settings

Obtaining and deploying the QNN runtime is the same as for ailia SDK. For the Android AndroidManifest.xml and build.gradle settings, and for placing the QNN DLLs and the NPU-side .so / .cat files on Windows on Arm, see the ailia SDK QNN setup. Use QAIRT 2.47.0.260601, the same version used to generate the models.

On Android, a .qnn cannot be opened directly from a path inside assets. Copy it to the app's private storage or download it on first launch, then pass that real file path to the API below.

Code Sample: Opening the Model

Normally you pass separate encoder and decoder ONNX files at Open. When using a QNN Model, pass the same .qnn path as both the encoder and the decoder. For the model type, specify the model the .qnn was built from (Large V3 Turbo or Medium). The transcription APIs that follow are the same as when using ONNX.

C

#include "ailia_speech.h"
#include "ailia_speech_util.h"

const char *qnn_path = "/data/user/0/your.package/files/whisper-large-v3-turbo-sm7635-int16.qnn";

struct AILIASpeech *speech = NULL;
ailiaSpeechCreate(&speech, env_id, AILIA_MULTITHREAD_AUTO, memory_mode,
                  AILIA_SPEECH_TASK_TRANSCRIBE, AILIA_SPEECH_FLAG_NONE,
                  ailiaSpeechUtilGetCallback(), AILIA_SPEECH_API_CALLBACK_VERSION);
/* Pass the same .qnn as the encoder and the decoder */
ailiaSpeechOpenModelFileA(speech, qnn_path, qnn_path,
                          AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_LARGE_V3_TURBO);

Unity (C#)

using System.IO;
using UnityEngine;
using ailiaSpeech;

string qnnPath = Path.Combine(Application.persistentDataPath, "whisper-large-v3-turbo-sm7635-int16.qnn");
var speech = new AiliaSpeechModel();
// Pass the same .qnn as the encoder and the decoder
speech.Open(qnnPath, qnnPath, envId, memoryMode,
            AiliaSpeech.AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_LARGE_V3_TURBO,
            AiliaSpeech.AILIA_SPEECH_TASK_TRANSCRIBE,
            AiliaSpeech.AILIA_SPEECH_FLAG_NONE, "auto");

Flutter

import 'package:ailia_speech/ailia_speech.dart';

const qnnPath = '/data/user/0/your.package/files/whisper-large-v3-turbo-sm7635-int16.qnn';
final speech = AiliaSpeechModel();
// Pass the same .qnn as the encoder and the decoder
await speech.open(
  encoderFile: qnnPath,
  decoderFile: qnnPath,
  modelType: AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_LARGE_V3_TURBO,
);

JNI (Kotlin)

context is the Android Activity or Application Context.

val qnnPath = "${context.filesDir}/whisper-large-v3-turbo-sm7635-int16.qnn"
val speech = AiliaSpeech(
    envId = -1,
    task = AiliaSpeech.AILIA_SPEECH_TASK_TRANSCRIBE,
    flags = AiliaSpeech.AILIA_SPEECH_FLAG_NONE,
)
// Pass the same .qnn as the encoder and the decoder
speech.openModel(qnnPath, qnnPath,
    AiliaSpeech.AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_LARGE_V3_TURBO)

Benchmark

For measurement results on the NPU, see the ailia Speech NPU (QNN) Benchmark.

Evaluation App

The evaluation APK for Android is ailia-models-kotlin-qnn.apk. It runs on devices with Snapdragon 8+ Gen 1 (sm8475) or Snapdragon 7s Gen 3 (sm7635).

The evaluation app for Windows on Arm is ailia-models-flutter-qnn-v261020.zip. It runs on Snapdragon X Plus or QCS6490.

Limitations

• A QNN Model must be converted for each SoC; files built for another SoC cannot be used.
• The currently supported models are Whisper Large V3 Turbo / Medium.