ailia LLM NPU (QNN) Guide

Overview

ailia LLM 1.5.0 and later support fast inference on the NPU (Hexagon) of Qualcomm SoCs. For the NPU, ailia LLM uses a proprietary implementation developed by ailia Inc. that is separate from llama.cpp.

Using the NPU runs an LLM quickly and with low power consumption. In addition to text LLMs and VLMs (Vision Language Models) that take images as input, ailia LLM also supports audio input, with processing on the NPU. Gemma 4 E2B and E4B also run on mid-range devices such as the Snapdragon 7s.

In our measurements, prefill is up to 4.2x faster on Android devices, with CPU use during prefill kept below one thirtieth of the CPU path (Snapdragon 8+ Gen 1, 2,048-token input, compared with the CPU on the same device).

QNN (Qualcomm AI Engine Direct) is a backend API provided by Qualcomm. ailia LLM converts GGUF, the model format of llama.cpp, into an SoC-specific QNN Model (.qnn) on a PC, and runs it on the NPU with ailia LLM on the device. The GGUF file is not needed on the device; only the QNN Model is deployed.

Supported Platforms

• Android (Hexagon v69 or later, e.g. Snapdragon 8 Gen 1 / 8+ Gen 1, 7s Gen 3, 8 Gen 2 or later)
• Windows on Arm (Snapdragon X Plus / sc8380xp, Hexagon v73)
• QCS6490 (Windows, Hexagon v68, E2B only; ailia LLM 1.5.0.1 or later)

On Windows on Arm, use the QNN-enabled ailia LLM ARM64 build and a .qnn file that matches the SoC of the device. The E2B text, image, and audio packages have been verified on a Snapdragon X Plus device and a QCS6490 device (Windows 11 IoT Enterprise). Models built for a different SoC cannot be reused.

Supported Models

Gemma 4 E2B and Gemma 4 E4B (both supporting text, image, and audio input) are currently supported. For image and audio input, a QNN Model for the image / audio encoder (mmproj) is used in addition to the text QNN Model. The supported models depend on the Hexagon version of the SoC.

QNN Models are separate files for each model and each Snapdragon model (SoC). Pre-converted model files are provided, so please use the file that matches the SoC of your target device.

Model downloads

Pre-converted QNN Models for Gemma 4 can be downloaded below. To use VLM or audio input, download both the text model and the mmproj file. E4B requires Hexagon v73 or later.

All of these are for ailia LLM 1.5.0 (QAIRT 2.47.0.260601), with a context length of 8192 and the AR-256 / AR-1 configuration.

Project Setup

The following settings are for running precompiled .qnn files in your application. This is not a procedure for compiling models. Use the same QNN runtime as the one used to generate the models, QAIRT 2.47.0.260601.

Windows on Arm: getting the QNN runtime

Get QAIRT 2.47.0.260601 (distribution ZIP) from the Qualcomm AI Engine Direct SDK page. A Qualcomm account may be required for the download. See the QNN setup for ailia SDK for where the files are located in the distribution and how to choose the Hexagon version. For Snapdragon X Plus (Hexagon v73), use the QNN DLLs in lib/aarch64-windows-msvc and the NPU-side .so and .cat files in lib/hexagon-v73/unsigned of the SDK.

With the QNN-enabled Windows ARM64 build, place ailia_llm.dll, ailia_llm_qnn.dll, the shared GGML/llama DLLs, the QNN DLLs from the SDK, and the NPU-side .so and .cat files where your application can find them. The Qualcomm NPU driver is also required. ailia_qnn for ailia SDK and ailia_llm_qnn for ailia LLM are separate plug-ins. See the Windows ARM64 QNN instructions for details on building and deployment.

Android: integrating into an app

Place libailia_llm.so and libailia_llm_qnn.so from the distribution, the bundled libggml*.so / libllama*.so and CPU variants, and the QNN Stub / Skel for the target Hexagon generation in app/src/main/jniLibs/arm64-v8a/ of your Android project. When libailia_llm.so is loaded, the QNN plug-in is lazily loaded when needed. Check the redistribution terms of the SDK.

Add the FastRPC declaration and native library extraction to <application> in app/src/main/AndroidManifest.xml. The Skel libraries must be visible as regular files.

<application android:extractNativeLibs="true" ...>
    <uses-native-library android:name="libcdsprpc.so" android:required="false" />
</application>

Add the QNN runtime of the same generation to the dependencies in app/build.gradle.

dependencies {
    implementation 'com.qualcomm.qti:qnn-runtime:2.47.0'
}

A path inside assets cannot be passed for a .qnn file. Copy or download it to the app's private storage on first launch, and pass that real file path to the APIs below.

Evaluation app

The evaluation APK for Android is ailia-models-kotlin-qnn.apk.

The evaluation app for Windows on Arm is ailia-models-flutter-qnn-v261008.zip. It runs on Snapdragon X Plus and QCS6490.

Code Samples: Getting the SoC Name and Opening the Model

First get the SoC name (e.g. sm7635), then choose a text model compiled for the same SoC, such as gemma4-e2b-<soc>.qnn. The returned string is the SoC name, not the whole file name. n_ctx=0 selects the context length embedded in the package. Do not reuse a file built for a different SoC on an SoC for which no model is published. For image and audio input, open the text model first, then add the -mmproj.qnn of the same model and SoC with the regular projector API.

The following are Android examples for the text model. The model file is assumed to be in the private storage described above. The QNN-enabled build for Windows on Arm uses the same API; replace the file path with the location on Windows. Without the QNN plug-in the result is UNIMPLEMENTED, and on an unsupported SoC it is OTHER_ERROR.

C

#include "ailia_llm.h"
#include <stdio.h>

/* model_dir: the app's private storage. On success the caller destroys the instance. */
int open_qnn_model(const char *model_dir, struct AILIALLM **out) {
    if (model_dir == NULL || out == NULL) return AILIA_LLM_STATUS_INVALID_ARGUMENT;
    *out = NULL;
    const char *soc = NULL; /* owned by the library; do not free */
    int status = ailiaLLMGetQNNModelName(&soc);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    char path[512];
    int n = snprintf(path, sizeof(path), "%s/gemma4-e2b-%s.qnn", model_dir, soc);
    if (n < 0 || (size_t)n >= sizeof(path)) return AILIA_LLM_STATUS_INVALID_ARGUMENT;
    struct AILIALLM *llm = NULL;
    status = ailiaLLMCreate(&llm);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    status = ailiaLLMOpenModelFileA(llm, path, 0);
    if (status != AILIA_LLM_STATUS_SUCCESS) { ailiaLLMDestroy(llm); return status; }
    *out = llm;
    return AILIA_LLM_STATUS_SUCCESS;
}

Python

from pathlib import Path
import ailia_llm

soc = ailia_llm.AiliaLLM.get_qnn_model_name()
model_path = Path("/data/user/0/your.package/files") / f"gemma4-e2b-{soc}.qnn"
llm = ailia_llm.AiliaLLM()
llm.open(str(model_path), n_ctx=0)

Flutter

import 'package:ailia_llm/ailia_llm_model.dart';

final soc = AiliaLLMModel.getQNNModelName();
final model = AiliaLLMModel();
model.open('/data/user/0/your.package/files/gemma4-e2b-$soc.qnn', 0);
// Call model.close() when done

JNI (Kotlin)

context is the Context of an Android Activity or Application.

import axip.ailia_llm.AiliaLLM

val soc = AiliaLLM.getQNNModelName()
val llm = AiliaLLM()
val modelPath = "${context.filesDir}/gemma4-e2b-$soc.qnn"
llm.openModelFile(modelPath, 0)
// Call llm.close() when done

Unity (C#)

using System;
using System.IO;
using UnityEngine;
using ailiaLLM;

string soc = AiliaLLMModel.GetQNNModelName();
var llm = new AiliaLLMModel();
if (!llm.Create()) throw new Exception("ailiaLLMCreate failed");
string modelPath = Path.Combine(Application.persistentDataPath, $"gemma4-e2b-{soc}.qnn");
if (!llm.Open(modelPath, 0)) throw new Exception(llm.GetErrorDetail());
// Call llm.Dispose() when done

Architecture Details

A QNN Model stores a fixed-length prefill graph (AR-N) and a decode graph (AR-1) in the same QNN context, sharing the weights and the KV cache between them. Weights are quantized to 4 bits (W4) at conversion time. In FP16 models, activations and the KV cache are processed in FP16.

ailia LLM Compiler supports both FP16 and Int16, which can be selected at conversion time according to the target SoC. The Snapdragon 7s Gen 3 NPU uses Hexagon v73 but does not support FP16, so ailia LLM uses an Int16-quantized model on this SoC. The Qualcomm Snapdragon 7s Gen 3 product brief (PDF, Artificial Intelligence section on page 2) lists INT4, INT8, and INT16 as supported NPU precisions; FP16 is not included. Select a model based on the precisions supported by the specific SoC, as well as its Hexagon version.

ailia LLM maps the QNN Model with mmap, restores the QNN context, and executes it on the Hexagon NPU (HTP). Prefill is processed with the fixed-length AR-N graph; when the prompt length is not a multiple of AR, the last slice is padded. Decode uses the AR-1 graph in the same context, so no KV cache copy occurs after prefill. For VLM, image patching and position embedding are performed on the CPU, and the 16-layer Vision Transformer, pooling and projection are executed in a single QNN graph. Audio input uses an Audio graph to run the audio encoder on the NPU. The per-layer embedding (PLE) frontend also runs inside each transformer graph, leaving only token-row lookup and media embedding replacement on the CPU. This improves prefill for the Int16 model compared with running that input processing on the CPU.

Benchmark

Decode / prefill throughput, CPU usage during prefill, VLM and audio input response time, and memory usage measured with Gemma 4 E2B / E4B on Snapdragon 8+ Gen 1 (SM8475), Snapdragon 7s Gen 3 (SM7635), Snapdragon X Plus (SC8380XP) and QCS6490 are available in the ailia LLM NPU (QNN) Benchmark.

Limitations

• QNN Models must be converted for each SoC, and a file built for another SoC cannot be used.
• The context length and the fixed prefill length (AR) are determined at conversion time and cannot be changed at runtime. To change them, reconvert the model with ailia LLM Compiler.
• For VLM, the token sequence of one image (256 tokens) must be processed without splitting, so the text model must be converted with AR-256 or larger.
• The currently supported models are Gemma 4 E2B / E4B. On Hexagon v68 / v69 only E2B can be used because of the 2 GB size limit.

Comparison with Other Approaches

The table below compares ailia LLM with the main approaches for running LLMs on Qualcomm SoCs on Android. ailia LLM can use the NPU, runs Gemma 4 on the NPU, and supports relatively old SoCs with Hexagon v69 or later.