ailia LLM NPU (QNN) Guide

Overview

ailia LLM 1.5.0 and later support fast inference on the NPU (Hexagon) of Qualcomm SoCs. For the NPU, ailia LLM uses a proprietary implementation developed by ailia Inc. that is separate from llama.cpp.

Using the NPU runs an LLM quickly and with low power consumption. In addition to text LLMs and VLMs (Vision Language Models) that take images as input, ailia LLM also supports audio input, with processing on the NPU. Gemma 4 E2B and E4B also run on mid-range devices such as the Snapdragon 7s.

In our measurements, prefill is up to 4.2x faster on Android devices, with CPU use during prefill kept below one thirtieth of the CPU path (Snapdragon 8+ Gen 1, 2,048-token input, compared with the CPU on the same device).

QNN (Qualcomm AI Engine Direct) is a backend API provided by Qualcomm. ailia LLM achieves NPU inference by converting GGUF, the model format of llama.cpp, into a format that can be executed by QNN.

Comparison with Other Approaches

The table below compares ailia LLM with the main approaches for running LLMs on Qualcomm SoCs on Android. ailia LLM can use the NPU, runs Gemma 4 on the NPU, and supports relatively old SoCs with Hexagon v69 or later.

Supported Platforms

• Android (Hexagon v69 or later, e.g. Snapdragon 8 Gen 1 / 8+ Gen 1, 7s Gen 3, 8 Gen 2 or later)
• Experimental: QCS6490 (Hexagon v68, E2B only)

Workflow

A GGUF file is converted into a QNN Model on a PC, and the converted QNN Model is executed by ailia LLM on the device. The GGUF file is not needed on the device; only the QNN Model is deployed.

A QNN Model stores a fixed-length prefill graph (AR-N) and a decode graph (AR-1) in the same QNN context, sharing the weights and the KV cache between them. Weights are quantized to 4 bits (W4) at conversion time. In FP16 models, activations and the KV cache are processed in FP16.

ailia LLM Compiler supports both FP16 and Int16, which can be selected at conversion time according to the target SoC. The Snapdragon 7s Gen 3 NPU uses Hexagon v73 but does not support FP16, so ailia LLM uses an Int16-quantized model on this SoC. The Qualcomm Snapdragon 7s Gen 3 product brief (PDF, Artificial Intelligence section on page 2) lists INT4, INT8, and INT16 as supported NPU precisions; FP16 is not included. Select a model based on the precisions supported by the specific SoC, as well as its Hexagon version.

Supported Models

Gemma 4 E2B and Gemma 4 E4B (both supporting text, image, and audio input) are currently supported. For image and audio input, a QNN Model for the image / audio encoder (mmproj) is used in addition to the text QNN Model. The supported models depend on the Hexagon version of the SoC.

QNN Models are separate files for each model and each Snapdragon model (SoC). Pre-converted model files are provided, so please use the file that matches the SoC of your target device.

Model downloads

Pre-converted QNN Models for Gemma 4 can be downloaded below. To use VLM or audio input, download both the text model and the mmproj file. E4B requires Hexagon v73 or later.

All of these are for ailia LLM 1.5.0 (QAIRT 2.47.0.260601), with a context length of 8192 and the AR-256 / AR-1 configuration.

Usage

Integrating into an Android app

In addition to libailia_llm.so, bundle libailia_llm_qnn.so in jniLibs/arm64-v8a of your application. The app loads only libailia_llm.so; libailia_llm_qnn.so is lazily loaded as a plug-in only when a QNN Model is opened.

You need to declare the use of libcdsprpc.so in AndroidManifest.xml. Also enable extractNativeLibs, because the NPU-side libraries (libQnnHtpV*Skel.so) must be reachable through a regular filesystem path.

<application android:extractNativeLibs="true" ...>
    <uses-native-library android:name="libcdsprpc.so" android:required="false"/>
</application>

Add the following to build.gradle so that the QNN runtime is automatically bundled into the apk.

implementation 'com.qualcomm.qti:qnn-runtime:2.47.0'

Loading a QNN Model

A QNN Model (.qnn) is passed to the same model loading API as a GGUF file. When the context size is set to 0, the fixed context length embedded in the QNN Model is used. For image or audio input, open the mmproj QNN Model as the projector after opening the text model.

#include "ailia_llm.h"
struct AILIALLM *llm = nullptr;
ailiaLLMCreate(&llm);
ailiaLLMOpenModelFileA(llm, "gemma4-e2b-sm8475.qnn", /*ctx_size=*/0);
ailiaLLMOpenMultimodalProjectorFileA(llm, "gemma4-e2b-sm8475-mmproj.qnn"); // image / audio input only
// The prompt / generate APIs are the same as for GGUF
ailiaLLMDestroy(llm);

In Kotlin (JNI), pass the path of the QNN Model instead of the GGUF path in the same way.

val llm = AiliaLLM()
llm.openModelFile(qnnModelPath, 0)            // n_ctx=0 uses the fixed length in the QNN Model
llm.openMultimodalProjectorFile(mmprojQnnPath) // image / audio input only

Evaluation app

An evaluation apk for Android can be downloaded from the link below. The QNN-enabled build is ailia-models-kotlin-qnn.apk.

https://github.com/ailia-ai/ailia-models-kotlin/releases/tag/v260927

Architecture

ailia LLM maps the QNN Model with mmap, restores the QNN context, and executes it on the Hexagon NPU (HTP). Prefill is processed with the fixed-length AR-N graph; when the prompt length is not a multiple of AR, the last slice is padded. Decode uses the AR-1 graph in the same context, so no KV cache copy occurs after prefill. For VLM, image patching and position embedding are performed on the CPU, and the 16-layer Vision Transformer, pooling and projection are executed in a single QNN graph. Audio input uses an Audio graph to run the audio encoder on the NPU. The per-layer embedding (PLE) frontend also runs inside each transformer graph, leaving only token-row lookup and media embedding replacement on the CPU. This improves prefill for the Int16 model compared with running that input processing on the CPU.

Benchmark

The following results were measured with Gemma 4 on two devices. Snapdragon 8+ Gen 1 (SM8475 / Hexagon v69) uses the E2B FP16 model, and Snapdragon 7s Gen 3 (SM7635 / Hexagon v73) uses the E2B and E4B Int16 models. All were measured with QAIRT 2.47, a context length of 8192 and the AR-256 / AR-1 configuration.

The CPU numbers run the same Q4_0 GGUF through the standard llama.cpp path on the same device (8 decoder workers, context 8192). The shipping configuration is the build with OpenMP disabled, which is the "CPU" column. "CPU (OpenMP)" is a comparison build made with GGML_OPENMP=ON and OMP_NUM_THREADS=2.

Decode throughput

Throughput with 81 input tokens and 1,000 generated tokens.

With E2B on SM7635, NPU decode is 1.10x faster than the CPU (1.53x faster than the OpenMP build). On the other hand, E2B on SM8475 (0.81x) and E4B on SM7635 (0.93x) are slower than the CPU. Decode processes one token at a time, so it is dominated by weight loading and does not benefit much from the parallel compute of the NPU.

Prefill throughput

Throughput (tokens/s) for input lengths from 64 to 2,048 tokens. After a two-token warmup, SetPrompt is called twice without evaluation to clear the KV cache, and exactly one token is generated at each input length.

• Prefill throughput is actual input tokens / time of the first generate call.
• With a 2,048-token input, the NPU is about 4.21x faster than the CPU for E2B on SM8475 (5.87x faster than the OpenMP build), about 2.20x faster for E2B on SM7635 (2.32x), and about 1.29x faster for E4B (1.44x).
• For short inputs such as 64 tokens the NPU is at a disadvantage, because the fixed-length AR-256 graph is mostly padding.

CPU use during prefill

Process CPU time measured around the first generate call of a 2,048-token prefill, expressed as a share of all 8 cores fully occupied (8 cores = 100%). This is the utilization of the application process, not of the whole device.

On the NPU, CPU use during prefill stays around 3%, leaving the CPU available for the rest of the application. Running on the CPU occupies nearly all 8 cores, which raises heat and interferes with other work.

VLM (image input) latency

Medians over the same 10 images at 768x768 with 100 output tokens each. TTFT is the prompt setup including the image encoder plus the first generate call, excluding model opening.

Image TTFT is about 12.7x faster for E2B on SM8475, about 11.4x faster for E2B on SM7635 and about 7.7x faster for E4B. Both the CPU and the NPU consume the same 768x768 pixels and the same 256 image embeddings.

Audio input latency

Medians over 10 FLEURS recordings (2 each in 5 languages, generated to natural EOS with a 256-token cap). TTFT is the prompt setup including the audio encoder plus the first generate call, excluding model opening.

Audio TTFT is about 9.2x faster for E2B on SM8475, about 6.8x faster for E2B on SM7635 and about 7.3x faster for E4B.

Memory usage

E2B: memory usage on SM7635 (Hexagon v73) while running the LLM, VLM and ALM configurations with 8 generated tokens (in MiB, context length 8192). The CPU path uses the Q4_0 GGUF with the BF16 mmproj, and the NPU path uses the Int16 QNN Model. Models are loaded with mmap, so anonymous memory (RssAnon, the working memory actually allocated) and file-backed mappings (RssFile, reusable page cache) are listed separately. Peak is the peak resident size of the process (VmHWM).

• Anonymous memory on the NPU is about a quarter of the CPU path (0.25x for LLM, 0.27x for VLM, 0.23x for ALM), and peak resident size stays at 0.36x to 0.45x.
• The CPU path expands all weights when the model is opened, taking 4,690.4 MiB, while the NPU path opens with 98.4 MiB and loads the compiled context on the first generate call. Between opening the model and the first generation it therefore holds about 1/48 as much.
• Destroying the model returns both paths to about 87 MiB, and calling mallopt(M_PURGE, 0) brings it down to about 23 MiB.
• The NPU values cover what is visible from the process; allocations that stay inside the DSP are not included. With only 8 generated tokens they also exclude the KV cache, which grows with longer inputs.

E4B was measured on the same device as a single LLM run per path with 81 input tokens and 100 output tokens (in MiB). The RssAnon / RssFile split was not captured, so live PSS is listed instead. These conditions differ from the E2B table.

With E4B as well, the NPU peak is about a third of the CPU path. This measurement was taken with the device at Thermal Status 1 and is affected by Android page reclamation.

Limitations

• QNN Models must be converted for each SoC, and a file built for another SoC cannot be used.
• The context length and the fixed prefill length (AR) are determined at conversion time and cannot be changed at runtime. To change them, reconvert the model with ailia LLM Compiler.
• For VLM, the token sequence of one image (256 tokens) must be processed without splitting, so the text model must be converted with AR-256 or larger.
• The currently supported models are Gemma 4 E2B / E4B. On Hexagon v68 / v69 only E2B can be used because of the 2 GB size limit.