ailia LLM NPU (QNN) Benchmark

For how to use the QNN backend, see the ailia LLM NPU (QNN) Guide.

Test Conditions

The following results were measured with Gemma 4 on two devices. Snapdragon 8+ Gen 1 (SM8475 / Hexagon v69) uses the E2B FP16 model, and Snapdragon 7s Gen 3 (SM7635 / Hexagon v73) uses the E2B and E4B Int16 models. All were measured with QAIRT 2.47, a context length of 8192 and the AR-256 / AR-1 configuration.

The CPU numbers run the same Q4_0 GGUF through the standard llama.cpp path on the same device (8 decoder workers, context 8192). The shipping configuration is the build with OpenMP disabled, which is the "CPU" column. "CPU (OpenMP)" is a comparison build made with GGML_OPENMP=ON and OMP_NUM_THREADS=2.

Decode throughput

Throughput with 81 input tokens and 1,000 generated tokens.

With E2B on SM7635, NPU decode is 1.10x faster than the CPU (1.53x faster than the OpenMP build). On the other hand, E2B on SM8475 (0.81x) and E4B on SM7635 (0.93x) are slower than the CPU. Decode processes one token at a time, so it is dominated by weight loading and does not benefit much from the parallel compute of the NPU.

Prefill throughput

Throughput (tokens/s) for input lengths from 64 to 2,048 tokens. After a two-token warmup, SetPrompt is called twice without evaluation to clear the KV cache, and exactly one token is generated at each input length.

• Prefill throughput is actual input tokens / time of the first generate call.
• With a 2,048-token input, the NPU is about 4.21x faster than the CPU for E2B on SM8475 (5.87x faster than the OpenMP build), about 2.20x faster for E2B on SM7635 (2.32x), and about 1.29x faster for E4B (1.44x).
• For short inputs such as 64 tokens the NPU is at a disadvantage, because the fixed-length AR-256 graph is mostly padding.

CPU use during prefill

Process CPU time measured around the first generate call of a 2,048-token prefill, expressed as a share of all 8 cores fully occupied (8 cores = 100%). This is the utilization of the application process, not of the whole device.

On the NPU, CPU use during prefill stays around 3%, leaving the CPU available for the rest of the application. Running on the CPU occupies nearly all 8 cores, which raises heat and interferes with other work.

VLM (image input) latency

Medians over the same 10 images at 768x768 with 100 output tokens each. TTFT is the prompt setup including the image encoder plus the first generate call, excluding model opening.

Image TTFT is about 12.7x faster for E2B on SM8475, about 11.4x faster for E2B on SM7635 and about 7.7x faster for E4B. Both the CPU and the NPU consume the same 768x768 pixels and the same 256 image embeddings.

Audio input latency

Medians over 10 FLEURS recordings (2 each in 5 languages, generated to natural EOS with a 256-token cap). TTFT is the prompt setup including the audio encoder plus the first generate call, excluding model opening.

Audio TTFT is about 9.2x faster for E2B on SM8475, about 6.8x faster for E2B on SM7635 and about 7.3x faster for E4B.

Memory usage

E2B: memory usage on SM7635 (Hexagon v73) while running the LLM, VLM and ALM configurations with 8 generated tokens (in MiB, context length 8192). The CPU path uses the Q4_0 GGUF with the BF16 mmproj, and the NPU path uses the Int16 QNN Model. Models are loaded with mmap, so anonymous memory (RssAnon, the working memory actually allocated) and file-backed mappings (RssFile, reusable page cache) are listed separately. Peak is the peak resident size of the process (VmHWM).

• Anonymous memory on the NPU is about a quarter of the CPU path (0.25x for LLM, 0.27x for VLM, 0.23x for ALM), and peak resident size stays at 0.36x to 0.45x.
• The CPU path expands all weights when the model is opened, taking 4,690.4 MiB, while the NPU path opens with 98.4 MiB and loads the compiled context on the first generate call. Between opening the model and the first generation it therefore holds about 1/48 as much.
• Destroying the model returns both paths to about 87 MiB, and calling mallopt(M_PURGE, 0) brings it down to about 23 MiB.
• The NPU values cover what is visible from the process; allocations that stay inside the DSP are not included. With only 8 generated tokens they also exclude the KV cache, which grows with longer inputs.

E4B was measured on the same device as a single LLM run per path with 81 input tokens and 100 output tokens (in MiB). The RssAnon / RssFile split was not captured, so live PSS is listed instead. These conditions differ from the E2B table.

With E4B as well, the NPU peak is about a third of the CPU path. This measurement was taken with the device at Thermal Status 1 and is affected by Android page reclamation.