ailia LLM NPU (QNN) Benchmark

Measured results of running Gemma 4 on the Qualcomm SoC NPU with the ailia LLM 1.5.0 QNN backend, compared with CPU execution on the same device.

For how to use the QNN backend, see the ailia LLM NPU (QNN) Guide.

MTP: This page shows results as of ailia LLM 1.5.0, without MTP. For results with MTP, see the ailia LLM MTP Benchmark.

Android

Test Conditions

Snapdragon 8+ Gen 1 (SM8475 / Hexagon v69)

E2B / FP16

Decode throughput
PathThroughput
CPU9.971 tokens/s
CPU (OpenMP)8.482 tokens/s
NPU (QNN)8.062 tokens/s

NPU decode is 0.81x the CPU.

SM8475 E2B per-token decode throughput over 1000 generated tokens (CPU / CPU OpenMP / NPU)
SM8475 (E2B / FP16) per-token decode throughput over 1000 generated tokens
Prefill throughput
Path6412825651210242048
CPU109.35995.72288.24385.66379.94071.244
CPU (OpenMP)54.18455.33361.47660.31955.50051.064
NPU (FP16)76.244144.406277.775302.667301.791299.827

With a 2,048-token input, the NPU is about 4.21x faster than the CPU (5.87x faster than the OpenMP build).

SM8475 E2B prefill throughput and host CPU use during prefill (CPU / CPU OpenMP / NPU)
SM8475 (E2B / FP16) prefill throughput (left) and host process CPU use during a 2,048-token prefill (right)
VLM (image input) and audio input latency
InputPathTTFT (median)Decode (median)Output tokens
ImageCPU83.63 s12.081 tokens/s100
ImageCPU (OpenMP)128.22 s7.491 tokens/s100
ImageNPU (FP16)6.60 s7.033 tokens/s100
AudioCPU37.29 s10.416 tokens/s21-89
AudioCPU (OpenMP)36.49 s10.539 tokens/s21-89
AudioNPU (FP16)4.06 s6.495 tokens/s21-90

NPU TTFT is about 12.7x faster than the CPU for image input and about 9.2x faster for audio input.

SM8475 E2B VLM and audio TTFT breakdown (CPU / CPU OpenMP / NPU)
SM8475 (E2B / FP16) VLM and audio TTFT breakdown (encoder, and decoder prefill plus the first token)
CPU use during prefill
PathCPU use
CPU98.7%
CPU (OpenMP)86.3%
NPU (QNN)3.0%

Snapdragon 7s Gen 3 (SM7635 / Hexagon v73)

E2B / Int16

Decode throughput
PathThroughput
CPU5.805 tokens/s
CPU (OpenMP)4.195 tokens/s
NPU (QNN)6.410 tokens/s

NPU decode is 1.10x faster than the CPU (1.53x faster than the OpenMP build).

SM7635 E2B per-token decode throughput over 1000 generated tokens (CPU / CPU OpenMP / NPU)
SM7635 (E2B / Int16) per-token decode throughput over 1000 generated tokens
Prefill throughput
Path6412825651210242048
CPU91.55077.06480.49676.18769.75759.192
CPU (OpenMP)27.44149.99962.79370.96569.22856.059
NPU (Int16)33.60166.523130.329133.404131.367130.252

With a 2,048-token input, the NPU is about 2.20x faster than the CPU (2.32x faster than the OpenMP build).

SM7635 E2B prefill throughput and host CPU use during prefill (CPU / CPU OpenMP / NPU)
SM7635 (E2B / Int16) prefill throughput (left) and host process CPU use during a 2,048-token prefill (right)
VLM (image input) and audio input latency
InputPathTTFT (median)Decode (median)Output tokens
ImageCPU95.01 s5.891 tokens/s100
ImageCPU (OpenMP)97.45 s3.569 tokens/s100
ImageNPU (Int16)8.35 s6.444 tokens/s100
AudioCPU47.22 s5.166 tokens/s21-89
AudioCPU (OpenMP)35.37 s3.846 tokens/s21-89
AudioNPU (Int16)6.98 s6.356 tokens/s22-60

NPU TTFT is about 11.4x faster than the CPU for image input and about 6.8x faster for audio input.

SM7635 E2B VLM and audio TTFT breakdown (CPU / CPU OpenMP / NPU)
SM7635 (E2B / Int16) VLM and audio TTFT breakdown (encoder, and decoder prefill plus the first token)
CPU use during prefill
PathCPU use
CPU96.4%
CPU (OpenMP)97.3%
NPU (QNN)2.4%
Memory usage

Memory usage while running the LLM, VLM and ALM configurations with 8 generated tokens (in MiB, context length 8192). The CPU path uses the Q4_0 GGUF with the BF16 mmproj, and the NPU path uses the Int16 QNN Model. Models are loaded with mmap, so anonymous memory (RssAnon, the working memory actually allocated) and file-backed mappings (RssFile, reusable page cache) are listed separately. Peak is the peak resident size of the process (VmHWM).

WorkloadPathVmHWMRssAnonRssFile
LLMCPU4574.21524.12896.5
LLMNPU (Int16)1666.8386.41276.2
VLMCPU4881.22512.440.3
VLMNPU (Int16)2161.2673.31483.7
ALMCPU4873.72623.01509.1
ALMNPU (Int16)2202.5591.61606.7

E4B / Int16

Decode throughput
PathThroughput
CPU3.351 tokens/s
CPU (OpenMP)3.203 tokens/s
NPU (QNN)3.128 tokens/s

NPU decode is 0.93x the CPU.

SM7635 E4B per-token decode throughput over 1000 generated tokens (CPU / CPU OpenMP / NPU)
SM7635 (E4B / Int16) per-token decode throughput over 1000 generated tokens
Prefill throughput
Path6412825651210242048
CPU1.69729.10528.41762.66364.38360.897
CPU (OpenMP)6.74627.82440.09761.87766.98054.372
NPU (Int16)20.30440.89080.70083.75883.31478.258

With a 2,048-token input, the NPU is about 1.29x faster than the CPU (1.44x faster than the OpenMP build).

SM7635 E4B prefill throughput and host CPU use during prefill (CPU / CPU OpenMP / NPU)
SM7635 (E4B / Int16) prefill throughput (left) and host process CPU use during a 2,048-token prefill (right)
VLM (image input) and audio input latency
InputPathTTFT (median)Decode (median)Output tokens
ImageCPU115.03 s3.332 tokens/s100
ImageCPU (OpenMP)96.90 s2.642 tokens/s100
ImageNPU (Int16)15.01 s3.098 tokens/s100
AudioCPU89.91 s3.053 tokens/s21-91
AudioCPU (OpenMP)45.99 s2.947 tokens/s21-91
AudioNPU (Int16)12.37 s3.018 tokens/s21-91

NPU TTFT is about 7.7x faster than the CPU for image input and about 7.3x faster for audio input.

SM7635 E4B VLM and audio TTFT breakdown (CPU / CPU OpenMP / NPU)
SM7635 (E4B / Int16) VLM and audio TTFT breakdown (encoder, and decoder prefill plus the first token)
CPU use during prefill
PathCPU use
CPU97.5%
CPU (OpenMP)96.7%
NPU (QNN)2.6%
Memory usage

A reference measurement with a single LLM run per path, 81 input tokens and 100 output tokens (in MiB). The RssAnon / RssFile split was not captured, so live PSS is listed instead. These conditions differ from the E2B table.

WorkloadPathVmHWMLive PSS
LLMCPU5569.84432.7
LLMCPU (OpenMP)5518.23922.2
LLMNPU (Int16)1976.01801.8

With E4B as well, the NPU peak is about a third of the CPU path. This measurement was taken with the device at Thermal Status 1 and is affected by Android page reclamation.

Windows

Test Conditions

Snapdragon X Plus (SC8380XP / Hexagon v73)

E2B / FP16

Decode throughput
PathThroughput
CPU32.505 tokens/s
GPU12.590 tokens/s
NPU (QNN)7.988 tokens/s

NPU decode is 0.25x the CPU.

Snapdragon X Plus E2B per-token decode throughput over 1000 generated tokens (CPU / GPU / NPU)
SC8380XP (E2B / FP16) per-token decode throughput over 1000 generated tokens
Prefill throughput
Path6412825651210242048
CPU249.314205.403214.986194.638176.262155.627
GPU152.814184.829201.471190.109168.583149.012
NPU (FP16)116.418229.981443.136467.341461.637460.136

With a 2,048-token input, the NPU is about 2.96x faster than the CPU.

Snapdragon X Plus E2B prefill throughput and host CPU use during prefill (CPU / GPU / NPU)
SC8380XP (E2B / FP16) prefill throughput (left) and host process CPU use during a 2,048-token prefill (right)
VLM (image input) and audio input latency
InputPathTTFTDecodeOutput tokens
ImageCPU55.48 s37.546 tokens/s100
ImageGPU57.74 s12.770 tokens/s100
ImageNPU (FP16)7.06 s7.961 tokens/s100
AudioCPU28.13 s37.924 tokens/s33
AudioGPU27.86 s13.064 tokens/s33
AudioNPU (FP16)4.55 s8.162 tokens/s33

NPU TTFT is about 7.9x faster than the CPU for image input and about 6.2x faster for audio input.

Snapdragon X Plus E2B VLM and audio TTFT breakdown (CPU / GPU / NPU)
SC8380XP (E2B / FP16) VLM and audio TTFT breakdown (prompt setup and the first generate call)
CPU use during prefill
PathCPU use
CPU49.6%
GPU10.8%
NPU (QNN)2.9%

The CPU runs on the 4 performance cores, so CPU usage stays around 50%.

E4B / FP16

Decode throughput
PathThroughput
CPU15.230 tokens/s
GPU7.161 tokens/s
NPU (QNN)4.651 tokens/s

NPU decode is 0.31x the CPU.

Snapdragon X Plus E4B per-token decode throughput over 1000 generated tokens (CPU / GPU / NPU)
SC8380XP (E4B / FP16) per-token decode throughput over 1000 generated tokens
Prefill throughput
Path6412825651210242048
CPU83.79398.738115.542106.40297.41980.109
GPU80.661102.021111.457110.693101.19593.148
NPU (FP16)60.101119.511254.001262.182257.062258.672

With a 2,048-token input, the NPU is about 3.23x faster than the CPU.

Snapdragon X Plus E4B prefill throughput and host CPU use during prefill (CPU / GPU / NPU)
SC8380XP (E4B / FP16) prefill throughput (left) and host process CPU use during a 2,048-token prefill (right)
VLM (image input) and audio input latency
InputPathTTFTDecodeOutput tokens
ImageCPU54.09 s20.161 tokens/s100
ImageGPU57.98 s7.230 tokens/s100
ImageNPU (FP16)10.22 s4.644 tokens/s100
AudioCPU27.67 s19.444 tokens/s33
AudioGPU33.24 s7.094 tokens/s33
AudioNPU (FP16)7.11 s4.606 tokens/s33

NPU TTFT is about 5.3x faster than the CPU for image input and about 3.9x faster for audio input.

Snapdragon X Plus E4B VLM and audio TTFT breakdown (CPU / GPU / NPU)
SC8380XP (E4B / FP16) VLM and audio TTFT breakdown (prompt setup and the first generate call)
CPU use during prefill
PathCPU use
CPU48.9%
GPU6.9%
NPU (QNN)2.3%

The CPU runs on the 4 performance cores, so CPU usage stays around 50%.

QCS6490 (Hexagon v68)

E2B / Int16

Decode throughput
PathThroughput
CPU7.983 tokens/s
NPU (QNN)4.405 tokens/s

NPU decode is 0.55x the CPU.

QCS6490 E2B per-token decode throughput over 1000 generated tokens (CPU / NPU)
QCS6490 (E2B / Int16) per-token decode throughput over 1000 generated tokens
Prefill throughput
Path6412825651210242048
CPU63.77373.74568.78759.80350.62351.408
NPU (Int16)49.211104.323203.224208.770181.354201.447

With a 2,048-token input, the NPU is about 3.92x faster than the CPU.

QCS6490 E2B prefill throughput and host CPU use during prefill (CPU / NPU)
QCS6490 (E2B / Int16) prefill throughput (left) and host process CPU use during a 2,048-token prefill (right)
VLM (image input) and audio input latency
InputPathTTFTDecodeOutput tokens
ImageCPU73.81 s8.603 tokens/s100
ImageNPU (Int16)9.13 s4.361 tokens/s100
AudioCPU34.85 s9.081 tokens/s33
AudioNPU (Int16)7.08 s4.373 tokens/s33

NPU TTFT is about 8.1x faster than the CPU for image input and about 4.9x faster for audio input.

QCS6490 E2B VLM and audio TTFT breakdown (CPU / NPU)
QCS6490 (E2B / Int16) VLM and audio TTFT breakdown (prompt setup and the first generate call)
CPU use during prefill
PathCPU use
CPU49.2%
NPU (QNN)3.1%

The CPU runs on the 4 performance cores, so CPU usage stays around 50%.