ailia LLM MTP (Multi-Token Prediction) Guide

This page describes the models and API needed for MTP as a delta from the ailia LLM NPU (QNN) Guide. NPU setup (deploying the QNN runtime, Android configuration, getting the SoC name) is the same as in the QNN Guide. For measurements, see the ailia LLM MTP Benchmark.

What is MTP?

An LLM normally generates text one token per inference. Because each token depends on all the tokens before it, generating 1,000 tokens means running the large main model 1,000 times in sequence. NPUs are good at computing many rows at once, but when processing one token at a time, weight loading dominates and the NPU's compute cannot be used effectively.

MTP (Multi-Token Prediction) uses a small assistant (draft model) to guess the next few tokens ahead of time, and then has the large main model verify those guesses all at once. In ailia LLM, the assistant proposes 3 candidate tokens, and the main model evaluates 4 rows ("1 committed token + 3 candidates") in a single inference (the AR-4 graph). Candidates that match the main model's prediction are accepted as is, and at the first mismatch the token predicted by the main model is used instead.

Even if every candidate is rejected, the one token predicted by the main model is always committed. The more guesses hit, the fewer times the main model runs and the faster generation becomes. Because the output tokens are determined by the main model's predictions, MTP increases speed without degrading output quality (however, due to numerical differences between AR-4 and AR-1, the token sequence is not guaranteed to be identical to normal generation). For sampling with temperature, candidates are accepted based on the probability ratio between the main model and the assistant (speculative sampling), so the output follows the main model's probability distribution.

Enabling MTP

Switch the main model to the MTP-capable version (-mtp.qnn), deploy the assistant (-mtp-assistant.qnn), and add one line that opens the assistant after opening the main model. No changes are needed to the code that sets prompts and generates.

ailiaLLMOpenModelFileA(llm, "gemma4-e2b-sm7635-int16-mtp.qnn", 0);               /* main model (MTP-capable) */
ailiaLLMOpenMtpModelFileA(llm, "gemma4-e2b-sm7635-int16-mtp-assistant.qnn", 3);  /* added */
/* ailiaLLMSetPrompt / ailiaLLMGenerate stay unchanged */

Differences from the QNN Guide

• ailia LLM 1.5.1 or later is required. Deploy libailia_llm.so and libailia_llm_qnn.so (ailia_llm.dll and ailia_llm_qnn.dll on Windows) from the same version.
• Setting prompts (ailiaLLMSetPrompt / ailiaLLMSetMultimodalPrompt) and calling ailiaLLMGenerate work the same as before. Even with MTP, each ailiaLLMGenerate call returns one token, delivering committed tokens in order from an internal queue.
• For image and audio input, use the mmproj (-mmproj.qnn) for the same SoC, model and precision as the main model.

Model Download

NPU (QNN)

Always use the main model and assistant as a pair for the same SoC, model and precision. All files target ailia LLM 1.5.1 (QAIRT 2.47.0.260601) with a context length of 8192. Both the main model and the assistant use 4-bit weights (W4); FP16 / Int16 refers to the activation precision.

For image and audio input, use the -mmproj.qnn for the same SoC, model and precision as the main model.

If the assistant is not opened, the MTP-capable main model generates one token at a time with AR-1 as before. A single MTP-capable main model file is enough even if you switch MTP on and off.

CPU

On the CPU, use a regular Gemma 4 GGUF for the main model and the MTP GGUF for the assistant.

The assistant must match the main model's vocabulary and hidden size. Pair an E2B main model with the E2B assistant, and an E4B main model with the E4B assistant.

Evaluation App

The MTP-enabled evaluation APK for Android is ailia-models-kotlin-qnn.apk.

The MTP-enabled evaluation app for Windows on Arm is ailia-models-flutter-qnn-v261017.zip. It runs on Snapdragon X Plus and QCS6490.

API for Opening the Draft Model (Assistant)

Open the assistant with ailiaLLMOpenMtpModelFileA / W. As with ailiaLLMOpenModelFileA / W for the main model, choose the variant according to the path string type.

int ailiaLLMOpenMtpModelFileA(struct AILIALLM *llm, const char *path, unsigned int n_draft);
int ailiaLLMOpenMtpModelFileW(struct AILIALLM *llm, const wchar_t *path, unsigned int n_draft);

Call the APIs in the following order.

1. ailiaLLMCreate
2. Open the main model with ailiaLLMOpenModelFileA / W (-mtp.qnn on the NPU)
3. Open the assistant with ailiaLLMOpenMtpModelFileA / W (only once, before setting the first prompt)
4. For image or audio input, open the mmproj with ailiaLLMOpenMultimodalProjectorFileA / W
5. Generate as usual with ailiaLLMSetPrompt / ailiaLLMSetMultimodalPrompt and ailiaLLMGenerate

• On failure, an error code is returned and MTP is not enabled. Normal generation can continue with the same instance.
• Opening an assistant on a main model without MTP support (a conventional .qnn without AR-4) returns an error.
• To remove or replace an enabled assistant, destroy the instance (ailiaLLMDestroy) and create a new one. A second call returns INVALID_STATE.
• NPU MTP reserves 4 tokens of headroom for verification, so if the input exceeds the context length minus 4, setting the prompt returns CONTEXT_FULL.

Code Samples

These examples get the SoC name on the NPU and open the MTP-capable main model and assistant. The model files are assumed to be in the app's private storage. The precision (fp16 / int16) depends on the SoC; choose it according to the table above.

C

#include "ailia_llm.h"
#include <stdio.h>
#include <string.h>

/* model_dir: the app's private storage. On success, the caller must Destroy. */
int open_qnn_mtp_model(const char *model_dir, struct AILIALLM **out) {
    if (model_dir == NULL || out == NULL) return AILIA_LLM_STATUS_INVALID_ARGUMENT;
    *out = NULL;
    const char *soc = NULL;                                     /* owned by the library; do not free */
    int status = ailiaLLMGetQNNModelName(&soc);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    /* sm8475 / sc8380xp use FP16, sm7635 / qcs6490 use Int16 */
    const char *precision =
        (strcmp(soc, "sm8475") == 0 || strcmp(soc, "sc8380xp") == 0) ? "fp16" : "int16";
    char target[512], assistant[512];
    int n1 = snprintf(target, sizeof(target), "%s/gemma4-e2b-%s-%s-mtp.qnn", model_dir, soc, precision);
    int n2 = snprintf(assistant, sizeof(assistant), "%s/gemma4-e2b-%s-%s-mtp-assistant.qnn", model_dir, soc, precision);
    if (n1 < 0 || (size_t)n1 >= sizeof(target) || n2 < 0 || (size_t)n2 >= sizeof(assistant))
        return AILIA_LLM_STATUS_INVALID_ARGUMENT;

    struct AILIALLM *llm = NULL;
    status = ailiaLLMCreate(&llm);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    status = ailiaLLMOpenModelFileA(llm, target, 0);            /* main model */
    if (status == AILIA_LLM_STATUS_SUCCESS)
        status = ailiaLLMOpenMtpModelFileA(llm, assistant, 3);  /* draft model */
    if (status != AILIA_LLM_STATUS_SUCCESS) { ailiaLLMDestroy(llm); return status; }
    *out = llm;
    return AILIA_LLM_STATUS_SUCCESS;
}

On the CPU, specify a GGUF for the main model and the MTP GGUF for the assistant.

ailiaLLMOpenModelFileA(llm, "gemma-4-E2B-it-qat-Q4_0.gguf", 4096);
ailiaLLMOpenMtpModelFileA(llm, "mtp-gemma-4-E2B-it-Q4_0.gguf", 3);

Python

from pathlib import Path
import ailia_llm

soc = ailia_llm.AiliaLLM.get_qnn_model_name()
precision = "fp16" if soc in ("sm8475", "sc8380xp") else "int16"
model_dir = Path("/data/user/0/your.package/files")

llm = ailia_llm.AiliaLLM()
llm.open(str(model_dir / f"gemma4-e2b-{soc}-{precision}-mtp.qnn"), n_ctx=0)
llm.open_mtp_model(str(model_dir / f"gemma4-e2b-{soc}-{precision}-mtp-assistant.qnn"), n_draft=3)

for delta in llm.generate([{"role": "user", "content": "Introduce yourself."}]):
    print(delta, end="", flush=True)

On the CPU, call llm.open_mtp_model("mtp-gemma-4-E2B-it-Q4_0.gguf", n_draft=3) after llm.open("gemma-4-E2B-it-qat-Q4_0.gguf"). On failure, AiliaLLMError is raised.

Flutter

import 'package:ailia_llm/ailia_llm_model.dart';

final soc = AiliaLLMModel.getQNNModelName();
final precision = (soc == 'sm8475' || soc == 'sc8380xp') ? 'fp16' : 'int16';
const dir = '/data/user/0/your.package/files';

final model = AiliaLLMModel();
model.open('$dir/gemma4-e2b-$soc-$precision-mtp.qnn', 0);
model.openMtpModel('$dir/gemma4-e2b-$soc-$precision-mtp-assistant.qnn', 3);
// Call model.close() when done

On the CPU, call model.openMtpModel('mtp-gemma-4-E2B-it-Q4_0.gguf', 3) after model.open('gemma-4-E2B-it-qat-Q4_0.gguf', 4096). On failure, an Exception is thrown; the main model stays open and MTP is not enabled.

JNI (Kotlin)

context is the Android Activity or Application Context.

import axip.ailia_llm.AiliaLLM

val soc = AiliaLLM.getQNNModelName()
val precision = if (soc == "sm8475" || soc == "sc8380xp") "fp16" else "int16"
val dir = context.filesDir

val llm = AiliaLLM()
llm.openModelFile("$dir/gemma4-e2b-$soc-$precision-mtp.qnn", 0)
llm.openMtpModel("$dir/gemma4-e2b-$soc-$precision-mtp-assistant.qnn", 3)
// Call llm.close() when done

On the CPU, call llm.openMtpModel("mtp-gemma-4-E2B-it-Q4_0.gguf", 3) after llm.openModelFile("gemma-4-E2B-it-qat-Q4_0.gguf", 4096). On failure, a RuntimeException is thrown.

Unity (C#)

using System;
using System.IO;
using UnityEngine;
using ailiaLLM;

string soc = AiliaLLMModel.GetQNNModelName();
string precision = (soc == "sm8475" || soc == "sc8380xp") ? "fp16" : "int16";
string dir = Application.persistentDataPath;

var llm = new AiliaLLMModel();
if (!llm.Create()) throw new Exception("ailiaLLMCreate failed");
if (!llm.Open(Path.Combine(dir, $"gemma4-e2b-{soc}-{precision}-mtp.qnn"), 0))
    throw new Exception(llm.GetErrorDetail());
if (!llm.OpenMtpModel(Path.Combine(dir, $"gemma4-e2b-{soc}-{precision}-mtp-assistant.qnn"), 3))
    throw new Exception(llm.GetErrorDetail());
// Call llm.Dispose() when done

Combining with image and audio input

Open the mmproj for the same SoC, model and precision after opening the assistant. Prompts are set in the same way as in the Multimodal Guide.

ailiaLLMOpenModelFileA(llm, "gemma4-e2b-sm7635-int16-mtp.qnn", 0);
ailiaLLMOpenMtpModelFileA(llm, "gemma4-e2b-sm7635-int16-mtp-assistant.qnn", 3);
ailiaLLMOpenMultimodalProjectorFileA(llm, "gemma4-e2b-sm7635-int16-mmproj.qnn");
/* then ailiaLLMSetMultimodalPrompt / ailiaLLMGenerate */

Limitations

• A CPU GGUF assistant cannot be combined with an NPU main model.
• MTP speeds up decode (token generation). Prefill (prompt processing) and TTFT are largely unchanged.
• The speedup depends on the input and model. See the ailia LLM MTP Benchmark for measurements.