ailia_llm package

Classes

class ailia_llm.AiliaLLM

Bases: object

Main class for ailia LLM inference.

Provides text generation using large language models via the ailia LLM native backend. Supports both text-only and multimodal (vision/audio) generation.

Notes

When using QNN, call close() explicitly on the same thread that created the instance, whether it is the main thread or a worker thread. For other backends, explicit cleanup is optional; resources are also released automatically by __del__.

Examples

>>> import ailia_llm
>>> llm = ailia_llm.AiliaLLM()
>>> llm.open("model.gguf")
>>> for delta in llm.generate([{"role": "user", "content": "Hello!"}]):
...     print(delta, end="", flush=True)

__init__()

Create an ailia LLM instance.

Allocates a native LLM context. Call open() to load a model before generating text. When using QNN, call close() on the same thread that creates this instance.

Raises:

AiliaLLMError – If the native context creation fails.

close()

Release the native LLM instance.

Explicit cleanup is only required when using QNN to ensure release on the same thread that created the instance. Call this method on that thread, whether it is the main thread or a worker thread. For other backends, explicit cleanup is optional; resources are also released automatically by __del__.

Repeated calls are safe; the instance cannot be used after closing.

context_full()

Check whether the context window is full.

Returns:

True if the context window was exhausted during the last generate() or generate_multimodal() call.

Return type:

bool

Examples

>>> for delta in llm.generate(messages):
...     print(delta, end="")
>>> if llm.context_full():
...     print("Warning: context window was full.")

generate(prompts, top_k=40, top_p=0.9, temp=0.4, dist=1234)

Generate text from chat-style prompts.

This is a unified generator function that automatically detects whether the prompts contain media_data and routes to the appropriate internal API. It yields text fragments (delta tokens) as they are produced by the model.

Parameters:

• prompts (list[dict]) – Message dictionaries contain "role" and "content" strings. Optional "media_data" is a list of entries with "media_type" ("image" or "audio"), "file_path" or "data", and optional "width"/"height". Audio data contains an encoded WAV/MP3/FLAC file, not raw PCM. Tool results use role "tool" and a string "content" (see set_tools()).

• top_k (int, optional, default=40) – Top-k sampling parameter. Limits the token candidates to the top-k most probable tokens.

• top_p (float, optional, default=0.9) – Top-p (nucleus) sampling parameter. Limits the token candidates to those within the cumulative probability p.

• temp (float, optional, default=0.4) – Sampling temperature. Higher values produce more random output.

• dist (int, optional, default=1234) – Random seed for reproducible generation.

Yields:

str – Delta text fragments as they are generated by the model. Concatenating all yielded strings produces the full response.

Raises:

• AiliaLLMError – If prompt setting or generation fails.

• RuntimeError – If media_data is provided but multimodal projector is not loaded.

Notes

If the context window becomes full during generation, the generator stops early. Use context_full() to check whether the context was exhausted.

For multimodal generation with media_data, you must first call open_multimodal_projector() to load the projector file.

The yielded text is for streaming previews. While tools are configured, this legacy method fails with INVALID_STATE. Use generate_json() and get_response_json(), and keep structured assistant messages in history.

Examples

Text-only generation:

>>> import ailia_llm
>>> llm = ailia_llm.AiliaLLM()
>>> llm.open("model.gguf")
>>> messages = [
...     {"role": "system", "content": "You are a helpful assistant."},
...     {"role": "user", "content": "What is Python?"}
... ]
>>> for delta in llm.generate(messages):
...     print(delta, end="", flush=True)

Multimodal generation with images:

>>> import ailia_llm
>>> llm = ailia_llm.AiliaLLM()
>>> llm.open("model.gguf")
>>> llm.open_multimodal_projector("mmproj-model.gguf")
>>> messages = [
...     {"role": "user", "content": "Describe this image: <__media__>",
...      "media_data": [{"media_type": "image", "file_path": "photo.jpg"}]}
... ]
>>> for delta in llm.generate(messages):
...     print(delta, end="", flush=True)

Tool use (function calling) with Gemma 4:

>>> llm.set_tools([{"type": "function", "function": {
...     "name": "get_weather", "description": "Get the current weather.",
...     "parameters": {"type": "object", "properties": {"city": {"type": "string"}},
...                    "required": ["city"]}}}])
>>> messages = [{"role": "user", "content": "What is the weather in Tokyo?"}]
>>> list(llm.generate_json(messages))
>>> response = llm.get_response_json()
>>> messages.append(response)
>>> for call in response.get("tool_calls", []):
...     messages.append({"role": "tool", "tool_call_id": call["id"], "content": "Sunny, 25C"})
>>> for delta in llm.generate_json(messages):
...     print(delta, end="", flush=True)

generate_json(messages, top_k=40, top_p=0.9, temp=0.4, dist=1234)

Generate from structured JSON-compatible message dictionaries.

Required while tools are set. User content can be an array of text, image or audio parts with file_path or base64 data. Load a compatible projector first for media; submit image and audio in separate prompts. Retrieve history with get_response_json(). Yielded deltas are optional display output; the SDK accumulates them.

generated_token_count()

Get the token count of the most recently generated output.

Returns:

Number of tokens generated in the last generate() or generate_multimodal() call.

Return type:

int

Raises:

AiliaLLMError – If the token count retrieval fails.

Examples

>>> for delta in llm.generate(messages):
...     print(delta, end="")
>>> llm.generated_token_count()
128

static get_backend_list()

Return all available backends and their selection indices.

Returns:

Each item contains backend_name, device_name, and idx. Use backend_name to select a backend type, then pass idx to set_backend() before opening a model.

Return type:

list of dict

get_error_detail()

Return the detail of the last failed native operation.

get_multimodal_capabilities()

Query the multimodal capabilities of the loaded model.

Returns:

A dictionary with the following keys:

• "vision" (bool): Whether the model supports image input.

• "audio" (bool): Whether the model supports audio input.

Return type:

dict

Raises:

AiliaLLMError – If the capability query fails.

Examples

>>> import ailia_llm
>>> llm = ailia_llm.AiliaLLM()
>>> llm.open("model.gguf")
>>> llm.open_multimodal_projector("mmproj-model.gguf")
>>> caps = llm.get_multimodal_capabilities()
>>> caps["vision"]
True

static get_qnn_model_name()

Return the QNN model name for the current device, e.g. "sm7635".

The name is an artifact stem, not a numeric SoC ID. Use it to select a matching precompiled text .qnn and multimodal projector .qnn. No model instance needs to be opened. The returned native string is copied before this method returns; callers must not free it.

Raises:

AiliaLLMError – If QNN is unavailable, the plug-in is absent, or the SoC is unsupported. The native error code is available as code.

get_response_json()

Return the buffered assistant object after generation.

Raises AiliaLLMError for invalid state or incomplete calls/thinking. Does not require collecting the deltas. A new prompt resets the buffer.

open(model_path, n_ctx=0)

Open and load a language model file.

Parameters:

• model_path (str) – Path to a GGUF model or a self-contained, SoC-specific QNN text package (.qnn). A matching QNN projector is opened separately with open_multimodal_projector() for vision or audio.

• n_ctx (int, optional, default=0) – Context window size (number of tokens). When 0, the default context size of the model is used.

Raises:

AiliaLLMError – If the model file cannot be opened or loaded.

Examples

>>> import ailia_llm
>>> llm = ailia_llm.AiliaLLM()
>>> llm.open("path/to/model.gguf")
>>> llm.open("path/to/model.gguf", n_ctx=4096)

open_mtp_model(model_path, n_draft)

Open an MTP assistant and enable draft/target verification decoding.

Use a matching Gemma 4 assistant GGUF, or assistant .qnn for a QNN target. Call once after opening the target and before setting a prompt. Supports text generation with CPU/GPU Gemma 4 GGUF and QNN models. n_draft is required: must be at least 1; QNN supports up to 3 draft tokens.

open_multimodal_projector(mmproj_path)

Open a multimodal projector file for vision or audio support.

Must be called after open() to enable multimodal capabilities such as image or audio understanding.

Parameters:

mmproj_path (str) – Path to the multimodal projector GGUF or self-contained QNN package.

Raises:

AiliaLLMError – If the projector file cannot be opened or loaded.

Examples

>>> import ailia_llm
>>> llm = ailia_llm.AiliaLLM()
>>> llm.open("model.gguf")
>>> llm.open_multimodal_projector("mmproj-model.gguf")

prompt_token_count()

Get the token count of the most recently set prompt.

Returns:

Number of tokens in the prompt that was last passed to generate() or generate_multimodal().

Return type:

int

Raises:

AiliaLLMError – If the token count retrieval fails.

Examples

>>> for delta in llm.generate(messages):
...     print(delta, end="")
>>> llm.prompt_token_count()
42

set_backend(index)

Select a backend before open() (GPU, CPU, or HTP (QNN)).

With no selection, model format selects the backend automatically. Explicit CPU/GPU rejects .qnn and explicit HTP (QNN) rejects GGUF.

set_num_threads(n_threads)

Set shared CPU threads before open; default/0 selects automatically.

Android uses performance (big/prime) cores when topology is available. Applies to target decode/prefill, assistant and VLM/ALM CPU processing.

set_thinking(enable)

Enable or disable thinking (reasoning output) for thinking models.

Parameters:

enable (bool) – True to enable thinking output, False to disable it.

Raises:

AiliaLLMError – If the operation fails.

set_tools(tools)

Set the tool (function) definitions for tool use (function calling).

The tools are rendered into the prompt through the chat template on the next generate() call, the model output is constrained to the tool call syntax, and the raw output can be converted into tool calls with get_response_json(). Available for models whose chat template supports tool calling (e.g. Gemma 4).

While tools are set, the legacy generate() prompt path fails with INVALID_STATE. Use generate_json() from the first turn and get_response_json() afterwards. Deltas are optional streaming previews and need not be concatenated.

Parameters:

tools (list[dict] or None) – OpenAI-compatible tool definitions. None or an empty list clears the tools.

Raises:

AiliaLLMError – If the tool definitions are invalid or the operation fails.

Examples

>>> llm.set_tools([
...     {
...         "type": "function",
...         "function": {
...             "name": "get_weather",
...             "description": "Get the current weather of a city.",
...             "parameters": {
...                 "type": "object",
...                 "properties": {"city": {"type": "string"}},
...                 "required": ["city"],
...             },
...         },
...     }
... ])

token_count(text)

Count the number of tokens in the given text.

Parameters:

text (str) – Input text to tokenize and count.

Returns:

Number of tokens in the text.

Return type:

int

Raises:

AiliaLLMError – If the token counting operation fails.

Examples

>>> import ailia_llm
>>> llm = ailia_llm.AiliaLLM()
>>> llm.open("model.gguf")
>>> llm.token_count("Hello, world!")
4