ailia LLM

Getting Started

Choose your platform and run your first local chat completion.

Install

Install the ailia LLM Python package from PyPI.

pip3 install ailia_llm

Haven't installed Python or git yet? Start with Setting up your Python environment (Windows / Mac / Linux).

View on PyPI

Run a Sample

Download example.py from ailia-models and run it. The script downloads the Gemma 4 E2B GGUF file on first run and streams a chat completion. For image input, audio input, function calling, or MTP (Multi-Token Prediction), fetch example_vlm.py, example_alm.py, example_tool_use.py, or example_mtp.py from the same folder instead.

wget https://raw.githubusercontent.com/ailia-ai/ailia-models/master/large_language_model/gemma4/example.py
python3 example.py

On Windows, use python instead of python3.

example.py example_vlm.py (image input) example_alm.py (audio input) example_tool_use.py (function calling) example_mtp.py (MTP)

Clone Samples

Clone the C++ sample repository and initialize submodules. The ailia-llm-cpp binding is included as a submodule.

git clone https://github.com/ailia-ai/ailia-models-cpp.git
cd ailia-models-cpp
git submodule init
git submodule update

On macOS only, clear the quarantine attribute on the bundled dylibs:

./xattr.sh

Sample Repository gemma4 sample

Build & Run

Fetch the one-month evaluation license, install CMake, build, and run the gemma4 sample. gemma4.sh downloads the GGUF model on first run, then streams the chat response.

# Fetch a one-month evaluation license
cd ailia
python3 download_license.py
cd ..

# macOS
brew install cmake
# Linux: apt install cmake
# Windows: install CMake and Visual Studio

cmake -DWITH_OPENCV=OFF .
cmake --build .
cd large_language_model/gemma4
./gemma4.sh    # use gemma4.bat on Windows

Full C++ Setup Guide

Install via UPM

Open Window > Package Manager in Unity (2021.3.10f1+), click + > Add package from git URL, and enter the binding URL below.

https://github.com/ailia-ai/ailia-llm-unity.git

Unity API Reference

Run a Sample

Clone ailia-models-unity, open it in the Unity Editor (2021.3.10f1+), drop a GGUF file under StreamingAssets/, and play the LargeLanguageModel scene to stream chat tokens.

git clone https://github.com/ailia-ai/ailia-models-unity.git

AiliaLargeLanguageModelSample.cs

Add to pubspec

Add ailia LLM as a git dependency in your Flutter project's pubspec.yaml, then run flutter pub get. Flutter 3.19.6 or later is required. On macOS, set com.apple.security.app-sandbox to false in macos/Runner/Release.entitlements and Debug.entitlements.

dependencies:
  ailia_llm:
    git:
      url: https://github.com/ailia-ai/ailia-llm-flutter.git
      ref: main

Flutter API Reference

Run a Sample

Clone the Flutter sample repository for a ready-to-run chat app.

git clone https://github.com/ailia-ai/ailia-models-flutter.git
cd ailia-models-flutter
flutter pub get
flutter run

Sample Repository

Use the API in Your Project

Minimal examples for streaming a chat completion in your own application.

import ailia_llm

model = ailia_llm.AiliaLLM()
model.open("gemma-4-E2B-it-Q4_K_M.gguf")

messages = [{"role": "user", "content": "あなたの名前は何ですか？"}]
for delta in model.generate(messages):
    print(delta, end="")

#include "ailia_llm.h"

struct AILIALLM *llm = nullptr;
ailiaLLMCreate(&llm);
ailiaLLMOpenModelFileA(llm, "gemma-4-E2B-it-Q4_K_M.gguf", /*ctx_size=*/0);

AILIALLMChatMessage messages[] = {
    {"user", "あなたの名前は何ですか？"},
};
ailiaLLMSetPrompt(llm, messages, 1);

unsigned int done = 0;
while (!done) {
    ailiaLLMGenerate(llm, &done);
    if (done) break;   // when done is set, GetDeltaText returns the previous token

    unsigned int size = 0;
    ailiaLLMGetDeltaTextSize(llm, &size);
    std::vector<char> buf(size);
    ailiaLLMGetDeltaText(llm, buf.data(), size);
    printf("%s", buf.data());
}
ailiaLLMDestroy(llm);

using ailiaLLM;

var llm = new AiliaLLMModel();
llm.Create();
llm.Open("gemma-4-E2B-it-Q4_K_M.gguf");

var messages = new List<AiliaLLMChatMessage> {
    new AiliaLLMChatMessage("user", "あなたの名前は何ですか？"),
};
llm.SetPrompt(messages);

bool done = false;
while (!done) {
    llm.Generate(ref done);
    if (!done) Debug.Log(llm.GetDeltaText());   // when done, GetDeltaText returns the previous token
}
llm.Close();

import 'package:ailia_llm/ailia_llm.dart';

final llm = AiliaLLMModel();
await llm.open('gemma-4-E2B-it-Q4_K_M.gguf');

final messages = [AiliaLLMChatMessage(role: 'user', content: 'あなたの名前は何ですか？')];
await for (final delta in llm.generate(messages)) {
  stdout.write(delta);
}

val llm = AiliaLLM()
llm.openModelFile(modelPath, contextSize)
llm.setSamplingParams(40, 0.9f, 0.4f, 1234)

llm.setPrompt(arrayOf(
    AiliaLLMChatMessage("user", "あなたの名前は何ですか？"),
))

var done = false
while (!done) {
    done = llm.generate()
    if (!done) Log.d("ailia", llm.getDeltaText())   // when done, getDeltaText returns the previous token
}

Advanced Usage

Guides for NPU inference, tool use, and image and audio input.

ailia LLM NPU (QNN) Guide

docs.ailia.ai

ailia LLM NPU (QNN) Benchmark

docs.ailia.ai

ailia LLM MTP (Multi-Token Prediction) Guide

docs.ailia.ai

ailia LLM MTP Benchmark

docs.ailia.ai

ailia LLM Tool Use Guide

docs.ailia.ai

ailia LLM Multimodal Guide

docs.ailia.ai

FAQ

Common questions about ailia LLM.

What model formats are supported?

ailia LLM loads GGUF files, the format used by llama.cpp. You can convert Hugging Face checkpoints to GGUF with the convert_hf_to_gguf.py script bundled with llama.cpp, or download pre-converted GGUF weights from Hugging Face.

Which model architectures are supported?

Llama, Gemma, Mistral, Qwen, Phi, DeepSeek, and other architectures supported by llama.cpp. Compatibility tracks the upstream llama.cpp project.

How do I stream tokens as they are generated?

model.generate(messages) returns an iterator that yields delta strings as the model decodes each token. Iterate over it and append to a buffer (or print directly) for streaming UX.

How much memory do I need?

Memory usage roughly equals the GGUF file size plus the KV cache and intermediate tensors. As a rule of thumb at Q4 quantization: 2B models ≈ 2 GB, 7B ≈ 5 GB, 13B ≈ 9 GB. Use smaller models or higher quantization (Q4 → Q3) on memory-constrained devices.

Does it support GPU acceleration?

Yes. ailia LLM uses Metal on iOS and macOS, and Vulkan on Windows. (Unlike ailia SDK / Speech / Voice, ailia LLM does not require cuDNN.) Inference falls back to CPU when no GPU is available. See the Vulkan Setup Guide for details.

Does it support NPU acceleration?

Yes. On supported Android and Windows on Arm devices, the QNN backend runs inference on the Hexagon NPU. A GGUF file is converted into an SoC-specific QNN Model on a PC and executed on the device. Gemma 4 E2B / E4B support text, image, and audio input, depending on the SoC. See the ailia LLM NPU (QNN) Guide for details.

Does it support VLM / ALM?

Yes. ailia LLM supports VLMs with image input and ALMs with audio input. Load a compatible model and multimodal projector, then pass images or audio with ailiaLLMSetMultimodalPrompt or ailiaLLMSetPromptJson. Check the model's capabilities with ailiaLLMGetMultimodalCapabilities. See the Multimodal Guide for details.

Does it support tool use (function calling)?

Yes. ailia LLM 1.5.0 and later support Gemma 4 function calling. Define tools with ailiaLLMSetTools and pass the JSON history with ailiaLLMSetPromptJson. After generation, retrieve the response containing tool_calls with ailiaLLMGetResponseJson, then return each result as a message with the tool role. See the ailia LLM Tool Use Guide for details.

Getting Started

Install

Run a Sample

Clone Samples

Build & Run

Install via UPM

Run a Sample

Add to pubspec

Run a Sample

System Requirements

Operating Systems

Languages & Compilers

Model Format

Memory Guidance

Use the API in Your Project

Advanced Usage

API Reference by Platform

Python

C++

Unity

Flutter

JNI

FAQ

Materials

Related Articles