API Usage

Overview of ailia LLM API

Basic usage

This is an example of using ailia LLM.

#include "ailia_llm.h"
 
int main(int argc, char *argv[]){
    struct AILIALLM* llm;
    unsigned int n_ctx = 8192;
    ailiaLLMCreate(&llm);
    ailiaLLMOpenModelFileA(llm, "../models/gemma-2-2b-it-Q4_K_M.gguf", n_ctx);
 
    std::vector<AILIALLMChatMessage> messages;
    AILIALLMChatMessage message;
 
    message.role = "system";
    message.content = "語尾に「くま」をつけて回答してください。";
    messages.push_back(message);
 
    message.role = "user";
    message.content = "こんにちは。";
    messages.push_back(message);
 
    ailiaLLMSetPrompt(llm, &messages[0], messages.size());
 
    std::string text = "";
    while(true) {
        unsigned int done = 0;
        int status = ailiaLLMGenerate(llm, &done);
        if (done == 1 || status != AILIA_LLM_STATUS_SUCCESS){
            break;
        }
        unsigned int size = 0;
        ailiaLLMGetDeltaTextSize(llm, &size);
        std::vector<char> delta(size);
        ailiaLLMGetDeltaText(llm, &delta[0], delta.size());
        text = text + std::string(&delta[0]);
    }
 
    ailiaLLMDestroy(llm);
 
    return 0;
}

Image input (VLM)

Use a vision-capable text model and its matching projector. Call ailiaLLMCreate, ailiaLLMOpenModelFileA, then ailiaLLMOpenMultimodalProjectorFileA; check vision_support with ailiaLLMGetMultimodalCapabilities. Each <__media__> marker must match one entry in media_data, in order.

/* llm: text model and matching vision-capable projector already opened. */
int set_image_prompt(struct AILIALLM *llm, const char *image_path) {
    unsigned int vision = 0, audio = 0;
    int status = ailiaLLMGetMultimodalCapabilities(llm, &vision, &audio);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    if (!vision || !image_path) return AILIA_LLM_STATUS_INVALID_ARGUMENT;
 
    AILIALLMMediaData media = {0};
    media.media_type = "image";
    media.file_path = image_path; /* UTF-8 local image path */
    AILIALLMMultimodalChatMessage message = {0};
    message.role = "user";
    message.content = "Describe this image: <__media__>";
    message.media_data = &media;
    message.media_count = 1;
    return ailiaLLMSetMultimodalPrompt(llm, &message, 1);
}

Generate text with the same APIs as the basic example. Besides a file path, images accept encoded bytes (data / data_size, width and height zero) or raw RGB bytes (width * height * 3).

Audio input (ALM)

Use an audio-capable text model and its matching mmproj containing an audio encoder. A vision-only projector cannot process audio. Tasks and languages depend on the model. Audio synthesis/output is not supported.

1. Create the session and open the text model with ailiaLLMOpenModelFileA.
2. Open its projector with ailiaLLMOpenMultimodalProjectorFileA.
3. Check that ailiaLLMGetMultimodalCapabilities reports nonzero audio support.
4. Set an audio prompt using the example below.
5. On success, use ailiaLLMGenerate, ailiaLLMGetDeltaTextSize and ailiaLLMGetDeltaText as in the basic example. Check every return value; use ailiaLLMGetErrorDetail to inspect failures.

#include "ailia_llm.h"
 
/* llm: text model and matching audio-capable projector already opened. */
int set_audio_prompt(struct AILIALLM *llm, const char *audio_path) {
    unsigned int vision = 0, audio = 0;
    int status = ailiaLLMGetMultimodalCapabilities(llm, &vision, &audio);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    if (!audio || !audio_path) return AILIA_LLM_STATUS_INVALID_ARGUMENT;
 
    AILIALLMMediaData media = {0};
    media.media_type = "audio";
    media.file_path = audio_path; /* UTF-8 WAV/MP3/FLAC path */
    /* data=NULL, data_size=0, width=0, height=0 */
 
    AILIALLMMultimodalChatMessage message = {0};
    message.role = "user";
    message.content = "Transcribe this audio: <__media__>";
    message.media_data = &media;
    message.media_count = 1;
    return ailiaLLMSetMultimodalPrompt(llm, &message, 1);
}

Input is a WAV/MP3/FLAC file path or an encoded file buffer. For buffer input, set file_path=NULL, data and data_size; data takes precedence over a path. The size must fit an unsigned int. Keep the buffer and strings alive until the call returns. Set both width and height to zero. Raw PCM and live microphone streaming are not supported by this API. Media markers and array entries must match in count and order.

Audio on Android QNN

The same open APIs accept a precompiled text .qnn and an mmproj .qnn containing audio contexts and CPU weights. Source GGUF files are not required on the device. Use the correct SoC target, QNN libraries and matching base/plugin builds. Old vision-only packages do not acquire audio capability automatically.

Feature extraction and subsampling remain on CPU; all 12 Conformer blocks and the final projection execute in one HTP graph. Padding and history-preserving chunking support short/long inputs. Regenerate older linear audio packages with the current compiler. Mixed image/audio in one prompt is unsupported; use separate prompts. Audio prompts do not reuse previous-prompt KV. Audio tokens are causal and may cross AR256 boundaries, but the resulting token sequence must fit the model context. QNN-owned sparse linear weights cannot fall back to CPU.

Android QNN models

With a QNN-enabled build, open a device-specific, precompiled text .qnn using ailiaLLMOpenModelFileA. For VLM/ALM, open its matching projector .qnn using ailiaLLMOpenMultimodalProjectorFileA. Unlike GGUF files, these packages are SoC-specific. Match the text/projector pair, QNN runtime and plug-in versions. Without an explicit backend selection, a .qnn package selects HTP (QNN). Explicit CPU/GPU selection rejects .qnn, and explicit HTP (QNN) rejects GGUF.

const char *stem = NULL;
int status = ailiaLLMGetQNNModelName(&stem); /* e.g. "sm7635" */
if (status != AILIA_LLM_STATUS_SUCCESS) return status;
/* Distribution filenames may also include a model prefix. Select the
   text/projector packages for this stem and the current SoC. */
status = ailiaLLMOpenModelFileA(llm, text_qnn_path, 0);
if (status != AILIA_LLM_STATUS_SUCCESS) return status;
status = ailiaLLMOpenMultimodalProjectorFileA(llm, mmproj_qnn_path);
if (status != AILIA_LLM_STATUS_SUCCESS) return status;

Skip the projector for text-only use. n_ctx=0 selects the package context length. Model-name lookup returns UNIMPLEMENTED for a non-QNN build or absent plug-in, and OTHER_ERROR on unsupported SoCs. Inspect failures with ailiaLLMGetErrorDetail.

Tool Use (JSON API)

Use ailiaLLMSetPromptJson from the first user message while tools are configured. The legacy ailiaLLMSetPrompt and ailiaLLMSetMultimodalPrompt return INVALID_STATE until tools are cleared. Define tools with ailiaLLMSetTools, set the JSON messages array, and call ailiaLLMGenerate until done. Deltas remain available through ailiaLLMGetDeltaText for previews; retrieving or concatenating them is optional. Retrieve assistant JSON with ailiaLLMGetResponseJsonSize / ailiaLLMGetResponseJson, append the object to history, execute complete calls, and append tool results with matching tool_call_id. Set the next JSON prompt to continue.

The SDK accumulates generated output. Reads do not consume it; a new prompt resets it.

For example, set the following tool definition and initial message. The application executes tools, validating the returned function name and arguments.

[{"type":"function","function":{"name":"get_weather","description":"Get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]

[{"role":"user","content":"Weather in Tokyo?"}]

/* llm: model already opened; tools_json/messages_json contain the JSON above. */
int run_tool_turn(struct AILIALLM *llm, const char *tools_json,
                  const char *messages_json, char **response) {
    int status = ailiaLLMSetTools(llm, tools_json);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    status = ailiaLLMSetPromptJson(llm, messages_json);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    unsigned int done = 0;
    while (!done) {
        status = ailiaLLMGenerate(llm, &done);
        if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    }
    unsigned int size = 0;
    status = ailiaLLMGetResponseJsonSize(llm, &size);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    *response = (char *)malloc(size); /* caller frees */
    if (!*response) return AILIA_LLM_STATUS_OTHER_ERROR;
    status = ailiaLLMGetResponseJson(llm, *response, size);
    if (status != AILIA_LLM_STATUS_SUCCESS) { free(*response); *response = NULL; }
    return status;
}

Include <stdlib.h> and pass a valid char **response to this example.

Structured history

[
 {"role":"user","content":"Weather in Tokyo?"},
 {"role":"assistant","content":"","tool_calls":[
  {"id":"call_0","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Tokyo\"}"}}
 ]},
 {"role":"tool","tool_call_id":"call_0","content":"Snow, -3 C"}
]

Content is not reparsed as tool syntax. Supply assistant thinking in reasoning_content. Arguments must be a string containing a JSON object. Call IDs/names must be nonempty; IDs are unique per assistant, but may be reused in later turns. Consecutive tool results match the preceding assistant by ID, not order. Unknown IDs, duplicate results and conflicting names are rejected. Tool content must be a string; serialize objects first.

Images and audio

Load a compatible projector with ailiaLLMOpenMultimodalProjectorFileA and use ordered user content parts:

[{"role":"user","content":[
 {"type":"text","text":"Describe this image and audio."},
 {"type":"image","file_path":"/path/to/image.jpg"},
 {"type":"audio","file_path":"/path/to/audio.wav"}
]}]

Each image/audio part specifies exactly one of file_path or data. Data is standard padded Base64 containing encoded file bytes (JPEG/PNG, WAV/MP3/FLAC, etc.). No URL fetching, video or raw RGB/PCM is supported. Media works with or without tools and is re-evaluated each prompt. An unloaded/incompatible projector returns INVALID_STATE. Query image/audio support with ailiaLLMGetMultimodalCapabilities.

Incomplete output and errors

GetResponseJson rejects unfinished calls/thinking with PARSE_ERROR. It never repairs arguments or returns executable partial results. SetPromptJson rejects incomplete arguments, invalid JSON/types/ID associations and empty arrays with INVALID_ARGUMENT. If GetResponseJson returns PARSE_ERROR, do not append the failed assistant. Append a retry user message to the existing history; no partial content/reasoning retrieval is needed. Do not invent tool results.

After clearing tools with SetTools(NULL), SetPromptJson still accepts historical tool calls and results for summaries or final answers. Retrieve response JSON before clearing tools, which invalidates the output parser. Use finally to clear tools even when an exception occurs. An unset prompt or changed tools/thinking returns INVALID_STATE; before generation an initialized prompt returns an empty assistant. File failures return ERROR_FILE_API, invalid Base64 returns INVALID_ARGUMENT, and undecodable media buffers return ERROR_BUFFER_API.