ailia LLM Multimodal Guide

Overview and flow

In ailia LLM, images and audio can be input by loading a multimodal projector (mmproj), which contains the image and audio encoders, together with the model. The flow is the same in every language.

Use a projector that matches the model. Inputting media before the projector is opened, or a media type it does not support, results in an error.

Input methods

There are two ways to input images and audio.

Either method can be used to input images and audio. To pass raw RGB, input with message structs; to combine media with Tool Use, input with a JSON history. See the Tool Use Guide for details on Tool Use. Image files can be JPG, PNG, TGA, BMP, PSD, GIF, HDR or PIC, and audio files can be WAV, MP3, FLAC and similar formats.

Input with message structs

Write <__media__> in the content of a message, and specify the images and audio to place there in the message's media list, in the same order. The media type is "image" or "audio".

{"role": "user", "content": "Describe this image. <__media__>",
 "media_data": [{"media_type": "image", "file_path": "/path/to/image.jpg"}]}

The example above shows the concept. See the samples below for how to write it in each language.

Input with JSON history

Specify images and audio in order inside the content array of a user message, alongside text entries. Every language passes this JSON (or the equivalent arrays and dictionaries).

[{"role":"user","content":[
  {"type":"text","text":"Describe this image and audio."},
  {"type":"image","file_path":"/path/to/image.jpg"},
  {"type":"audio","file_path":"/path/to/audio.wav"}
]}]

An image or audio entry specifies exactly one of file_path or data (Base64).

API for each language

C++ sample (SetMultimodalPrompt)

Opens the model and the projector, passes an image with message structs, and generates a response. The response is built by concatenating GetDeltaText.

#include "ailia_llm.h"
#include <cstdio>
#include <string>
#include <vector>

static int open_projector(AILIALLM* llm) {
    int status = ailiaLLMOpenMultimodalProjectorFileA(llm, "/path/to/mmproj.gguf");
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    unsigned int vision = 0, audio = 0;
    status = ailiaLLMGetMultimodalCapabilities(llm, &vision, &audio);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    // Check image and audio support with vision_support / audio_support.
    return vision ? AILIA_LLM_STATUS_SUCCESS : AILIA_LLM_STATUS_INVALID_STATE;
}

static int describe_image(AILIALLM* llm) {
    AILIALLMMediaData media = {};
    media.media_type = "image";
    media.file_path = "/path/to/image.jpg"; // a buffer can also be given with data and data_size
    AILIALLMMultimodalChatMessage message = {};
    message.role = "user";
    message.content = "Describe this image. <__media__>";
    message.media_data = &media;
    message.media_count = 1;
    int status = ailiaLLMSetMultimodalPrompt(llm, &message, 1);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    std::string text;
    unsigned int done = 0;
    while (!done) {
        status = ailiaLLMGenerate(llm, &done);
        if (status != AILIA_LLM_STATUS_SUCCESS) return status;
        if (done) break;
        unsigned int size = 0;
        status = ailiaLLMGetDeltaTextSize(llm, &size);
        if (status != AILIA_LLM_STATUS_SUCCESS) return status;
        std::vector<char> delta(size);
        status = ailiaLLMGetDeltaText(llm, delta.data(), size);
        if (status != AILIA_LLM_STATUS_SUCCESS) return status;
        text += delta.data();
    }
    printf("%s\n", text.c_str());
    return AILIA_LLM_STATUS_SUCCESS;
}

int main() {
    AILIALLM* llm = nullptr;
    if (ailiaLLMCreate(&llm) != AILIA_LLM_STATUS_SUCCESS) return 1;
    int status = ailiaLLMOpenModelFileA(llm, "gemma-4-E2B-it-Q4_K_M.gguf", 4096);
    if (status == AILIA_LLM_STATUS_SUCCESS) status = open_projector(llm);
    if (status == AILIA_LLM_STATUS_SUCCESS) status = describe_image(llm);
    if (status != AILIA_LLM_STATUS_SUCCESS) {
        fprintf(stderr, "error %d: %s\n", status, ailiaLLMGetErrorDetail(llm));
    }
    ailiaLLMDestroy(llm);
    return status == AILIA_LLM_STATUS_SUCCESS ? 0 : 1;
}

C++ sample (SetPromptJson)

The same process with a JSON history. The history is built with nlohmann/json.

#include "ailia_llm.h"
#include <nlohmann/json.hpp>
#include <cstdio>
#include <string>
#include <vector>
using json = nlohmann::json;

static int open_projector(AILIALLM* llm) {
    int status = ailiaLLMOpenMultimodalProjectorFileA(llm, "/path/to/mmproj.gguf");
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    unsigned int vision = 0, audio = 0;
    status = ailiaLLMGetMultimodalCapabilities(llm, &vision, &audio);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    // Check image and audio support with vision_support / audio_support.
    return vision ? AILIA_LLM_STATUS_SUCCESS : AILIA_LLM_STATUS_INVALID_STATE;
}

static int describe_image_json(AILIALLM* llm) {
    json messages = json::array({{{"role", "user"}, {"content", json::array({
        {{"type", "text"}, {"text", "Describe this image."}},
        {{"type", "image"}, {"file_path", "/path/to/image.jpg"}}
    })}}});
    std::string input = messages.dump();
    int status = ailiaLLMSetPromptJson(llm, input.c_str());
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    unsigned int done = 0;
    while (!done) {
        status = ailiaLLMGenerate(llm, &done);
        if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    }
    unsigned int size = 0;
    status = ailiaLLMGetResponseJsonSize(llm, &size);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    std::vector<char> output(size);
    status = ailiaLLMGetResponseJson(llm, output.data(), size);
    if (status != AILIA_LLM_STATUS_SUCCESS) return status;
    try {
        json response = json::parse(output.data());
        printf("%s\n", response.at("content").get<std::string>().c_str());
    } catch (const json::exception&) {
        return AILIA_LLM_STATUS_INVALID_ARGUMENT;
    }
    return AILIA_LLM_STATUS_SUCCESS;
}

int main() {
    AILIALLM* llm = nullptr;
    if (ailiaLLMCreate(&llm) != AILIA_LLM_STATUS_SUCCESS) return 1;
    int status = ailiaLLMOpenModelFileA(llm, "gemma-4-E2B-it-Q4_K_M.gguf", 4096);
    if (status == AILIA_LLM_STATUS_SUCCESS) status = open_projector(llm);
    if (status == AILIA_LLM_STATUS_SUCCESS) status = describe_image_json(llm);
    if (status != AILIA_LLM_STATUS_SUCCESS) {
        fprintf(stderr, "error %d: %s\n", status, ailiaLLMGetErrorDetail(llm));
    }
    ailiaLLMDestroy(llm);
    return status == AILIA_LLM_STATUS_SUCCESS ? 0 : 1;
}

The Kotlin, Flutter, Python and Unity samples below input one image file with message structs. The media is placed at the <__media__> position in content. To input audio, set media_type to "audio".

Kotlin sample

Attach AiliaLLMMediaData to an AiliaLLMMultimodalChatMessage, input it with setPrompt, and concatenate getDeltaText() until generate() returns true.

import axip.ailia_llm.AiliaLLM
import axip.ailia_llm.AiliaLLMMediaData
import axip.ailia_llm.AiliaLLMMultimodalChatMessage

fun describeImage(modelPath: String, mmprojPath: String, imagePath: String): String {
    AiliaLLM().use { llm ->
        llm.openModelFile(modelPath, 4096)
        llm.openMultimodalProjectorFile(mmprojPath)
        check(llm.getMultimodalCapabilities().visionSupport) { "Image input is not supported" }
        val message = AiliaLLMMultimodalChatMessage(
            "user",
            "Describe this image. <__media__>",
            AiliaLLMMediaData("image", filePath = imagePath) // use "audio" for audio
        )
        llm.setPrompt(arrayOf(message))
        val text = StringBuilder()
        while (!llm.generate()) {
            text.append(llm.getDeltaText())
        }
        return text.toString()
    }
}

Flutter sample

Add media_data (List<Map<String, dynamic>>) to the message Map, input it with setPrompt, and concatenate the deltas until generate() returns null.

import 'package:ailia_llm/ailia_llm_model.dart';

String describeImage(String modelPath, String mmprojPath, String imagePath) {
  final llm = AiliaLLMModel();
  try {
    llm.open(modelPath, 4096);
    llm.openMultimodalProjectorFile(mmprojPath);
    if (llm.getMultimodalCapabilities()['vision'] != true) {
      throw StateError('Image input is not supported');
    }
    llm.setPrompt([
      {
        'role': 'user',
        'content': 'Describe this image. <__media__>',
        'media_data': <Map<String, dynamic>>[
          {'media_type': 'image', 'file_path': imagePath}, // use 'audio' for audio
        ],
      },
    ]);
    final text = StringBuffer();
    String? delta;
    while ((delta = llm.generate()) != null) {
      text.write(delta);
    }
    return text.toString();
  } finally {
    llm.close();
  }
}

Python sample

Passing message dicts with media_data to generate uses input with message structs. Specify media_data with file_path.

import ailia_llm

model = ailia_llm.AiliaLLM()
model.open("gemma-4-E2B-it-Q4_K_M.gguf", n_ctx=4096)
model.open_multimodal_projector("/path/to/mmproj.gguf")
if not model.get_multimodal_capabilities()["vision"]:
    raise RuntimeError("Image input is not supported")

messages = [{
    "role": "user",
    "content": "Describe this image. <__media__>",
    "media_data": [{"media_type": "image", "file_path": "/path/to/image.jpg"}],  # use "audio" for audio
}]
for delta in model.generate(messages):
    print(delta, end="", flush=True)
print()

Unity sample

Set a list of AiliaLLMMediaData on an AiliaLLMMultimodalChatMessage and input it with SetMultimodalPrompt. It returns false on failure, so get the details with GetErrorDetail().

using System;
using System.Collections.Generic;
using System.Text;
using ailiaLLM;

public static class MultimodalSample
{
    public static string DescribeImage(string modelPath, string mmprojPath, string imagePath)
    {
        var llm = new AiliaLLMModel();
        try
        {
            if (!llm.Create()) throw new Exception("ailiaLLMCreate failed");
            if (!llm.Open(modelPath, 4096)) throw new Exception(llm.GetErrorDetail());
            if (!llm.OpenMultimodalProjector(mmprojPath)) throw new Exception(llm.GetErrorDetail());
            bool vision = false, audio = false;
            if (!llm.GetMultimodalCapabilities(ref vision, ref audio) || !vision)
                throw new Exception("Image input is not supported");
            var messages = new List<AiliaLLMMultimodalChatMessage>
            {
                new AiliaLLMMultimodalChatMessage
                {
                    role = "user",
                    content = "Describe this image. <__media__>",
                    media_data = new List<AiliaLLMMediaData>
                    {
                        new AiliaLLMMediaData { media_type = "image", file_path = imagePath } // use "audio" for audio
                    }
                }
            };
            if (!llm.SetMultimodalPrompt(messages)) throw new Exception(llm.GetErrorDetail());
            var text = new StringBuilder();
            bool done = false;
            while (true)
            {
                if (!llm.Generate(ref done)) throw new Exception(llm.GetErrorDetail());
                if (done) break;
                text.Append(llm.GetDeltaText());
            }
            return text.ToString();
        }
        finally
        {
            llm.Dispose();
        }
    }
}

C API details

Structs, JSON rules and error statuses used in C / C++. In Kotlin, Flutter and Python, errors raise exceptions; in Unity, the methods return false (or an empty string), so get the details with GetErrorDetail().

Projector and support

After opening the model, load the corresponding projector with ailiaLLMOpenMultimodalProjectorFileA / W. Image and audio support can be checked with vision_support / audio_support of ailiaLLMGetMultimodalCapabilities.

ailiaLLMSetMultimodalPrompt (message structs)

Include <__media__> in the content of an AILIALLMMultimodalChatMessage, set the corresponding media in order in media_data, and set their number in media_count. The content of the messages is copied internally, so it can be freed after the call. The response is built by concatenating ailiaLLMGetDeltaText.

Set media_type of AILIALLMMediaData to "image" or "audio". Media is given in one of the following ways.

• File: file_path (UTF-8). Ignored when data is set.
• Encoded buffer: set data and data_size, with width / height set to 0.
• Raw RGB: set width * height * 3 bytes of RGB data in data, and set width / height. Images only.

ailiaLLMSetPromptJson (JSON history)

An image or audio entry must specify exactly one of file_path or data. data is the file content encoded in standard Base64 (with padding). Fetching from a URL, video, and raw RGB/PCM are not supported. A history that contains images or audio is re-evaluated every time. The response is retrieved with ailiaLLMGetResponseJson.

Errors

• Specifying images or audio with a model or projector without media support, or before the projector is loaded, is INVALID_STATE.
• Calling ailiaLLMSetMultimodalPrompt while tools are set is INVALID_STATE.
• A file read failure is ERROR_FILE_API, and a failure to load a buffer (including media decoded from Base64) is ERROR_BUFFER_API.
• Invalid Base64 passed to ailiaLLMSetPromptJson is INVALID_ARGUMENT.