ailia Speech Voice Trigger Guide

Overview

When waiting for voice commands or building a voice agent, you do not know when the user will start speaking. The application has to monitor the microphone input for a long time and receive a per-utterance recognition result once the user finishes speaking.

The regular Buffered API returns 1 not only when VAD detects the end of an utterance, but also when 30 seconds worth of audio have accumulated. In a standby use case with long silences, this causes an unnecessary Transcribe every 30 seconds.

The VadBuffered API does not return 1 just because 30 seconds worth of data have accumulated. It returns 1 only when VAD detects the end of an utterance. This lets you wait for voice input beyond 30 seconds and run speech recognition only when an utterance has finished.

Behavior

The VadBuffered API behaves as follows.

• It returns 1 only when speech of speech_sec or longer is followed by silence of no_speech_sec or longer. Accumulating 30 seconds worth of data alone does not make it return 1.
• Silence before the utterance and noise shorter than speech_sec are discarded, so speech after a long silence is recognized from its beginning.
• While speech continues, data is accumulated until a boundary is detected. Since Transcribe processes at most 30 seconds per call, speech longer than 30 seconds is processed by calling Transcribe repeatedly while VadBuffered returns 1.
• After FinalizeInputData is called, it returns 1 even without a boundary. Use this to process the remaining audio when recording stops.
• To limit the accumulated length, check the number of unprocessed frames with ailiaSpeechGetPosition.

The following setup is required.

• A VAD model (Silero VAD) must be opened with OpenVad.
• The detection thresholds (VAD threshold, speech start duration speech_sec, and speech end duration no_speech_sec) must be set in advance with SetSilentThreshold.
• Since recognition waits for an utterance boundary, use AILIA_SPEECH_FLAG_NONE (normal mode) instead of AILIA_SPEECH_FLAG_LIVE. If you need tentative results, you can also use the IntermediateCallback.

Standby Flow

Feed the PCM received from the microphone with PushInputData while checking VadBuffered, and run Transcribe when it returns 1. Since a single utterance may exceed 30 seconds, call Transcribe repeatedly while VadBuffered returns 1, and handle the resulting text as one voice command. After that, you can keep waiting for the next utterance without resetting any state.

To stop recording, call FinalizeInputData and run Transcribe while VadBuffered returns 1 to process the remaining audio. Call ResetTranscribeState before pushing audio again.

Code Samples

Kotlin (JNI)

The voice agent in ailia-models-kotlin detects the end of the user's utterance with the VadBuffered API, receives a per-utterance recognition result, and then proceeds to generate the agent's response. The thresholds below are the same values used by the voice agent.

val speech = AiliaSpeech(
    envId = -1,
    task = AiliaSpeech.AILIA_SPEECH_TASK_TRANSCRIBE,
    flags = AiliaSpeech.AILIA_SPEECH_FLAG_NONE, // VadBuffered requires non-LIVE mode
)
speech.openModel(encoderPath, decoderPath,
    AiliaSpeech.AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_SMALL)
speech.openVad(vadPath, AiliaSpeech.AILIA_SPEECH_VAD_TYPE_SILERO)
// Values used by the voice agent: threshold=0.5, speechSec=0.25, noSpeechSec=1.0
speech.setSilentThreshold(0.5f, 0.25f, 1.0f)

while (listening) {
    // 1. Push mic PCM; silence before the utterance is discarded by the SDK
    while (true) {
        val chunk = micQueue.poll() ?: break
        speech.pushInputData(chunk.pcm, chunk.channels, chunk.samples, sampleRate)
    }

    // 2. Transcribe only when VAD detects the end of an utterance
    val lines = mutableListOf<String>()
    while (speech.getVadBuffered() == 1) {
        speech.transcribe() // processes up to 30 seconds per call
        for (i in 0 until speech.getTextCount()) {
            lines += speech.getText(i).text
        }
    }
    if (lines.isNotEmpty()) {
        onVoiceCommand(lines.joinToString("")) // one utterance = one voice command
    }
}

Flutter

Use isVadBuffered(). When a VAD model is passed to open(), a threshold of 0.5, speech_sec of 1.0 second, and no_speech_sec of 1.0 second are configured. To change the thresholds, call setSilentThreshold() after open().

final speech = AiliaSpeechModel();
speech.create(false, false, envId); // liveTranscribe = false
speech.open(File(encoderPath), File(decoderPath), File(vadPath), "auto",
    AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_SMALL);
// Optional: override the thresholds configured by open()
speech.setSilentThreshold(0.5, 0.25, 1.0);

while (listening) {
  // 1. Push mic PCM; silence before the utterance is discarded by the SDK
  while (micQueue.isNotEmpty) {
    final chunk = micQueue.removeFirst();
    speech.pushInputData(chunk.pcm, sampleRate, chunk.channels);
  }

  // 2. Transcribe only when VAD detects the end of an utterance
  final lines = <String>[];
  while (speech.isVadBuffered()) {
    for (final text in speech.transcribe()) { // processes up to 30 seconds per call
      lines.add(text.text);
    }
  }
  if (lines.isNotEmpty) {
    onVoiceCommand(lines.join()); // one utterance = one voice command
  }
}

C++

ailiaSpeechCreate(&speech, env_id, AILIA_MULTITHREAD_AUTO, memory_mode,
                  AILIA_SPEECH_TASK_TRANSCRIBE, AILIA_SPEECH_FLAG_NONE,
                  ailiaSpeechUtilGetCallback(), AILIA_SPEECH_API_CALLBACK_VERSION);
ailiaSpeechOpenModelFileA(speech, encoder_path, decoder_path,
                          AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_SMALL);
ailiaSpeechOpenVadFileA(speech, "silero_vad.onnx", AILIA_SPEECH_VAD_TYPE_SILERO);
ailiaSpeechSetSilentThreshold(speech, 0.5f, 0.25f, 1.0f);

while (listening) {
    // 1. Push mic PCM; silence before the utterance is discarded by the SDK
    while (mic_queue_pop(&chunk)) {
        ailiaSpeechPushInputData(speech, chunk.pcm, chunk.channels, chunk.samples, sample_rate);
    }

    // 2. Transcribe only when VAD detects the end of an utterance
    std::string lines;
    while (true) {
        unsigned int buffered = 0;
        ailiaSpeechVadBuffered(speech, &buffered);
        if (buffered == 0) break;
        ailiaSpeechTranscribe(speech); // processes up to 30 seconds per call

        unsigned int n;
        ailiaSpeechGetTextCount(speech, &n);
        for (unsigned int i = 0; i < n; i++) {
            AILIASpeechText text;
            ailiaSpeechGetText(speech, &text, AILIA_SPEECH_TEXT_VERSION, i);
            lines += text.text;
        }
    }
    if (!lines.empty()) {
        on_voice_command(lines); // one utterance = one voice command
    }
}

Unity

With SetVadBufferedMode(true), the worker thread of the AiliaSpeechModel class uses the VadBuffered API instead of the Buffered API and transcribes only when VAD detects the end of an utterance. Set it after OpenVad.

While IsProcessing() returns true, the worker thread is still processing the queue up to the VAD boundary. Pushing new PCM during this time starts processing again and may pick up a partial utterance, so keep accumulating the microphone PCM in waveQueue, and once IsProcessing() becomes false, drain the results with GetResults() and then pass the accumulated PCM.

AiliaSpeechModel ailia_speech = new AiliaSpeechModel();
List<float[]> waveQueue = new List<float[]>();

void OnEnable(){
    ailia_speech.Open(encoder_path, decoder_path, env_id, memory_mode,
        AiliaSpeech.AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_SMALL,
        AiliaSpeech.AILIA_SPEECH_TASK_TRANSCRIBE, AiliaSpeech.AILIA_SPEECH_FLAG_NONE, "auto");
    ailia_speech.OpenVad(vad_path, AiliaSpeech.AILIA_SPEECH_VAD_TYPE_SILERO);
    // Transcribe only when VAD detects the end of an utterance
    ailia_speech.SetVadBufferedMode(true);
    ailia_speech.SetSilentThreshold(0.5f, 0.25f, 1.0f);
}

void Update(){
    // Wait until the worker thread has processed the queue up to the VAD boundary
    if (ailia_speech.IsProcessing()){
        return; // keep accumulating mic PCM in waveQueue
    }

    // Confirmed results
    List<string> lines = ailia_speech.GetResults();
    if (lines.Count > 0){
        OnVoiceCommand(string.Join("", lines)); // one utterance = one voice command
    }

    // Push mic PCM; silence before the utterance is discarded by the SDK
    if (waveQueue.Count == 0){
        return;
    }
    float[] waveData = waveQueue.SelectMany(x => x).ToArray();
    waveQueue.Clear();
    ailia_speech.Transcribe(waveData, frequency, channels, false);
}

void OnDisable(){
    ailia_speech.Close();
}

Python

In the ailia_speech package (pip install ailia_speech), passing vad_boundary = True to transcribe_step() makes it use the VadBuffered API internally, so recognition runs only when VAD detects the end of an utterance. Specifying vad_type in initialize_model() downloads the VAD model and sets a threshold of 0.5, speech_sec of 1.0 seconds, and no_speech_sec of 1.0 seconds. Call set_silent_threshold() to change the thresholds.

import ailia_speech

speech = ailia_speech.Whisper(
    env_id = -1,
    task = ailia_speech.AILIA_SPEECH_TASK_TRANSCRIBE,
    flags = ailia_speech.AILIA_SPEECH_FLAG_NONE) # VadBuffered requires non-LIVE mode
speech.initialize_model(model_path = "./models/",
    model_type = ailia_speech.AILIA_SPEECH_MODEL_TYPE_WHISPER_MULTILINGUAL_SMALL,
    vad_type = ailia_speech.AILIA_SPEECH_VAD_TYPE_SILERO)
# Optional: override the thresholds configured by initialize_model()
speech.set_silent_threshold(0.5, 0.25, 1.0)

while listening:
    # 1. Push mic PCM; silence before the utterance is discarded by the SDK
    pcm = mic_queue.get() # np.float32 PCM

    # 2. Transcribe only when VAD detects the end of an utterance
    lines = []
    for text in speech.transcribe_step(pcm, sample_rate,
                                       complete = False, vad_boundary = True):
        lines.append(text["text"]) # processes up to 30 seconds per iteration

    if lines:
        on_voice_command("".join(lines)) # one utterance = one voice command