ailia_tokenizer/axip.ailia_tokenizer/AiliaTokenizer/encode

encode

fun encode(text: String): IntArray

Encodes a text into token IDs.

Return

Array of token IDs.

Parameters

text

Text to tokenize (UTF-8).