AiliaTokenizer

class AiliaTokenizer(tokenizerType: Int = AILIA_TOKENIZER_TYPE_XLM_ROBERTA)

Tokenizer for natural language processing models.

This class provides text tokenization (encoding to token IDs) and vocabulary lookup via the ailia Tokenizer native library.

tokenizerType

Type of the tokenizer, one of the AILIA_TOKENIZER_TYPE_* constants.

Constructors

constructor(tokenizerType: Int = AILIA_TOKENIZER_TYPE_XLM_ROBERTA)

Creates a tokenizer instance of the given type.

Constants of the tokenizer.

fun close()

Releases the native resources held by the tokenizer.

Encodes a text into token IDs.

Returns the vocabulary word corresponding to a token ID.

fun loadFiles(modelPath: String? = null, vocabPath: String? = null, mergePath: String? = null, dictionaryPath: String? = null)

Loads the resource files required by the tokenizer.