ailia_tokenizer/axip.ailia_tokenizer/AiliaTokenizer/AiliaTokenizer

AiliaTokenizer

constructor(tokenizerType: Int = AILIA_TOKENIZER_TYPE_XLM_ROBERTA)

Creates a tokenizer instance of the given type.

Parameters

tokenizerType

Type of the tokenizer, one of the AILIA_TOKENIZER_TYPE_* constants.