ailia_tokenizer/axip.ailia_tokenizer

Package-level declarations

Types

Link copied to clipboard
class AiliaTokenizer(tokenizerType: Int = AILIA_TOKENIZER_TYPE_XLM_ROBERTA)

Tokenizer for natural language processing models.