ailia_tokenizer/axip.ailia_tokenizer/AiliaTokenizer/Companion

Companion

object Companion

Constants of the tokenizer.

AILIA_TOKENIZER_TYPE_* defines the tokenizer type passed to the constructor, and AILIA_TOKENIZER_FLAG_* defines the tokenizer option.

Properties

Link copied to clipboard
const val AILIA_TOKENIZER_FLAG_NONE: Int = 0

Default flag (no option).

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_BERT: Int = 8

Tokenizer for BERT models. Requires a vocabulary file.

const val AILIA_TOKENIZER_TYPE_BERT_JAPANESE_CHARACTER: Int = 5

Character tokenizer for Japanese BERT models. Requires a MeCab dictionary and a vocabulary file.

const val AILIA_TOKENIZER_TYPE_BERT_JAPANESE_WORDPIECE: Int = 4

WordPiece tokenizer for Japanese BERT models. Requires a MeCab dictionary and a vocabulary file.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_CLIP: Int = 1

Tokenizer for CLIP models.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_GEMMA: Int = 11

Tokenizer for Gemma models. Requires a SentencePiece model file.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_GPT2: Int = 9

Tokenizer for GPT-2 models. Requires a vocabulary file and a merge file.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_LLAMA: Int = 10

Tokenizer for Llama models. Requires a SentencePiece model file.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_MARIAN: Int = 3

Tokenizer for Marian machine translation models. Requires a SentencePiece model file.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_ROBERTA: Int = 7

Tokenizer for RoBERTa models. Requires a vocabulary file and a merge file.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_T5: Int = 6

Tokenizer for T5 models. Requires a SentencePiece model file.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_WHISPER: Int = 0

Tokenizer for Whisper models.

Link copied to clipboard
const val AILIA_TOKENIZER_TYPE_XLM_ROBERTA: Int = 2

Tokenizer for XLM-RoBERTa models. Requires a SentencePiece model file.