ailiaTokenizerAddSpecialTokens method

int ailiaTokenizerAddSpecialTokens(
  1. Pointer<AILIATokenizer> net,
  2. Pointer<Pointer<Char>> tokens,
  3. int count
)

Add SpecialToken

Parameters:

  • net: A tokenizer instance pointer
  • tokens: Token(UTF8)
  • count: The number of tokens

Returns:

If this function is successful, it returns AILIA_STATUS_SUCCESS , or an error code otherwise.

This is valid only for AILIA_TOKENIZER_TYPE_ROBERTA and AILIA_TOKENIZER_TYPE_GPT2.

Implementation

int ailiaTokenizerAddSpecialTokens(
  ffi.Pointer<AILIATokenizer> net,
  ffi.Pointer<ffi.Pointer<ffi.Char>> tokens,
  int count,
) {
  return _ailiaTokenizerAddSpecialTokens(
    net,
    tokens,
    count,
  );
}