ailiaTokenizerEncodeWithSpecialTokens method

int ailiaTokenizerEncodeWithSpecialTokens(
  1. Pointer<AILIATokenizer> net,
  2. Pointer<Char> utf8
)

Perform encode with special tokens

Parameters:

  • net: A tokenizer instance pointer
  • text: Text for encode (UTF8)

Returns:

If this function is successful, it returns AILIA_STATUS_SUCCESS , or an error code otherwise.

Get the encoded result with ailiaTokenizerGetTokens API. Similarly to split_special_tokens=False, special tokens will be output.

Implementation

int ailiaTokenizerEncodeWithSpecialTokens(
  ffi.Pointer<AILIATokenizer> net,
  ffi.Pointer<ffi.Char> utf8,
) {
  return _ailiaTokenizerEncodeWithSpecialTokens(
    net,
    utf8,
  );
}