ailiaTokenizerDecodeWithSpecialTokens method

int ailiaTokenizerDecodeWithSpecialTokens(
  1. Pointer<AILIATokenizer> net,
  2. Pointer<Int> tokens,
  3. int token_count
)

Perform decode with special tokens

Parameters:

  • net: A tokenizer instance pointer
  • tokens: Tokens for decode
  • token_count: The number of tokens

Returns:

If this function is successful, it returns AILIA_STATUS_SUCCESS , or an error code otherwise.

Get the decoded result with ailiaTokenizerGetText API. Similarly to skip_special_tokens=False, special tokens will be output.

Implementation

int ailiaTokenizerDecodeWithSpecialTokens(
  ffi.Pointer<AILIATokenizer> net,
  ffi.Pointer<ffi.Int> tokens,
  int token_count,
) {
  return _ailiaTokenizerDecodeWithSpecialTokens(
    net,
    tokens,
    token_count,
  );
}