ailiaTokenizerGetVocab method

int ailiaTokenizerGetVocab(
  1. Pointer<AILIATokenizer> net,
  2. int token,
  3. Pointer<Pointer<Char>> vocab
)

Perform encode

Parameters:

  • net: A tokenizer instance pointer
  • token: Token
  • text: Text of vocab (UTF8)

Returns:

If this function is successful, it returns AILIA_STATUS_SUCCESS , or an error code otherwise.

There is no need to release the vocab. The validity period of the vocab will last until the next time the ailiaTokenizer API is called.

Implementation

int ailiaTokenizerGetVocab(
  ffi.Pointer<AILIATokenizer> net,
  int token,
  ffi.Pointer<ffi.Pointer<ffi.Char>> vocab,
) {
  return _ailiaTokenizerGetVocab(
    net,
    token,
    vocab,
  );
}