ailiaTokenizerGetCharEnds method

int ailiaTokenizerGetCharEnds(
  1. Pointer<AILIATokenizer> net,
  2. Pointer<Int> char_ends,
  3. int count
)

Gets the Char end positions.

Parameters:

  • net: A tokenizer instance pointer
  • char_ends: Char end position
  • count: Token count

Returns:

If this function is successful, it returns AILIA_STATUS_SUCCESS , or an error code otherwise.

If ailiaTokenizerEncode() is not run at all, the function returns AILIA_STATUS_INVALID_STATE . This is valid only for AILIA_TOKENIZER_TYPE_ROBERTA and AILIA_TOKENIZER_TYPE_BERT. The character end positions in UTF-32 units corresponding to each token are returned.

Implementation

int ailiaTokenizerGetCharEnds(
  ffi.Pointer<AILIATokenizer> net,
  ffi.Pointer<ffi.Int> char_ends,
  int count,
) {
  return _ailiaTokenizerGetCharEnds(
    net,
    char_ends,
    count,
  );
}