FTS5Tokenizer

public protocol FTS5Tokenizer

An interface to a custom FTS5 tokenizer.

  • Initializes an FTS5 tokenizer.

    Declaration

    Swift

    init(arguments: [String])

    Parameters

    arguments

    The tokenizer arguments used to create the FTS5 table.

  • Sets the text to be tokenized.

    Declaration

    Swift

    func setText(_ text: String, reason: Database.FTS5TokenizationReason)

    Parameters

    text

    The text to be tokenized.
    reason

    The reason tokenization is being requested.

  • Advances the tokenizer to the next token.

    Declaration

    Swift

    func advance() -> Bool

    Return Value

    true if a token was found, false otherwise

  • Returns the current token.

    Declaration

    Swift

    func currentToken() -> String?

    Return Value

    The current token or nil if none

  • Copies the current token in UTF-8 to the supplied buffer.

    Throws

    An error if buffer has insufficient capacity for the token

    Declaration

    Swift

    func copyCurrentToken(to buffer: UnsafeMutablePointer<UInt8>, capacity: Int) throws -> Int

    Parameters

    buffer

    A buffer to receive the current token encoded in UTF-8
    capacity

    The number of bytes availabe in buffer

    Return Value

    The number of bytes written to buffer