Foundation Models Text Generation API, gRPC: TokenizerService.Tokenize

Written by

Updated at November 26, 2024

RPC method for tokenizing text.

gRPC request

{
  "model_uri": "string",
  "text": "string"
}

Request for the service to tokenize input text.

Field

Description

model_uri

string

The identifier of the model to be used for tokenization.

text

string

Text to be tokenized.

{
  "tokens": [
    {
      "id": "int64",
      "text": "string",
      "special": "bool"
    }
  ],
  "model_version": "string"
}

Response containing tokenized content from request.

Field

Description

tokens[]

A list of tokens obtained from tokenization.

model_version

string

Model version (changes with model releases).

Represents a token, the basic unit of content, used by the foundation model.

Field	Description
id	int64 An internal token identifier.
text	string The textual representation of the token.
special	bool Indicates whether the token is special or not. Special tokens may define the model's behavior and are not visible to users.