Foundation Models Text Generation API, gRPC: TokenizerService.Tokenize
Written by
Updated at October 17, 2024
RPC method for tokenizing text.
gRPC request
rpc Tokenize (TokenizeRequest) returns (TokenizeResponse)
TokenizeRequest
{
"modelUri": "string",
"text": "string"
}
Request for the service to tokenize input text.
Field |
Description |
modelUri |
string The identifier of the model to be used for tokenization. |
text |
string Text to be tokenized. |
TokenizeResponse
{
"tokens": [
{
"id": "int64",
"text": "string",
"special": "bool"
}
],
"modelVersion": "string"
}
Response containing tokenized content from request.
Field |
Description |
tokens[] |
A list of tokens obtained from tokenization. |
modelVersion |
string Model version (changes with model releases). |
Token
Represents a token, the basic unit of content, used by the foundation model.
Field |
Description |
id |
int64 An internal token identifier. |
text |
string The textual representation of the token. |
special |
bool Indicates whether the token is special or not. Special tokens may define the model's behavior and are not visible to users. |