Foundation Models Text Generation API, REST: Tokenizer.Tokenize
Written by
Updated at October 17, 2024
RPC method for tokenizing text.
HTTP request
POST https://llm.api.cloud.yandex.net/foundationModels/v1/tokenize
Body parameters
{
"modelUri": "string",
"text": "string"
}
Request for the service to tokenize input text.
|
Field |
Description |
|
modelUri |
string The identifier of the model to be used for tokenization. |
|
text |
string Text to be tokenized. |
Response
HTTP Code: 200 - OK
{
"tokens": [
{
"id": "string",
"text": "string",
"special": "boolean"
}
],
"modelVersion": "string"
}
Response containing tokenized content from request.
|
Field |
Description |
|
tokens[] |
A list of tokens obtained from tokenization. |
|
modelVersion |
string Model version (changes with model releases). |
Token
Represents a token, the basic unit of content, used by the foundation model.
|
Field |
Description |
|
id |
string (int64) An internal token identifier. |
|
text |
string The textual representation of the token. |
|
special |
boolean Indicates whether the token is special or not. Special tokens may define the model's behavior and are not visible to users. |