Foundation Models Text Generation API, REST: Tokenizer.tokenize
Written by
Updated at March 12, 2024
RPC method for tokenizing text.
HTTP request
POST https://llm.api.cloud.yandex.net/foundationModels/v1/tokenize
Body parameters
{
"modelUri": "string",
"text": "string"
}
Request for the service to tokenize input text.
Field | Description |
---|---|
modelUri | string The identifier of the model to be used for tokenization. |
text | string Text to be tokenized. |
Response
HTTP Code: 200 - OK
{
"tokens": [
{
"id": "string",
"text": "string",
"special": true
}
],
"modelVersion": "string"
}
Response containing tokenized content from request.
Field | Description |
---|---|
tokens[] | object A list of tokens obtained from tokenization. |
tokens[]. id |
string (int64) An internal token identifier. |
tokens[]. text |
string The textual representation of the token. |
tokens[]. special |
boolean (boolean) Indicates whether the token is special or not. Special tokens may define the model's behavior and are not visible to users. |
modelVersion | string Model version (changes with model releases). |