YandexGPT API, REST: Tokenizer.tokenize
Статья создана
Обновлена 6 декабря 2023 г.
RPC method for tokenizing input text.
HTTP request
POST https://llm.api.cloud.yandex.net/llm/v1alpha/tokenize
Body parameters
{
"model": "string",
"text": "string"
}
Request to tokenize input text.
Field | Description |
---|---|
model | string The name or identifier of the model to be used for tokenization. Possible values for now: The maximum string length in characters is 50. |
text | string The input text to tokenize. |
Response
HTTP Code: 200 - OK
{
"tokens": [
{
"id": "string",
"text": "string",
"special": true
}
]
}
Tokenization response.
Field | Description |
---|---|
tokens[] | object A list of tokens obtained from tokenization. |
tokens[]. id |
string (int64) An internal token identifier. |
tokens[]. text |
string The textual representation of the token. |
tokens[]. special |
boolean (boolean) Indicates whether the token is special or not. Special tokens define the model's behavior and are not visible to users. |