YandexGPT API, REST: Tokenizer.Tokenize

Written by

Updated at October 17, 2024

RPC method for tokenizing input text.

HTTP request

POST https://llm.api.cloud.yandex.net/llm/v1alpha/tokenize

{
  "model": "string",
  "text": "string"
}

Request to tokenize input text.

Field

Description

model

string

The name or identifier of the model to be used for tokenization.
Possible values for now: general, general:embedding.

text

string

The input text to tokenize.

HTTP Code: 200 - OK

{
  "tokens": [
    {
      "id": "string",
      "text": "string",
      "special": "boolean"
    }
  ]
}

Tokenization response.

Field

Description

tokens[]

A list of tokens obtained from tokenization.

Represents a token, the basic unit of text, used by the LLM.

Field	Description
id	string (int64) An internal token identifier.
text	string The textual representation of the token.
special	boolean Indicates whether the token is special or not. Special tokens define the model's behavior and are not visible to users.