YandexGPT API, gRPC: TokenizerService.Tokenize

Статья создана

Обновлена 17 октября 2024 г.

RPC method for tokenizing input text.

gRPC request

{
  "model": "string",
  "text": "string"
}

Request to tokenize input text.

Field

Description

model

string

The name or identifier of the model to be used for tokenization.
Possible values for now: general, general:embedding.

text

string

The input text to tokenize.

{
  "tokens": [
    {
      "id": "int64",
      "text": "string",
      "special": "bool"
    }
  ]
}

Tokenization response.

Field

Description

tokens[]

A list of tokens obtained from tokenization.

Represents a token, the basic unit of text, used by the LLM.

Field	Description
id	int64 An internal token identifier.
text	string The textual representation of the token.
special	bool Indicates whether the token is special or not. Special tokens define the model's behavior and are not visible to users.