Foundation Models Text Generation API, REST: Tokenizer.Tokenize

Written by

Updated at November 26, 2024

RPC method for tokenizing text.

HTTP request

POST https://llm.api.cloud.yandex.net/foundationModels/v1/tokenize

{
  "modelUri": "string",
  "text": "string"
}

Request for the service to tokenize input text.

Field

Description

modelUri

string

The identifier of the model to be used for tokenization.

text

string

Text to be tokenized.

HTTP Code: 200 - OK

{
  "tokens": [
    {
      "id": "string",
      "text": "string",
      "special": "boolean"
    }
  ],
  "modelVersion": "string"
}

Response containing tokenized content from request.

Field

Description

tokens[]

A list of tokens obtained from tokenization.

modelVersion

string

Model version (changes with model releases).

Represents a token, the basic unit of content, used by the foundation model.

Field	Description
id	string (int64) An internal token identifier.
text	string The textual representation of the token.
special	boolean Indicates whether the token is special or not. Special tokens may define the model's behavior and are not visible to users.