YandexGPT API, gRPC: TokenizerService
Статья создана
Обновлена 6 декабря 2023 г.
Service for tokenizing input text.
Call | Description |
---|---|
Tokenize | RPC method for tokenizing input text. |
Calls TokenizerService
Tokenize
RPC method for tokenizing input text.
rpc Tokenize (TokenizeRequest) returns (TokenizeResponse)
TokenizeRequest
Field | Description |
---|---|
model | string The name or identifier of the model to be used for tokenization. Possible values for now: general , general:embedding . The maximum string length in characters is 50. |
text | string The input text to tokenize. |
TokenizeResponse
Field | Description |
---|---|
tokens[] | Token A list of tokens obtained from tokenization. |
Token
Field | Description |
---|---|
id | int64 An internal token identifier. |
text | string The textual representation of the token. |
special | bool Indicates whether the token is special or not. Special tokens define the model's behavior and are not visible to users. |