Contact UsGet started
© 2024 Direct Cursus Technology L.L.C.

Estimating prompt size in tokens

Written by
Updated at December 27, 2024

Neural networks work with texts by representing words and sentences as tokens.

Foundation Models uses its own tokenizer for text processing. To calculate the token size of a text or prompt to a YandexGPT model, use the Tokenize method of the text generation API or Yandex Cloud ML SDK.

The token count of the same text may vary from one model to the next.

Getting started

To use the examples:

  1. Create a service account and assign the ai.languageModels.user role to it.

  2. Get the service account API key and save it.

    The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.

  3. Use the pip package manager to install the ML SDK library:

    pip install yandex-cloud-ml-sdk

Calculating prompt size

The example below estimates the size of a prompt to a YandexGPT model. The prompt context is set in the messages variable.

  1. Create a file named tokenize.py and paste the following code into it:

    #!/usr/bin/env python3

from __future__ import annotations
from yandex_cloud_ml_sdk import YCloudML

messages = "Generative models are managed using prompts. A good prompt should contain the context of your request to the model (instruction) and the actual task the model should complete based on the provided context. The more specific your prompt, the more accurate will be the results returned by the model."

def main():
    sdk = YCloudML(
        folder_id="<folder_ID>",
        auth="<API_key>",
    )

    model = sdk.models.completions("yandexgpt")

    result = model.tokenize(messages)

    for token in result:
        print(token)


if __name__ == "__main__":
    main()

    Where:

    Note

    As input data for a request, Yandex Cloud ML SDK can accept a string, a dictionary, an object of the TextMessage class, or an array containing any combination of these data types. For more information, see Yandex Cloud ML SDK usage.

    • messages: List of messages that set the context for the model:

      • role: Message sender's role:

        • user: Used for sending user messages to the model.
        • system: Used to set the query context and define the model's behavior.
        • assistant: Used for responses generated by the model. In chat mode, the model's responses tagged with the assistant role are included in the message to save the conversation context. Do not send user messages with this role.

      • text: Message text.

  2. Run the created file:

    python3 tokenize.py
    The result of a prompt is a list of tokens obtained by the tokenizer:
    {"tokens":
  [{"id":"1","text":"\u003cs\u003e","special":true},
  {"id":"6010","text":"▁Gener","special":false},
  {"id":"1748","text":"ative","special":false},
  {"id":"7789","text":"▁models","special":false},
  {"id":"642","text":"▁are","special":false},
  {"id":"15994","text":"▁managed","special":false},
  {"id":"1772","text":"▁using","special":false},
  {"id":"80536","text":"▁prompts","special":false},
  {"id":"125820","text":".","special":false},
  {"id":"379","text":"▁A","special":false},
  {"id":"1967","text":"▁good","special":false},
  {"id":"19099","text":"▁prompt","special":false},
  {"id":"1696","text":"▁should","special":false},
  {"id":"11195","text":"▁contain","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"7210","text":"▁context","special":false},
  {"id":"346","text":"▁of","special":false},
  {"id":"736","text":"▁your","special":false},
  {"id":"4104","text":"▁request","special":false},
  {"id":"342","text":"▁to","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"2718","text":"▁model","special":false},
  {"id":"355","text":"▁(","special":false},
  {"id":"105793","text":"instruction","special":false},
  {"id":"125855","text":")","special":false},
  {"id":"353","text":"▁and","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"9944","text":"▁actual","special":false},
  {"id":"7430","text":"▁task","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"2718","text":"▁model","special":false},
  {"id":"1696","text":"▁should","special":false},
  {"id":"7052","text":"▁complete","special":false},
  {"id":"4078","text":"▁based","special":false},
  {"id":"447","text":"▁on","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"6645","text":"▁provided","special":false},
  {"id":"7210","text":"▁context","special":false},
  {"id":"125820","text":".","special":false},
  {"id":"671","text":"▁The","special":false},
  {"id":"1002","text":"▁more","special":false},
  {"id":"4864","text":"▁specific","special":false},
  {"id":"736","text":"▁your","special":false},
  {"id":"19099","text":"▁prompt","special":false},
  {"id":"125827","text":",","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"1002","text":"▁more","special":false},
  {"id":"16452","text":"▁accurate","special":false},
  {"id":"912","text":"▁will","special":false},
  {"id":"460","text":"▁be","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"4168","text":"▁results","special":false},
  {"id":"13462","text":"▁returned","special":false},
  {"id":"711","text":"▁by","special":false},
  {"id":"292","text":"▁the","special":false},
  {"id":"2718","text":"▁model","special":false},
  {"id":"125820","text":".","special":false}],
"modelVersion":"23.10.2024"
}

See also

Previous
Disabling request logging
Next
Sending a request in prompt mode