Estimating prompt size in tokens
Neural networks work with texts by representing words and sentences as tokens.
Foundation Models uses its own tokenizer for text processing. To calculate the token size of a text or prompt to a YandexGPT model, use the Tokenize method of the text generation API or Yandex Cloud ML SDK.
The token count of the same text may vary from one model to the next.
Getting started
To use the examples:
-
Create a service account and assign the
ai.languageModels.user
role to it. -
Get the service account API key and save it.
The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.
-
Use the pip
package manager to install the ML SDK library:pip install yandex-cloud-ml-sdk
Get API authentication credentials as described in Authentication with the Yandex Foundation Models API.
To use the examples, install cURL
Calculating prompt size
The example below estimates the size of a prompt to a YandexGPT model.
-
Create a file named
token.py
and paste the following code into it:#!/usr/bin/env python3 from __future__ import annotations from yandex_cloud_ml_sdk import YCloudML messages = "Generative models are managed using prompts. A good prompt should contain the context of your request to the model (instruction) and the actual task the model should complete based on the provided context. The more specific your prompt, the more accurate will be the results returned by the model." def main(): sdk = YCloudML( folder_id="<folder_ID>", auth="<API_key>", ) model = sdk.models.completions("yandexgpt") result = model.tokenize(messages) for token in result: print(token) if __name__ == "__main__": main()
Where:
Note
As input data for a request, Yandex Cloud ML SDK can accept a string, a dictionary, an object of the
TextMessage
class, or an array containing any combination of these data types. For more information, see Yandex Cloud ML SDK usage.messages
: Message text.
-
<folder_ID>
: ID of the folder in which the service account was created. -
<API_key>
: Service account API key you got earlier required for authentication in the API.The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.
model
: Model version value. To learn more, see Accessing models.
-
Run the created file:
python3 token.py
The result of a prompt is a list of tokens obtained by the tokenizer.
Result
{"tokens": [{"id":"1","text":"\u003cs\u003e","special":true}, {"id":"6010","text":"▁Gener","special":false}, {"id":"1748","text":"ative","special":false}, {"id":"7789","text":"▁models","special":false}, {"id":"642","text":"▁are","special":false}, {"id":"15994","text":"▁managed","special":false}, {"id":"1772","text":"▁using","special":false}, {"id":"80536","text":"▁prompts","special":false}, {"id":"125820","text":".","special":false}, {"id":"379","text":"▁A","special":false}, {"id":"1967","text":"▁good","special":false}, {"id":"19099","text":"▁prompt","special":false}, {"id":"1696","text":"▁should","special":false}, {"id":"11195","text":"▁contain","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"7210","text":"▁context","special":false}, {"id":"346","text":"▁of","special":false}, {"id":"736","text":"▁your","special":false}, {"id":"4104","text":"▁request","special":false}, {"id":"342","text":"▁to","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"355","text":"▁(","special":false}, {"id":"105793","text":"instruction","special":false}, {"id":"125855","text":")","special":false}, {"id":"353","text":"▁and","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"9944","text":"▁actual","special":false}, {"id":"7430","text":"▁task","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"1696","text":"▁should","special":false}, {"id":"7052","text":"▁complete","special":false}, {"id":"4078","text":"▁based","special":false}, {"id":"447","text":"▁on","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"6645","text":"▁provided","special":false}, {"id":"7210","text":"▁context","special":false}, {"id":"125820","text":".","special":false}, {"id":"671","text":"▁The","special":false}, {"id":"1002","text":"▁more","special":false}, {"id":"4864","text":"▁specific","special":false}, {"id":"736","text":"▁your","special":false}, {"id":"19099","text":"▁prompt","special":false}, {"id":"125827","text":",","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"1002","text":"▁more","special":false}, {"id":"16452","text":"▁accurate","special":false}, {"id":"912","text":"▁will","special":false}, {"id":"460","text":"▁be","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"4168","text":"▁results","special":false}, {"id":"13462","text":"▁returned","special":false}, {"id":"711","text":"▁by","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"125820","text":".","special":false}, {"id":"3","text":"[NL]","special":true}, {"id":"29083","text":"▁Apart","special":false}, {"id":"728","text":"▁from","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"19099","text":"▁prompt","special":false}, {"id":"125827","text":",","special":false}, {"id":"1303","text":"▁other","special":false}, {"id":"4104","text":"▁request","special":false}, {"id":"9513","text":"▁parameters","special":false}, {"id":"912","text":"▁will","special":false}, {"id":"8209","text":"▁impact","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"125886","text":"'","special":false}, {"id":"125811","text":"s","special":false}, {"id":"5925","text":"▁output","special":false}, {"id":"2778","text":"▁too","special":false}, {"id":"125820","text":".","special":false}, {"id":"7597","text":"▁Use","special":false}, {"id":"12469","text":"▁Foundation","special":false}, {"id":"27947","text":"▁Models","special":false}, {"id":"118637","text":"▁Playground","special":false}, {"id":"2871","text":"▁available","special":false}, {"id":"728","text":"▁from","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"7690","text":"▁management","special":false}, {"id":"15302","text":"▁console","special":false}, {"id":"342","text":"▁to","special":false}, {"id":"2217","text":"▁test","special":false}, {"id":"736","text":"▁your","special":false}, {"id":"14379","text":"▁requests","special":false}, {"id":"125820","text":".","special":false}], "modelVersion":"23.10.2024" }
-
Create a file named
tbody.json
with the request parameters:{ "modelUri": "gpt://<folder_ID>/yandexgpt", "text": "Generative models are managed using prompts. A good prompt should contain the context of your request to the model (instruction) and the actual task the model should complete based on the provided context. The more specific your prompt, the more accurate will be the results returned by the model.\n Apart from the prompt, other request parameters will impact the model's output too. Use Foundation Models Playground available from the management console to test your requests." }
Where
<folder_ID>
is the ID of the Yandex Cloud folder for which your account has theai.languageModels.user
role or higher. -
Send a request to the model:
export IAM_TOKEN=<IAM_token> curl --request POST \ --header "Authorization: Bearer ${IAM_TOKEN}" \ --data "@tbody.json" \ "https://llm.api.cloud.yandex.net/foundationModels/v1/tokenize"
Where:
<IAM_token>
: Value of the IAM token you got for your account.tbody.json
: JSON file with request parameters.
The result of a prompt is a list of tokens obtained by the tokenizer.
Result
{"tokens": [{"id":"1","text":"\u003cs\u003e","special":true}, {"id":"6010","text":"▁Gener","special":false}, {"id":"1748","text":"ative","special":false}, {"id":"7789","text":"▁models","special":false}, {"id":"642","text":"▁are","special":false}, {"id":"15994","text":"▁managed","special":false}, {"id":"1772","text":"▁using","special":false}, {"id":"80536","text":"▁prompts","special":false}, {"id":"125820","text":".","special":false}, {"id":"379","text":"▁A","special":false}, {"id":"1967","text":"▁good","special":false}, {"id":"19099","text":"▁prompt","special":false}, {"id":"1696","text":"▁should","special":false}, {"id":"11195","text":"▁contain","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"7210","text":"▁context","special":false}, {"id":"346","text":"▁of","special":false}, {"id":"736","text":"▁your","special":false}, {"id":"4104","text":"▁request","special":false}, {"id":"342","text":"▁to","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"355","text":"▁(","special":false}, {"id":"105793","text":"instruction","special":false}, {"id":"125855","text":")","special":false}, {"id":"353","text":"▁and","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"9944","text":"▁actual","special":false}, {"id":"7430","text":"▁task","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"1696","text":"▁should","special":false}, {"id":"7052","text":"▁complete","special":false}, {"id":"4078","text":"▁based","special":false}, {"id":"447","text":"▁on","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"6645","text":"▁provided","special":false}, {"id":"7210","text":"▁context","special":false}, {"id":"125820","text":".","special":false}, {"id":"671","text":"▁The","special":false}, {"id":"1002","text":"▁more","special":false}, {"id":"4864","text":"▁specific","special":false}, {"id":"736","text":"▁your","special":false}, {"id":"19099","text":"▁prompt","special":false}, {"id":"125827","text":",","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"1002","text":"▁more","special":false}, {"id":"16452","text":"▁accurate","special":false}, {"id":"912","text":"▁will","special":false}, {"id":"460","text":"▁be","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"4168","text":"▁results","special":false}, {"id":"13462","text":"▁returned","special":false}, {"id":"711","text":"▁by","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"125820","text":".","special":false}, {"id":"3","text":"[NL]","special":true}, {"id":"29083","text":"▁Apart","special":false}, {"id":"728","text":"▁from","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"19099","text":"▁prompt","special":false}, {"id":"125827","text":",","special":false}, {"id":"1303","text":"▁other","special":false}, {"id":"4104","text":"▁request","special":false}, {"id":"9513","text":"▁parameters","special":false}, {"id":"912","text":"▁will","special":false}, {"id":"8209","text":"▁impact","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"2718","text":"▁model","special":false}, {"id":"125886","text":"'","special":false}, {"id":"125811","text":"s","special":false}, {"id":"5925","text":"▁output","special":false}, {"id":"2778","text":"▁too","special":false}, {"id":"125820","text":".","special":false}, {"id":"7597","text":"▁Use","special":false}, {"id":"12469","text":"▁Foundation","special":false}, {"id":"27947","text":"▁Models","special":false}, {"id":"118637","text":"▁Playground","special":false}, {"id":"2871","text":"▁available","special":false}, {"id":"728","text":"▁from","special":false}, {"id":"292","text":"▁the","special":false}, {"id":"7690","text":"▁management","special":false}, {"id":"15302","text":"▁console","special":false}, {"id":"342","text":"▁to","special":false}, {"id":"2217","text":"▁test","special":false}, {"id":"736","text":"▁your","special":false}, {"id":"14379","text":"▁requests","special":false}, {"id":"125820","text":".","special":false}], "modelVersion":"23.10.2024" }
See also
- Tokens
- Text generation overview
- Examples of working with ML SDK on GitHub