Sending a request in prompt mode

To use the examples of requests using SDK:

1. Create a service account and assign the ai.languageModels.user role to it.

2. Get the service account API key and save it.

   The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.

3. Use the pip package manager to install the ML SDK library:

   pip install yandex-cloud-ml-sdk
   

The example below is intended to be run in MacOS and Linux. To run it in Windows, see how to work with Bash in Microsoft Windows.

1. Create a file with the request body, e.g., body.json:

   {
     "modelUri": "gpt://<folder_ID>/yandexgpt",
     "completionOptions": {
       "stream": false,
       "temperature": 0.1,
       "maxTokens": "1000",
       "reasoningOptions": {
         "mode": "DISABLED"
       }
     },
     "messages": [
       {
         "role": "system",
         "text": "Translate the text"
       },
       {
         "role": "user",
         "text": "To be, or not to be: that is the question."
       }
     ]
   }
   

   Where:

   • modelUri: ID of the model that will be used to generate the response. The parameter contains the Yandex Cloud folder ID or the tuned model's ID.

   • completionOptions: Request configuration options:

     • stream: Enables streaming of partially generated text. It can either be true or false.

     • temperature: With a higher temperature, you get more creative and randomized responses from the model. Its values range from 0 to 1, inclusive. The default value is 0.3.

     • maxTokens: Sets a limit on the model's output in tokens. The maximum number of tokens per generation depends on the model. For more information, see Quotas and limits in Yandex Foundation Models.

     • reasoningOptions.mode: Reasoning mode parameters. This is an optional parameter. The default value is DISABLED. The possible values are:

       • DISABLED: Reasoning mode is disabled.
       • ENABLED_HIDDEN: Reasoning mode is enabled. The model will decide by itself whether or not to use this mode for each particular request.

   • messages: List of messages that set the context for the model:

     • role: Message sender's role:

       • user: Used for sending user messages to the model.
       • system: Used to set the query context and define the model's behavior.
       • assistant: Used for responses generated by the model. In chat mode, the model's responses tagged with the assistant role are included in the message to save the conversation context. Do not send user messages with this role.

     • text: Message text.

2. Send a request to the model by running this command:

   export FOLDER_ID=<folder_ID>
   export IAM_TOKEN=<IAM_token>
   curl \
     --request POST \
     --header "Content-Type: application/json" \
     --header "Authorization: Bearer ${IAM_TOKEN}" \
     --header "x-folder-id: ${FOLDER_ID}" \
     --data "@<path_to_JSON_file>" \
     "https://llm.api.cloud.yandex.net/foundationModels/v1/completion"
   

   Where:

   • FOLDER_ID: ID of the folder for which your account has the ai.languageModels.user role or higher.
   • IAM_TOKEN: IAM token you got before you started.
   Result:

   {
     "result": {
       "alternatives": [
         {
           "message": {
             "role": "assistant",
             "text": "To be, or not to be, that is the question."
           },
           "status": "ALTERNATIVE_STATUS_FINAL"
         }
       ],
       "usage": {
         "inputTextTokens": "28",
         "completionTokens": "10",
         "totalTokens": "38"
       },
       "modelVersion": "06.12.2023"
     }
   }
   

1. Create a file named generate-text.py and paste the following code into it:

   #!/usr/bin/env python3
   
   from __future__ import annotations
   from yandex_cloud_ml_sdk import YCloudML
   
   messages = [
       {
           "role": "system",
           "text": "Find errors in the text and correct them",
       },
       {
           "role": "user",
           "text": """Laminate flooring is sutiable for instalation in the kitchen or in a child's 
   room. It withsatnds moisturre and mechanical dammage thanks to 
   a 0.2 mm thick proctive layer of melamine films and 
   a wax-treated interlocking system.""",
       },
   ]
   
   
   def main():
       sdk = YCloudML(
           folder_id="<folder_ID>",
           auth="<API_key>",
       )
   
       result = (
           sdk.models.completions("yandexgpt").configure(temperature=0.5).run(messages)
       )
   
       for alternative in result:
           print(alternative)
   
   
   if __name__ == "__main__":
       main()
   

   Where:

   Note

   As input data for a request, Yandex Cloud ML SDK can accept a string, a dictionary, an object of the TextMessage class, or an array containing any combination of these data types. For more information, see Yandex Cloud ML SDK usage.

   • messages: List of messages that set the context for the model:

     • role: Message sender's role:

       • user: Used for sending user messages to the model.
       • system: Used to set the query context and define the model's behavior.
       • assistant: Used for responses generated by the model. In chat mode, the model's responses tagged with the assistant role are included in the message to save the conversation context. Do not send user messages with this role.

     • text: Message text.

   • <folder_ID>: ID of the folder in which the service account was created.

   • <API_key>: Service account API key you got earlier required for authentication in the API.

     The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.

   For more information about accessing a specific model version, see Accessing models.

2. Run the created file:

   python3 generate-text.py
   

   Result:

   Alternative(role='assistant', text='Laminate flooring is suitable for installation in the kitchen or in a child's room. It withstands moisture and mechanical damage thanks to a 0.2 mm thick protective layer of melamine films and a wax-treated interlocking system.', status=<AlternativeStatus.FINAL: 3>)
   

If the run_stream method is used, the server will provide not just the final text generation result but intermediate results as well. Each intermediate response contains the whole generation result that is currently available. Until the final response is received, the generation results may change as new messages arrive.

The difference the run_stream method makes can be seen most directly when creating and processing large texts.

1. Create a file named generate-text.py and paste the following code into it:

   #!/usr/bin/env python3
   
   from __future__ import annotations
   from yandex_cloud_ml_sdk import YCloudML
   
   messages = [
       {"role": "system", "text": "Find errors in the text and correct them"},
       {"role": "user", "text": "Erors wyll not corrct themselfs."},
   ]
   
   
   def main():
       sdk = YCloudML(
           folder_id="<folder_ID>",
           auth="<API_key>",
       )
   
       model = sdk.models.completions("yandexgpt")
   
       for result in model.configure(temperature=0.5).run_stream(messages):
           for alternative in result:
               print(alternative)
   
   
   if __name__ == "__main__":
       main()
   

   Where:

   • messages: List of messages that set the context for the model:

     • role: Message sender's role:

       • user: Used for sending user messages to the model.
       • system: Used to set the query context and define the model's behavior.
       • assistant: Used for responses generated by the model. In chat mode, the model's responses tagged with the assistant role are included in the message to save the conversation context. Do not send user messages with this role.

     • text: Message text.

   • <folder_ID>: ID of the folder in which the service account was created.

   • <API_key>: Service account API key you got earlier required for authentication in the API.

     The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.

   For more information about accessing a specific model version, see Accessing models.

2. Run the created file:

   python3 generate-text.py
   

   Result:

   Alternative(role='assistant', text='O', status=<AlternativeStatus.PARTIAL: 1>)
   Alternative(role='assistant', text='Errors will not correct themselves.', status=<AlternativeStatus.FINAL: 3>)