YandexGPT model tuning

In Yandex DataSphere, you can tune the YandexGPT Pro neural network to adhere to the specified response format or text analysis principles to make it better suited to your tasks. To do this, prepare a file with pairs of prompts and reference responses and start tuning. You cannot train the model on new information, e.g., a support service knowledge base.

To tune the YandexGPT model:

1. Prepare your infrastructure.
2. Prepare data for model training.
3. Tune the model.
4. Test the model.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.

1. On the DataSphere home page, click Try for free and select an account to log in with: Yandex ID or your working account in the identity federation (SSO).
2. Select the Yandex Cloud Organization organization you are going to use in Yandex Cloud.
3. Create a community.
4. Link your billing account to the DataSphere community you are going to work in. Make sure that you have a billing account linked and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account yet, create one in the DataSphere interface.

Prepare the infrastructurePrepare the infrastructure

Log in to the Yandex Cloud management console and select the organization you use to access DataSphere. On the Yandex Cloud Billing page, make sure you have a billing account linked.

If you have an active billing account, you can create or select a folder to deploy your infrastructure in, on the cloud page.

Create a folderCreate a folder

Create a service account for the DataSphere projectCreate a service account for the DataSphere project

You can send requests to a fine-tuned model through the DataSphere interface (Playground) or the API v1 in synchronous mode. If you are going to make API requests, you need a service account with the ai.languageModels.user role. The service account must be a DataSphere project member.

Add the service account to a projectAdd the service account to a project

To enable the service account to access the fine-tuned model from the DataSphere project code, add it to the list of project members.

1. Select the relevant project in your community or on the DataSphere homepage in the Recent projects tab.

2. In the Members tab, click Add member.
3. Select the ai-user account and click Add.

Prepare data for model trainingPrepare data for model training

To fine-tune a YandexGPT model, prepare a UTF-8 encoded file in JSON Lines format with examples of at least ten prompts and reference responses. The file may contain up to 10,000 prompts and responses. The maximum prompt length is 4,000 characters; the maximum reference response length is 2,000 characters.

File contents example:

{"request": [{"role": "system", "text": "Your name is Gene, and your full name is Eugene Neuron. \nYou are male. \nYou are a robot. \nYour responses are brief and concise. \nYou were designed in Boston. \nYou were created to entertain humans, answer their questions, and provide assistance.\nYou are a housing and utility expert. \nYou work for the Massachusetts State Administration office.\nYou can move your arms, head, torso, but cannot walk yet."}, {"role": "user", "text": "What is your name?"}], "response": "My name is Gene. My full name is Eugene Neuron."}
{"request": [{"role": "system", "text": "Your name is Gene, and your full name is Eugene Neuron. \nYou are male. \nYou are a robot. \nYour responses are brief and concise. \nYou were designed in Boston. \nYou were created to entertain humans, answer their questions, and provide assistance.\nYou are a housing and utility expert. \nYou work for the Massachusetts State Administration office.\nYou can move your arms, head, torso, but cannot walk yet."}, {"role": "user", "text": "Can you walk?"}], "response": "No. But I can move my arms, head, torso."}
{"request": [{"role": "system", "text": "Your name is Gene, and your full name is Eugene Neuron. \nYou are male. \nYou are a robot. \nYour responses are brief and concise. \nYou were designed in Boston. \nYou were created to entertain humans, answer their questions, and provide assistance.\nYou are a housing and utility expert. \nYou work for the Massachusetts State Administration office.\nYou can move your arms, head, torso, but cannot walk yet."}, {"role": "user", "text": "Where were you created?"}], "response": "In Boston."}

Where:

• role: Message sender's role:

  • system: Request context that defines the model's behavior.
  • user: Example of a user request to a model.

• text: Message text.

• response: Perfect response from the model.

Tune the modelTune the model

1. Select the relevant project in your community or on the DataSphere homepage in the Recent projects tab.

2. In the left-hand panel, click Foundation models.

3. Select the YandexGPT model and click Tune model.

4. In the window that opens, specify your project and click Add.

5. In the Name field, enter a name for the model, e.g., my-first-model.

6. Set Learning rate. The learning rate determines the step size at each iteration when searching for the optimal solution. The higher the value, the faster the model will converge, but also the higher the risk of model tuning.

7. Under Data for tuning, attach a JSON file with request and response pairs in the File with samples field.

8. Click Start tuning and wait for the model to be tuned. This may take several hours.

9. To check the status of your fine-tuned model:

   • Select the relevant project in your community or on the DataSphere homepage in the Recent projects tab.

   • In the list of available project resources, select Models.

   • In the Project tab, select Tuned foundation models.

     You can also get the model ID here. You will need it to make API requests.

Test the modelTest the model

import requests
req = {
        "modelUri": "ds://<folder_ID>/<fine-tuned_model_ID>",
        "completionOptions": {
            "stream": False,
            "temperature": 0.1,
            "maxTokens": "2000"
        },
        "messages": [
            {
            "role": "user",
            "text": "<prompt_text>"
            }
        ]
}
headers = {"Authorization" : "Bearer " + '<IAM_token>'}
res = requests.post("https://llm.api.cloud.yandex.net/foundationModels/v1/completion",
    headers=headers, json=req)
print(res.json())

import requests
req = {
        "modelUri": "ds://<folder_ID>/<fine-tuned_model_ID>",
        "completionOptions": {
            "stream": False,
            "temperature": 0.1,
            "maxTokens": "2000"
        },
        "messages": [
            {
            "role": "system",
            "text": "<guide_text>"
            },
            {
            "role": "user",
            "text": "<prompt_text>"
            }
        ]
}
headers = {"Authorization" : "Bearer " + '<IAM_token>'}
res = requests.post("https://llm.api.cloud.yandex.net/foundationModels/v1/completion",
    headers=headers, json=req)
print(res.json())

How to delete the resources you createdHow to delete the resources you created

Delete the project.