Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Creating a dataset

Written by
Improved by
Updated at May 13, 2025

Getting started

To use the examples:

You can start working from the management console right away.

  1. Create a service account and assign the ai.editor role to it.

  2. Get the service account API key and save it.

    The following examples use API key authentication. Yandex Cloud ML SDK also supports IAM token and OAuth token authentication. For more information, see Authentication in Yandex Cloud ML SDK.

  3. Use the pip package manager to install the ML SDK library:

    pip install yandex-cloud-ml-sdk

  1. Get API authentication credentials as described in Authentication with the Yandex Foundation Models API.

  2. To use the examples, install cURL.

  3. Install gRPCurl.

  4. (Optional) Install the jq JSON stream processor.

  5. Get an IAM token used for authentication in the API.

    Note

    The IAM token has a short lifetime: no more than 12 hours.

Prepare a UTF-8-encoded dataset file in JSON Lines format. Dataset contents structure depends on the type of dataset.

Get the JSON schema of the dataset

If you need a JSON schema to prepare your data, you can get the structure of any dataset type via an API request:

grpcurl \
  -H "Authorization: Bearer <IAM_token>" \
  -d '{"task_type": "<dataset_type>"}' \
  llm.api.cloud.yandex.net:443 yandex.cloud.ai.dataset.v1.DatasetService.ListUploadSchemas

Where:

  • <IAM_token>

  • <dataset_type>: Dataset type for which you are requesting the JSON schema. The possible values are:

    • Text generation: TextToTextGenerationRequest and TextToTextGeneration.
    • Multi-label classification: TextClassificationMultilabel.
    • Binary and multi-class classification: TextClassificationMulticlass.
    • Embedding tuning pairs: TextEmbeddingPairParams.
    • Embedding tuning triplets: TextEmbeddingTripletParams.
    • Vision language datasets: ImageTextToTextGenerationRequest and ImageTextToTextGeneration.

Upload the dataset

  1. In the management console, select the folder for which your account has the ai.playground.user and ai.datasets.editor roles or higher.

  2. In the list of services, select Foundation Models.

  3. In the left-hand panel, click Datasets.

  4. Click Create dataset.

  5. Enter a name and descriptions for the dataset. The naming requirements are as follows:

    • It must be from 2 to 63 characters long.
    • It may contain lowercase Latin letters, numbers, and hyphens.
    • It must start with a letter and cannot end with a hyphen.

  6. In the Type field, select dataset type.

  7. Optionally, add or delete dataset labels. They allow you to group resources into logical groups.

  8. Click Select file or drag the JSON file you created earlier to the loading area.

  9. Click Create dataset.

  1. Create a file named dataset-create.py and add the following code to it:

    #!/usr/bin/env python3

from __future__ import annotations

import pathlib
from yandex_cloud_ml_sdk import YCloudML
from yandex_cloud_ml_sdk.auth import YandexCloudCLIAuth


def main() -> None:

    sdk = YCloudML(
        folder_id="<folder_ID>",
        auth="<API_key>",
    )

    # Viewing the list of all uploaded datasets
    for dataset in sdk.datasets.list():
        print(f"List of existing datasets {dataset=}")

    # This way you can delete all previously uploaded datasets,
    # but better not do this unless you have to
    # for dataset in sdk.datasets.list():
    #    dataset.delete()

    # Creating a dataset
    dataset_draft = sdk.datasets.draft_from_path(
        task_type="<dataset_type>",
        path="<file_path>",
        upload_format="jsonlines",
        name="<name>",
    )

    # Waiting for the data to be uploaded and the dataset to be created
    operation = dataset_draft.upload_deferred()
    tuning_dataset = operation.wait()
    print(f"new {tuning_dataset=}")


if __name__ == "__main__":
    main()

    Where:

  2. Run the created file:

    python3 dataset-create.py

    Result:

    new tuning_dataset=Dataset(id='fds6vl5ttl0n********', folder_id='b1gt6g8ht345********', name='YandexGPT 
Lite tuning', description=None, metadata=None, created_by='ajegtlf2q28a********', created_at=datetime.
datetime(2025, 3, 13, 8, 12, 43), updated_at=datetime.datetime(2025, 3, 13, 8, 13, 17), labels=None, 
allow_data_logging=False, status=<DatasetStatus.READY: 3>, task_type='TextClassificationMulticlass', 
rows=4, size_bytes=5679, validation_errors=())

    Save the new dataset's ID (the id field value): you will need it to run the model.

  1. Create a dataset:

    grpcurl \
  -H "Authorization: Bearer <IAM_token>" \
  -d @ \
  llm.api.cloud.yandex.net:443 yandex.cloud.ai.dataset.v1.DatasetService/Create <<EOM
  {
    "folder_id": "<folder_ID>", 
    "name": "My awesome dataset", 
    "task_type": "<dataset_type>", 
    "upload_format": "jsonlines"
  }
EOM

    Where:

    Result:

    {
  "datasetId": "fds8hd01tset********",
  "dataset": {
    "datasetId": "fds8hd01tset********",
    "folderId": "b1gt6g8ht345********",
    "name": "My awesome dataset",
    "status": "DRAFT",
    "taskType": "<issue_type_specified_in_the_query>",
    "createdAt": "2025-01-20T14:51:34Z",
    "updatedAt": "2025-01-20T14:51:34Z",
    "createdById": "ajeg2b2et02f********",
    "createdBy": "ajeg2b2et02f********"
  }
}

    Save the new dataset's ID (the datasetId field value): you will need it to upload data to the dataset.

  2. Get a link to upload data into the dataset:

    grpcurl \
  -H "Authorization: Bearer <IAM_token>" \
  -d '{"dataset_id": "<dataset_ID>", "size_bytes": <dataset_size>}' \
  llm.api.cloud.yandex.net:443 yandex.cloud.ai.dataset.v1.DatasetService/GetUploadDraftUrl | jq

    Where:

    • <IAM_token>: IAM token of the service account you got before you started.
    • <dataset_ID>: Dataset ID you saved in the previous step.
    • <dataset_size>: Size in bytes of the file with data for tuning. In the terminal, you can get the file size using the ls -l <file_path> command.

    Result:

    {
  "datasetId": "fdso08c1u1cq********",
  "uploadUrl": "https://storage.yandexcloud.net/ai-fomo-drafts-prod/b1gt6g8ht345********/fdso08c1u1cq********?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250120T105352Z&X-Amz-SignedHeaders=content-length%3Bhost&X-Amz-Expires=86400&X-Amz-Credential=YCAJE_WuJJ9D1r6huCoc8I3yO%2F20250120%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=611d7951994ae939acf4d32cc0c154c738d02adb2a04707a704f34ca********"
}

    The uploadUrl field of the response contains a link you can use to upload your data into the dataset.

    Tip

    If you did not use jq, replace all \u0026 occurrences with & in the link to use it to upload the dataset.

  3. Upload your data by specifying the link you got in the previous step and the path to the fine-tuning data file:

    curl \
  --request PUT \
  --upload-file <path_to_file> \
  "<link>"

  4. After the data upload is complete, run the dataset validation:

    grpcurl \
  -H "Authorization: Bearer <IAM_token>" \
  -d '{"dataset_id": "<dataset_ID>"}' \
  llm.api.cloud.yandex.net:443 yandex.cloud.ai.dataset.v1.DatasetService/Validate

    Where:

    Result:

    {
  "id": "fdso01v2jdd4********",
  "createdAt": "2025-01-20T11:03:48Z",
  "modifiedAt": "2025-01-20T11:03:48Z"
}

    Save the validation operation ID (id field). You will need it in the next step.

  5. Dataset validation may take some time. To find out validation status and get an error report (if any), send this request:

    grpcurl \
  -H "Authorization: Bearer <IAM_token>" \
  -d '{"operation_id": "<validation_operation_ID>"}' \
  llm.api.cloud.yandex.net:443 yandex.cloud.operation.OperationService/Get

    Where:

    • <IAM_token>: IAM token of the service account you got before you started.
    • <validation_operation_ID>: ID of the validation operation you saved in the previous step.

    Result:

    {
  "id": "fdso01v2jdd4********",
  "createdAt": "2025-01-20T11:03:48Z",
  "modifiedAt": "2025-01-20T11:04:46Z",
  "done": true,
  "response": {
    "@type": "type.googleapis.com/yandex.cloud.ai.dataset.v1.ValidateDatasetResponse",
    "datasetId": "fdso08c1u1cq********",
    "isValid": true
  }
}

    The isValid field is set to true. This means the loaded dataset was validated successfully.

Previous
Using embeddings to search through the knowledge base
Next
Creating a dataset for tuning a text generation model