Migrating from Yandex Query

Once Yandex Query is discontinued, you will be able to work with your data in one of the following ways:

• Direct reading from Object Storage. Read data from the bucket using S3-compatible utilities, libraries, or your application.
• Yandex Managed Service for YDB. Run federated queries against files in Object Storage using external data sources and external tables. You can write the results to the output bucket.

How to choose your target environmentHow to choose your target environment

This section provides a brief comparison of methods for handling data once Yandex Query is no longer available.

Choose direct reading from Object Storage if:

• Data processing is already implemented in your application.
• Queries are simple and do not require SQL.
• You read files rarely or only once.

Choose federated queries in Yandex Managed Service for YDB if:

• Data must remain in the bucket.
• You want to retain SQL queries and the familiar model of working with external data.
• Same files are not read repeatedly.

Choose column-oriented tables in Yandex Managed Service for YDB if:

• Same data is used in recurring queries or dashboards.
• Speed of repetitive analytical queries is important.
• You are ready to load data from the bucket into the database.

The pricing structure is different for Yandex Query and Yandex Managed Service for YDB. In Yandex Query, you pay for the amount of data read by queries. When reading directly from Object Storage, operations and outbound traffic are charged. In Yandex Managed Service for YDB, dedicated computing resources and database storage are charged independently of the number of requests. For more details on resource costs, see the Object Storage pricing policy and Yandex Managed Service for YDB pricing policy.

Direct reading from Object StorageDirect reading from Object Storage

This scenario is suitable if you perform all required data processing on your end.

Getting startedGetting started

1. Create a service account to perform read operations.

2. Assign the storage.viewer role to the service account for the bucket or the folder containing it.

3. Create a static access key for the service account.

   Save key_id and secret: you will not see the secret key again.

Read the dataRead the data

import boto3
import pandas as pd

s3 = boto3.client(
    "s3",
    endpoint_url="https://storage.yandexcloud.net",
    aws_access_key_id="<key_ID>",
    aws_secret_access_key="<secret_key>",
)

frames = []
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="<bucket_name>", Prefix="<prefix>/"):
    for obj in page.get("Contents", []):
        if obj["Key"].endswith(".csv"):
            body = s3.get_object(Bucket="<bucket_name>", Key=obj["Key"])["Body"]
            frames.append(pd.read_csv(body))

df = pd.concat(frames)
print(df.groupby("service_name")["cost"].sum().sort_values(ascending=False))

Yandex Managed Service for YDBYandex Managed Service for YDB

Entity mappingEntity mapping

Getting startedGetting started

1. Create a Yandex Managed Service for YDB database with dedicated resources in the same cloud where the data buckets are located.

   Note

   External data sources are available only in databases with dedicated resources. In a serverless database, a query to an external source will fail with the External data sources are disabled error.

2. Assign the ydb.editor role for the database to the user who will be migrating the queries.

3. Create a service account and static access key, as described in this section. If your queries write results to a bucket, additionally assign the storage.uploader role to the service account.

4. Save the query texts, as well as connection and binding settings from Yandex Query; this includes names, bucket paths, format, compression, schema, and partitioning settings.

Federated queries: reading files from Object StorageFederated queries: reading files from Object Storage

This scenario mirrors the operation of Yandex Query: the data remains in the bucket, while the database reads it with every query.

Create an external data sourceCreate an external data source

1. Store the static key in the database secrets:

   CREATE SECRET s3_key_id WITH (value = "<key_ID>");
   CREATE SECRET s3_secret_key WITH (value = "<secret_key>");
   

2. Create an external data source. To avoid changing the query text, give it the same name as the connection in Yandex Query:

   CREATE EXTERNAL DATA SOURCE <connection_name> WITH (
       SOURCE_TYPE = "ObjectStorage",
       LOCATION = "https://storage.yandexcloud.net/<bucket_name>/",
       AUTH_METHOD = "AWS",
       AWS_ACCESS_KEY_ID_SECRET_PATH = "s3_key_id",
       AWS_SECRET_ACCESS_KEY_SECRET_PATH = "s3_secret_key",
       AWS_REGION = "ru-central1"
   );
   

   Secrets are not required for a public bucket: specify AUTH_METHOD = "NONE" and do not specify the AWS_* parameters.

3. Check whether the database can successfully read the bucket:

   SELECT *
   FROM <connection_name>.`<path>/`
   WITH (
       FORMAT = "csv_with_names",
       WITH_INFER = "true"
   )
   LIMIT 10;
   

Migrate data bindingsMigrate data bindings

For each binding, create an external table with the same columns, format, and compression:

CREATE EXTERNAL TABLE <binding_name> (
    date Date NOT NULL,
    service_name Utf8,
    cost Double
) WITH (
    DATA_SOURCE = "<connection_name>",
    LOCATION = "<path>/",
    FORMAT = "csv_with_names",
    COMPRESSION = "gzip"
);

If partitioning or partition projection is configured in the binding, copy the PARTITIONED_BY and projection.* settings without changing them.

Migrate queriesMigrate queries

Queries to connections are migrated unchanged if the external source name matches the connection name. Remove the bindings. prefix from binding requests:

SELECT service_name, SUM(cost) AS total
FROM bindings.`billing`
WHERE date >= Date("2026-09-01")
GROUP BY service_name
ORDER BY total DESC;

SELECT service_name, SUM(cost) AS total
FROM billing
WHERE date >= Date("2026-09-01")
GROUP BY service_name
ORDER BY total DESC;

You can run requests in the management console, or using the YDB CLI or SDK. For example, to save the result to a CSV file, run:

ydb \
  --endpoint <database_endpoint> \
  --database <path_to_database> \
  sql -s 'SELECT service_name, SUM(cost) AS total FROM billing GROUP BY service_name' \
  --format csv > result.csv

Consider performanceConsider performance

The speed of a federated query is limited by the network throughput of the database nodes during reads from Object Storage. Yandex Query distributed reads across a shared pool of nodes; consequently, the same query may take longer to run in a database with a small number of nodes. If speed is important, increase the number of database nodes or move your data to column-oriented tables.

Column-oriented tables: loading data into the databaseColumn-oriented tables: loading data into the database

If the same files are read repeatedly, e.g., by an auto-refreshing dashboard, upload them into a column-oriented table. Queries to the column-oriented table do not read the bucket again.

1. Create a column-oriented table:

   CREATE TABLE billing_data (
       date Date NOT NULL,
       resource_id Utf8 NOT NULL,
       service_name Utf8,
       cost Double,
       PRIMARY KEY (date, resource_id)
   )
   PARTITION BY HASH(date, resource_id)
   WITH (STORE = COLUMN);
   

2. Load data from the external table:

   INSERT INTO billing_data
   SELECT date, resource_id, service_name, cost
   FROM billing;
   

3. To load additional files, run the same query on a schedule with a filter based on date or path, e.g., from a Cloud Functions function with a timer trigger.

4. Switch queries and dashboards from the external table to billing_data.

Database connectionsDatabase connections

Connections with Yandex Query, Managed Service for PostgreSQL, Managed Service for ClickHouse®, Managed Service for MySQL®, and Yandex MPP Analytics for PostgreSQL are migrated to external data sources of the appropriate type. The syntax of SELECT * FROM<source>.<table> queries does not change. For more information on connection settings, see Federated queries.

Check the resultCheck the result

1. Run each migrated query against the same files both in Yandex Query and in the environment you migrated to.

2. Compare the number of rows and the checksums by the key columns:

   SELECT COUNT(*) AS rows, SUM(cost) AS total_cost
   FROM billing
   WHERE date BETWEEN Date("2026-09-01") AND Date("2026-09-30");
   

3. Switch the dashboards, schedules, and applications that accessed Yandex Query to your new environment.

If something went wrongIf something went wrong

If you were unable to resolve your issue, contact support and provide the following:

• Database ID.
• Query text and error text.
• Error time and query ID, if there is one in the output.