Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Working with data using Yandex Query

Written by
Updated at August 21, 2025

Yandex Query is an interactive service for serverless data analysis. You can use it to process information from various storages without having to create a dedicated cluster. Yandex Query supports Yandex Object Storage, Yandex Managed Service for PostgreSQL, and Yandex Managed Service for ClickHouse® data storages.

Follow this tutorial to connect to a Managed Service for PostgreSQL database and query it from a JupyterLab notebook with the help of Query.

  1. Set up your infrastructure.
  2. Get started in Query.
  3. Create a Managed Service for PostgreSQL cluster.
  4. Connect to the Managed Service for PostgreSQL data.

If you no longer need the resources you created, delete them.

Getting started

Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.

  1. On the DataSphere home page, click Try for free and select an account to log in with: Yandex ID or your working account with the identity federation (SSO).
  2. Select the Yandex Identity Hub organization you are going to use in Yandex Cloud.
  3. Create a community.
  4. Link your billing account to the DataSphere community you are going to work in. Make sure you have a linked billing account and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account yet, create one in the DataSphere interface.

The cost of infrastructure support for working with Managed Service for PostgreSQL data includes:

Set up your infrastructure

Log in to the Yandex Cloud management console and select the organization you use to access DataSphere. On the Yandex Cloud Billing page, make sure you have a billing account linked.

If you have an active billing account, you can go to the cloud page to create or select a folder to run your infrastructure.

Note

If you are using an identity federation to work with Yandex Cloud, you might not have access to billing details. In this case, contact your Yandex Cloud organization administrator.

Create a folder

  1. In the management console, select a cloud and click Create folder.
  2. Name your folder, e.g., data-folder.
  3. Click Create.

Create a service account for the DataSphere project

  1. Navigate to data-folder.
  2. In the list of services, select Identity and Access Management.
  3. Click Create service account.
  4. Name the service account, e.g., yq-sa.
  5. Click Add role and assign the following roles to the service account:
    • yq.editor to run Query queries.
    • managed-postgresql.viewer to view the contents of the Managed Service for PostgreSQL cluster.
  6. Click Create.

Add the service account to the project

To enable the service account to run a DataSphere project, add it to the list of project members.

  1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

  2. In the Members tab, click Add member.
  3. Select the yq-sa account and click Add.
  4. Switch your service account role to Editor.

Create an authorized key for a service account

To allow the service account to send queries using Query, create an authorized key.

The yandex_query_magic package provides magic commands for working in Jupyter. Install it to send queries to Query. Paste the code into the yq-storage.ipynb notebook cells.

  1. Open the DataSphere project:

    1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

    2. Click Open project in JupyterLab and wait for the loading to complete.
    3. Open the notebook tab.

  2. Install the yandex_query_magic package:

    %pip install yandex_query_magic

  3. Once the installation is complete, from the top panel, select Kernel ⟶ Restart kernel....

  4. Upload the extension:

    %load_ext yandex_query_magic

  5. Configure the connection by specifying the data-folder ID and the name of the authorized key secret:

    %yq_settings --folder-id <folder_ID> --env-auth yq_access_key

  6. Run a test query to Query:

    %yq select "Hello, world!"
  1. In the management console, navigate to data-folder.
  2. In the list of services, select Identity and Access Management.
  3. In the left-hand panel, select Service accounts.
  4. In the list that opens, select yq-sa.
  5. Click Create new key in the top panel and select Create authorized key.
  6. Select the encryption algorithm and click Create.
  7. Click Download file with keys.

Create a secret

To get an authorized key from the notebook, create a secret with the contents of the authorized key file.

  1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

  2. Under Project resources, click Secret.
  3. Click Create.
  4. In the Name field, enter a name for the secret: yq_access_key.
  5. In the Value field, paste the full contents of the authorized key file you downloaded.
  6. Click Create.

Create a notebook

Queries to the Managed Service for PostgreSQL database through Query will be sent from the notebook.

  1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

  2. Click Open project in JupyterLab and wait for the loading to complete.
  3. In the top panel, click File and select New ⟶ Notebook.
  4. Select a kernel and click Select.

Get started in Query

The yandex_query_magic package provides magic commands for working in Jupyter. Install it to send queries to Query. Paste the code into the yq-storage.ipynb notebook cells.

  1. Open the DataSphere project:

    1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

    2. Click Open project in JupyterLab and wait for the loading to complete.
    3. Open the notebook tab.

  2. Install the yandex_query_magic package:

    %pip install yandex_query_magic

  3. Once the installation is complete, from the top panel, select Kernel ⟶ Restart kernel....

  4. Upload the extension:

    %load_ext yandex_query_magic

  5. Configure the connection by specifying the data-folder ID and the name of the authorized key secret:

    %yq_settings --folder-id <folder_ID> --env-auth yq_access_key

  6. Run a test query to Query:

    %yq select "Hello, world!"

Create a Managed Service for PostgreSQL cluster

Any running Managed Service for PostgreSQL cluster with the Yandex Query access option enabled is suitable for sending queries.

  1. In the management console, select data-folder.
  2. Select Managed Service for PostgreSQL.
  3. Click Create cluster.
  4. In the Cluster name field, enter the cluster name, e.g., postgresql.
  5. Under Database:
    • Specify DB name, e.g., db1.
    • Specify Username and Password.
  6. Under Service settings, enable Yandex Query access and Access from the management console.
  7. You can leave the other settings at their defaults.
  8. Click Create cluster.

Create a table

In this step, you will create a test table with random numbers from 0 to 100.

  1. In the management console, open the postgresql cluster page and navigate to the SQL tab.

  2. Enter Username and Password you specified when creating the cluster.

  3. In the input window on the right, paste an SQL query:

    CREATE TABLE test (
    id SERIAL PRIMARY KEY,
    number INT
);

INSERT INTO test (number)
SELECT random() * 100
FROM generate_series(1, 100);

  4. Click Execute.

Connect to Managed Service for PostgreSQL data

To create a Query connection:

  1. In the management console, select data-folder.

  2. In the list of services, select Yandex Query.

  3. In the left-hand panel, select Connections.

  4. Click Create new.

  5. Name the connection, e.g., postgresql.

  6. Select the Managed Service for PostgreSQL connection type.

  7. Under Connection type parameters:

    • Cluster: Select the postgresql cluster you created earlier.
    • Service account: yq-sa.
    • Database: db1.
    • Enter Login and Password you specified when creating the cluster.

  8. Click Create.

To check the connection, get the table data from the notebook cell:

%yq SELECT * FROM postgresql.test

How to delete the resources you created

To stop paying for the resources you created:

Previous
Creating an MLFlow server for logging experiments and artifacts
Next
Federated data queries using Query