Building a data platform based on Yandex Cloud
- Data migration and DBMS version upgrades
- Replication and streaming data change capture
- Data loading and delivery
- Integration and exchange between data storage and processing systems
- Analytical calculations
- Pipeline orchestration
- Storage, archiving, and data lifecycle
- Performance and diagnostics
- Monitoring, logs, audit, billing
- Security and access
- Data schema management
- Cluster deployment and configuration
Data migration and DBMS version upgrades
Migration is a task that involves transferring data from one storage system to another, e.g., from on-premises databases to managed cloud databases, between clusters, or with a change of the DBMS type. The tutorials in this group describe migration scenarios for MySQL®, PostgreSQL, ClickHouse®, Greenplum®, MongoDB, YDB, OpenSearch, and other systems, as well as database version upgrades with minimal downtime.
Full list of tutorials
- Asynchronously replicating data from PostgreSQL to ClickHouse®
- Migrating a database from a third-party MySQL® cluster to a Yandex Managed Service for MySQL® cluster
- Migrating a database from a third-party PostgreSQL cluster to Managed Service for PostgreSQL
- Migrating a database from a third-party Valkey™ cluster to Yandex Managed Service for Valkey™
- Migrating databases from Yandex Managed Service for MySQL® to a third-party MySQL® cluster
- Migrating databases from MySQL® to ClickHouse® using Yandex Data Transfer
- Migrating databases from Greenplum® to PostgreSQL
- Migrating data to Managed Service for ClickHouse® using ClickHouse®
- Migrating data to Yandex Managed Service for ClickHouse® using Yandex Data Transfer
- Migrating data from a third-party OpenSearch cluster to Yandex Managed Service for OpenSearch using Yandex Data Transfer
- Migrating data from a third-party OpenSearch cluster to Yandex Managed Service for OpenSearch using Yandex Data Transfer
- Migrating data to Yandex StoreDoc
- Migrating data to Yandex Object Storage using Yandex Data Transfer
- Migrating data from AWS RDS for PostgreSQL to Yandex Managed Service for PostgreSQL using Yandex Data Transfer
- Migrating data from Yandex Managed Service for MySQL® to Yandex MPP Analytics for PostgreSQL using Yandex Data Transfer
- Migrating data from Managed Service for MySQL® to Managed Service for PostgreSQL using Data Transfer
- Migrating data from Managed Service for PostgreSQL to Managed Service for MySQL® using Data Transfer
- Migrating data from Yandex Managed Service for MySQL® to Yandex MPP Analytics for PostgreSQL using Yandex Data Transfer
- Migrating data from Managed Service for MySQL® to Managed Service for PostgreSQL using Data Transfer
- Migrating data from Managed Service for PostgreSQL to Managed Service for MySQL® using Data Transfer
- Migrating data from Yandex Object Storage to Yandex Managed Service for MySQL® using Yandex Data Transfer
- Migrating databases from Google BigQuery to Yandex Managed Service for ClickHouse®
- Migrating databases from Yandex Managed Service for MySQL® to Yandex Managed Service for YDB using Yandex Data Transfer
- Migrating databases from Yandex Managed Service for MySQL® to Yandex Object Storage
- Migrating databases from Greenplum® to ClickHouse®
- Migrating a Yandex Managed Service for PostgreSQL cluster to a different version using Yandex Data Transfer
- Migrating a Yandex StoreDoc cluster from version 4.4 to 6.0 using Yandex Data Transfer
- Migrating an HDFS Yandex Data Processing cluster to a different availability zone
- Upgrading a Managed Service for Apache Kafka® cluster to migrate from ZooKeeper to KRaft
Replication and streaming data change capture
Replication and CDC (Change Data Capture) are processes for continuous tracking of changes at the source and delivering them to the target in real time. The tutorials of this group describe the configuration of cross-cluster replication, change capture with the help of Debezium, and delivery of data streams to target systems.
Full list of tutorials
- Delivering data from Yandex Managed Service for MySQL® to Yandex Managed Service for Apache Kafka® using Debezium
- Delivering data from Yandex Managed Service for PostgreSQL to Yandex Managed Service for Apache Kafka® using Debezium
- Delivering data from Yandex Managed Service for PostgreSQL to Yandex Managed Service for YDB using Yandex Data Transfer
- Delivering data from Yandex Managed Service for YDB to Yandex Managed Service for Apache Kafka® using Yandex Data Transfer
- Processing Debezium CDC streams
- Syncing data from a third-party MySQL® cluster to Yandex Managed Service for MySQL® using Yandex Data Transfer
- Migrating a database from Managed Service for PostgreSQL
- Logical replication PostgreSQL
- Creating a schema registry to deliver data in Debezium CDC format from Apache Kafka®
Data loading and delivery
The tutorials describe architectural patterns for building data streams: from sources (external databases, files, logs, brokers) to storage systems and back. They cover scenarios including streaming via YDS, event delivery from Apache Kafka® and RabbitMQ, batch loading, and end-to-end transfer via Data Transfer.
Full list of tutorials
- Ingesting data into storage systems
- Smart log processing
- Data transfer in microservice architectures
- Adding data to ClickHouse®
- Fetching data from Yandex Managed Service for Apache Kafka® to Yandex Managed Service for ClickHouse®
- Fetching data from RabbitMQ to Yandex Managed Service for ClickHouse®
- Delivering data to ksqlDB
- Delivering data from a Data Streams queue to Managed Service for Apache Kafka® using Yandex Data Transfer
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex Data Streams using Yandex Data Transfer
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex Managed Service for YDB using Yandex Data Transfer
- Saving a Yandex Data Streams data stream in Yandex Managed Service for ClickHouse®
- Delivering data from an Data Streams queue to Managed Service for YDB using Yandex Data Transfer
- Yandex Managed Service for YDB change data capture and delivery to Yandex Data Streams
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex Managed Service for PostgreSQL using Yandex Data Transfer
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex MPP Analytics for PostgreSQL using Yandex Data Transfer
- Migrating a database from a third-party Apache Kafka® cluster to Yandex Managed Service for Apache Kafka®
- Configuring Kafka Connect to work with a Yandex Managed Service for Apache Kafka® cluster
- Moving data between Managed Service for Apache Kafka® clusters using Data Transfer
- Delivering data from Yandex Managed Service for MySQL® to Yandex Managed Service for Apache Kafka® using Yandex Data Transfer
- Delivering data from Yandex Managed Service for PostgreSQL to Yandex Managed Service for Apache Kafka® using Yandex Data Transfer
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex Managed Service for ClickHouse® using Yandex Data Transfer
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex StoreDoc using Yandex Data Transfer
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex Managed Service for MySQL® using Yandex Data Transfer
- Delivering data from Yandex Managed Service for Apache Kafka® to Yandex Managed Service for OpenSearch using Yandex Data Transfer
- Loading data from Yandex Direct to a Yandex Managed Service for ClickHouse® data mart using Yandex Cloud Functions, Yandex Object Storage, and Yandex Data Transfer
- Migrating data from a third-party Greenplum® or PostgreSQL cluster to Yandex MPP Analytics for PostgreSQL using Yandex Data Transfer
- Migrating MySQL® clusters
- Migrating to a third-party MySQL® cluster
- Migrating PostgreSQL clusters
- Migrating MongoDB clusters
- MySQL® change data capture and delivery to YDS
- PostgreSQL change data capture and delivery to YDS
Integration and exchange between data storage and processing systems
The tutorials of this group describe bidirectional data exchange between heterogeneous storage and computing systems: ClickHouse®, Greenplum®, PostgreSQL, YDB, OpenSearch, Object Storage, and external DBMSs via JDBC. The scenarios include delivering data from analytical clusters to relational databases, data exchange between Object Storage and managed database services, and integration between OpenSearch and other storage solutions.
Full list of tutorials
- Exchanging data between Yandex Managed Service for ClickHouse® and Yandex Data Processing
- Shared use of Yandex Data Processing tables through Apache Hive™ Metastore
- Importing data from Yandex Object Storage, processing and exporting to Yandex Managed Service for ClickHouse®
- Working with Apache Kafka® topics using PySpark jobs in Yandex Data Processing
- Yandex Managed Service for ClickHouse® integration with external Microsoft SQL Server database via ClickHouse® JDBC Bridge
- Yandex Managed Service for ClickHouse® integration with Oracle via ClickHouse® JDBC Bridge
- Using Yandex Object Storage in Yandex Managed Service for Apache Spark™
- Executing analytical queries in Yandex Managed Service for PostgreSQL with processing in Yandex Managed Service for ClickHouse® using pg_clickhouse and Yandex Data Transfer
- Loading data from Yandex Object Storage to Yandex Managed Service for ClickHouse® using Yandex Data Transfer
- Loading data from Yandex Object Storage to Yandex MPP Analytics for PostgreSQL using Yandex Data Transfer
- Migrating data from Yandex Object Storage to Yandex Managed Service for PostgreSQL using Yandex Data Transfer
- Loading data from Yandex Object Storage to Yandex Managed Service for YDB using Yandex Data Transfer
- Copying data from Managed Service for OpenSearch to Managed Service for ClickHouse® using Yandex Data Transfer
- Copying data from Managed Service for OpenSearch to Yandex MPP Analytics for PostgreSQL using Yandex Data Transfer
- Migrating data from Yandex Managed Service for OpenSearch to Yandex Managed Service for YDB using Yandex Data Transfer
- Loading data from Yandex Managed Service for OpenSearch to Yandex Object Storage using Yandex Data Transfer
- Migrating data from Yandex Managed Service for PostgreSQL to Yandex Managed Service for OpenSearch using Yandex Data Transfer
- Loading data from Yandex Managed Service for YDB to Yandex Managed Service for ClickHouse® using Yandex Data Transfer
- Migrating databases from Managed Service for PostgreSQL to Object Storage
- Loading data from Yandex Managed Service for YDB to Yandex Object Storage using Yandex Data Transfer
Analytical calculations
Tutorials on how to run analytical jobs: Apache Spark™, PySpark, Apache Hive™ Metastore, MapReduce. This includes scenarios for remote running of jobs, data reading and writing, Apache Spark™ integration with Apache Hive™ Metastore and DataSphere, and loading data from relational sources via Sqoop.
Full list of tutorials
- Working with Hive jobs
- Working with MapReduce jobs
- Working with PySpark jobs
- Working with Spark jobs
- Running Apache Hive jobs
- Launching and managing applications for Spark and PySpark
- Running jobs from remote hosts that are not part of the Yandex Data Processing cluster
- Working with an Object Storage table from a PySpark job using Apache Hive™ Metastore and Apache Iceberg™
- Running a PySpark job in Yandex Managed Service for YTsaurus
- Yandex Managed Service for Apache Spark™ integration with Apache Hive™ Metastore
- Transferring metadata across Yandex Data Processing clusters using Apache Hive™ Metastore
- Using Yandex Managed Service for Apache Spark™ in Yandex DataSphere
- Importing data from Yandex Managed Service for MySQL® to Yandex Data Processing using Sqoop
- Importing data from Yandex Managed Service for PostgreSQL to Yandex Data Processing using Sqoop
Pipeline orchestration
Tutorials on describing the automation and planning the execution of data processing jobs using Apache Airflow™. These scenarios include auto-running jobs, use of the Python SDK inside a DAG, joint execution of Apache Airflow™ and Apache Spark™, and setting up email notifications in the event of pipeline failures.
Full list of tutorials
- Automating Yandex Query tasks with Yandex Managed Service for Apache Airflow™
- Sending requests to the Yandex Cloud API via the Yandex Cloud Python SDK
- Automating operations using Yandex Managed Service for Apache Airflow™
- Running a PySpark job using Yandex Managed Service for Apache Airflow™
- Automating operations with Yandex Data Processing using Yandex Managed Service for Apache Airflow™
- Configuring an SMTP server to send e-mail notifications
Storage, archiving, and data lifecycle
Tutorials on data lifecycle management in storage systems: hybrid storage, sharding, cold storage policies and index rotation in OpenSearch, retention policies, data archiving.
Full list of tutorials
- Using hybrid storage in Yandex Managed Service for ClickHouse®
- Sharding Yandex Managed Service for ClickHouse® tables
- Sharding Yandex StoreDoc collections
- Configuring a cold storage policy in Yandex Managed Service for OpenSearch
- Configuring an index policy in Yandex Managed Service for OpenSearch
- Exporting Greenplum® data to a hybrid storage in Yandex Object Storage
- Monitoring message loss in an Apache Kafka® topic
- Syncing data from Apache Kafka® topics to an Object Storage bucket without using the internet
Performance and diagnostics
Bottleneck diagnostics, profiling, and troubleshooting performance issues in DBMSs and processing services. These scenarios will help you quickly isolate and fix issues related to load and query execution speed.
Full list of tutorials
- Diagnosing Managed Service for PostgreSQL cluster performance issues
- Managed Service for PostgreSQL performance analysis and optimization
- Managed Service for MySQL® performance analysis and tuning
- Performance analysis and tuning of Yandex StoreDoc
- Fixing string sorting issues in PostgreSQL after upgrading glibc
Monitoring, logs, audit, billing
Tutorials on how to process and analyze various types of telemetry including audit logs, Cloud Logging logs, and detailed billing files, how to set up OpenSearch notifications and work with Jupyter Notebook in conjunction with Yandex Query.
Full list of tutorials
Security and access
Scenarios for configuring secure access to data platform resources: authentication to OpenSearch via SAML and Keycloak, secret management when running PySpark jobs and via Connection Manager, and setting up DNS peering for a private ClickHouse® network.
Full list of tutorials
- Authenticating a Yandex Managed Service for OpenSearch cluster in OpenSearch Dashboards using Keycloak
- Using a Yandex Lockbox secret in a PySpark job to connect to Yandex Managed Service for PostgreSQL
- Configuring permissions for access to a secret created by Yandex Connection Manager for a Yandex Managed Service for PostgreSQL user
- Configuring Yandex Cloud DNS to access a Yandex Managed Service for ClickHouse® cluster from other cloud networks
Data schema management
Schema Registry is a service made to store and manage message schemas in Avro, JSON Schema, and Protobuf formats. The scenarios include creating and configuring a managed Schema Registry, working with REST APIs, and integration with Confluent Schema Registry to ensure data compatibility in stream processing.
Full list of tutorials
- Managing data schemas in Managed Service for Apache Kafka®
- Using Managed Schema Registry with Yandex Managed Service for Apache Kafka®
- Using Managed Schema Registry with Yandex Managed Service for Apache Kafka® through the REST API
- Using Confluent Schema Registry with Yandex Managed Service for Apache Kafka®
Cluster deployment and configuration
Specialized configuration and deployment scenarios for managed database clusters: installing the Apache Kafka® web interface, setting up a web server with a configuration file for the s3 protocol, initialization in Yandex Data Processing via init scripts, using a PostgreSQL cluster for 1C, storing PHP sessions in a Yandex Managed Service for Valkey™ cluster, setting up Managed Service for ClickHouse® for Graphite, PXF named queries, and connecting a lemmatizer to OpenSearch.
Full list of tutorials
- Unassisted deployment of the Apache Kafka® web interface
- Creating an external table from a Yandex Object Storage bucket table using a configuration file
- Mounting Yandex Object Storage buckets to the file system of Yandex Data Processing hosts
- Creating a PostgreSQL cluster for 1C:Enterprise
- Using a Yandex Managed Service for Valkey™ cluster as a PHP session storage
- Configuring Yandex Managed Service for ClickHouse® for Graphite
- Getting data from external sources using named queries
- Using the yandex-lemmer plugin in Yandex Managed Service for OpenSearch
- Connecting an MCP client to an OpenSearch cluster
- Distributed locks for 1C:Enterprise in a Yandex Managed Service for Valkey™ cluster
- Using a Yandex Managed Service for Valkey™ cluster as a cache storage for 1C-Bitrix: Site Management
- Using Yandex Managed Service for Valkey™ with the Search module and Yandex AI Studio models for vector search in a RAG system