About Yandex MetaData Hub

Yandex MetaData Hub is a service that provides data management capabilities in Yandex Cloud:

• Automatic creation and management of database connection parameters.
• Storing, getting schemas, and checking the evolution of data exchange schemas.
• Creating and managing Hive Metastore clusters.
• Search and visualization of meta information about data storages and links between them.

Connection managementConnection management

With Yandex Connection Manager, you can manage database connection parameters. Connections are created automatically when you create a new managed database cluster in Yandex Cloud, if support for Yandex Connection Manager is configured at cloud level. To configure the integration, contact support.
You cannot edit or delete any connection or secret created automatically together with a new cluster: they are updated automatically when editing user settings in a managed database cluster.

A connection contains the whole information about database connection parameters. The sensitive part of this information, such as the user password for database access, is stored in Yandex Lockbox as a secret.

Table metadata managementTable metadata management

You can create Hive Metastore clusters in Yandex MetaData Hub.

Hive Metastore is a table metadata server that:

• Provides client applications with the information on where to get the data to process and how to interpret it.
• Saves the table metadata between running the short-lived computing clusters.
• Shares the data space between concurrently run clusters.
• Links together different ETL systems and tools for working with shared data and simplifies their deployment.
• Provides fault tolerance, scalable storage, and metadata backup.
• Simplifies sending logs and metrics, as well as the update and migration processes.
• Has a key role in cloud data processing scenarios by enabling different tools (Spark, Trino, Hive) to access the same metadata.

Some Apache products, including Hive, Spark, and Impala, feature Metastore support.

Use casesUse cases

• Transferring metadata between Yandex Data Processing clusters using Metastore
• Shared use of Yandex Data Processing tables through Metastore

Data schema registryData schema registry

Schema Registry implements a schema registry, i.e., a centralized repository for managing and validating data schemas. The schema registry ensures safe data schema evolution, resolves data compatibility issues, and enhances system performance by reducing the volume of data transmitted over the network. Moreover, the schema registry will allow you to satisfy data security requirements and promote collaboration across teams. In Schema Registry, you can add schemas in Avro, JSON Schema, and Protobuf formats.

With a schema registry, you can define schemas for your data formats and versions and register them in the registry. After registering a schema, you can use it jointly in various systems and applications. When a supplier sends data to a message recipient, the data schema is included in the message title, and the schema registry ensures that the schema is valid and compatible with the expected one for the subject.

Use casesUse cases

• Creating a schema registry to deliver data in Debezium CDC format from Apache Kafka®

Metadata collection and markupMetadata collection and markup

This feature is at the Preview. Contact support to request access.

Data Catalog allows you to collect, analyze, and mark up metadata drawn from various sources. You can upload structural metadata, e.g., list of tables in a managed database cluster, their schemas, links between tables.

You can use Data Catalog to:

• Collect, store, and organize metadata.
• Find a dashboard with relevant business indicators.
• Analyze and interpret business indicators.
• Find data for your business needs.
• Find information sources behind a particular object.
• Find data owners, including passive ownership through subscription.
• Build a schema for data consumer.