About Yandex MetaData Hub

Yandex MetaData Hub is a service that provides data management capabilities in Yandex Cloud:

• Automatic creation and management of database connection parameters.
• Storing, getting schemas, and checking the evolution of data exchange schemas.
• Creating and managing Apache Hive™ Metastore clusters.
• Search and visualization of meta information about data storages and links between them.

Connection managementConnection management

With Yandex Connection Manager, you can manage database connection parameters. Connections are created automatically when you create a new managed database cluster in Yandex Cloud for PostgreSQL, ClickHouse®, and MySQL® clusters. For other types of clusters, you can create a connection manually.
Clusters created before the Connection Manager integration was implemented, will operate as they used to. For such clusters, you can enable integration manually in additional cluster settings.
You cannot edit or delete any connection or secret created automatically together with a new cluster: they are updated automatically when editing user settings in a managed database cluster.

A connection contains all the information about database connection parameters. The sensitive part of this information, such as the user password for database access, is stored in Yandex Lockbox as a secret.

Table metadata managementTable metadata management

You can create Apache Hive™ Metastore clusters in Yandex MetaData Hub.

Apache Hive™ Metastore is a table metadata server that:

• Provides client applications with the information on where to get the data to process and how to interpret it.
• Saves the table metadata between running the short-lived computing clusters.
• Shares the data space between concurrently run clusters.
• Links together different ETL systems and tools for working with shared data and simplifies their deployment.
• Provides fault tolerance, scalable storage, and metadata backup.
• Simplifies sending logs and metrics, as well as the update and migration processes.
• Has a key role in cloud data processing scenarios by enabling different tools (Spark, Trino, Hive) to access the same metadata.

Some Apache products, including Hive, Spark, and Impala, feature Apache Hive™ Metastore support.

Use casesUse cases

• Transferring metadata between Yandex Data Processing clusters using Apache Hive™ Metastore
• Shared use of Yandex Data Processing tables through Apache Hive™ Metastore

Data schema registryData schema registry

This feature is in the Preview stage.

Schema Registry implements a schema registry, i.e., a centralized repository for managing and validating data schemas. The schema registry ensures safe data schema evolution, resolves data compatibility issues, and enhances system performance by reducing the volume of data transmitted over the network. Moreover, the schema registry will allow you to satisfy data security requirements and promote collaboration across teams. In Schema Registry, you can add schemas in Avro, JSON Schema, and Protobuf formats.

With a schema registry, you can define schemas for your data formats and versions and register them in the registry. After registering a schema, you can use it jointly in various systems and applications. When a supplier sends data to a message recipient, the data schema is included in the message title, and the schema registry ensures that the schema is valid and compatible with the expected one for the subject.

Use casesUse cases

• Creating a schema registry to deliver data in Debezium CDC format from Apache Kafka®

Metadata collection and markupMetadata collection and markup

This feature is in the Preview stage.

Data Catalog allows you to collect, analyze, and mark up metadata drawn from various sources. You can upload structural metadata, e.g., list of tables in a managed database cluster, their schemas, links between tables.

You can use Data Catalog to:

• Collect, store, and organize metadata.
• Find a dashboard with relevant business indicators.
• Analyze and interpret business indicators.
• Find data for your business needs.
• Find information sources behind a particular object.
• Find data owners, including passive ownership through subscription.
• Build a schema for data consumer.

Apache® and Apache Hive™ are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.