About Yandex MetaData Hub

Yandex MetaData Hub is a service that provides data management capabilities in Yandex Cloud:

• Automatic creation and management of database connection parameters.
• Storing, getting schemas, and checking the evolution of data exchange schemas.
• Creating and managing Hive Metastore clusters.
• Search and visualization of meta information about data storages and links between them.

Connection managementConnection management

With Yandex Connection Manager, you can manage database connection parameters. Connections are created automatically when you create a new managed database cluster in Yandex Cloud, if support for Yandex Connection Manager is configured at cloud level. To configure the integration, contact support.
You cannot edit or delete any connection or secret created automatically together with a new cluster: they are updated automatically when editing user settings in a managed database cluster.

A connection contains the whole information about database connection parameters. The sensitive portion of this information, such as the user password for database access, is stored in Yandex Lockbox as a secret.

Storing table dataStoring table data

You can create Hive Metastore clusters in Yandex MetaData Hub.

Hive Metastore is a table metadata server that:

• Provides client applications with the information on where to get the data to process and how to interpret it.
• Saves the table metadata between running the short-lived computing clusters.
• Shares the data space between concurrently run clusters.
• Links together different ETL systems and tools for working with shared data and simplifies their deployment.
• Provides fault tolerance, scalable storage, and metadata backup.
• Simplifies sending logs and metrics, as well as the update and migration processes.

Data schema registryData schema registry

Schema Registry implements a schema registry, i.e., a centralized repository for managing and validating data schemas. The schema registry ensures safe data schema evolution, resolves data compatibility issues, and enhances system performance by reducing the volume of data transmitted over the network. Moreover, the schema registry will allow you to satisfy data security requirements and promote collaboration across teams. In Schema Registry, you can add schemas in Avro, JSON Schema, and Protobuf formats.

With a schema registry, you can define schemas for your data formats and versions and register them in the registry. After registering a schema, you can use it jointly in various systems and applications. When a supplier sends data to a message recipient, the data schema is included in the message title, and the schema registry ensures that the schema is valid and compatible with the expected one for the subject.