Data sources

A data source is a connection to a database or service used to ingest metadata into a catalog. User data is never ingested. When profiling is enabled, the system runs additional read queries against data or samples to calculate column and table statistics.

The metadata ingested into a catalog contains the following source information:

Source information	Description
Stored objects	Databases, schemas, tables, views, and indexes
Data structure	Column or field names, their data types and support for storing NULL values (nullability), nested columns or fields
Comments	Descriptions of tables and columns
Data lineage	Links between tables and views
Data profiling	Statistics on tables and columns
Deleted objects	Objects removed from the source since the previous ingestion

Source metadataSource metadata

The scope of ingested metadata depends on the data source:

• PostgreSQL
• MySQL®
• ClickHouse®
• OpenSearch
• Yandex MPP Analytics for PostgreSQL/Greenplum®
• Yandex StoreDoc/MongoDB
• Yandex Data Transfer
• Yandex WebSQL
• Yandex DataLens

PostgreSQLPostgreSQL

Metadata with the following source information is ingested from PostgreSQL into the catalog:

Source information	Description
Databases and schemas	Databases and schemas available in the source
Tables	Tables, their technical names, and parent schema mapping
Views	Views and their SQL definitions
Stored procedures	Stored procedures and their SQL definitions
Columns	Names, data types, and parent schema mapping
Comments	Table and column descriptions defined via COMMENT ON
Data lineage	Links between tables and views, as well as between views
Column-level data lineage	Links between columns in views
Data profiling	Statistics on tables, rows, and columns
Deleted objects	Objects removed from the source since the previous ingestion

Within a catalog, metadata has the following structure: database → schema → table/view → column.

There are two methods for data lineage tracing:

• Parsing the SQL definitions of views.

• Analyzing SQL queries.

  SQL query analysis requires PostgreSQL version 13 or higher and the pg_stat_statements extension.

For data profiling, the user needs the SELECT permissions for the tables.

MySQL®MySQL®

Metadata with the following source information is ingested from MySQL® into the catalog:

Source information	Description
Databases	Databases available in the source
Tables	Tables and their parent database mapping
Views	Views and their SQL definitions
Stored procedures	Stored procedures and their SQL definitions
Columns	Names and data types
Comments	Comments to tables and columns
Data lineage	Links between tables and views, as well as between views
Column-level data lineage	Links between source columns and view columns
Data profiling	Statistics on tables and columns
Deleted objects	Objects removed from the source since the previous ingestion

Metadata is ingested from the MySQL® system tables. Within a catalog, metadata has the following structure: database → table/view → column.

Data lineage is traced by parsing SQL definitions of views.

To collect metadata, the user needs the following privileges:

• SHOW VIEW for views.
• SELECT for tables to access data profiling metrics.

Learn more about user privileges in MySQL®.

ClickHouse®ClickHouse®

Metadata with the following source information is ingested from ClickHouse® into the catalog:

Source information	Description
Databases	Databases available in the source
Tables	User tables
Views	Views and their SQL definitions
Materialized views	Materialized views and their SQL definitions
Dictionaries	Dictionaries and their structure
Columns	Names and data types
Comments	Comments to tables and columns
Data lineage	Links between tables and views, as well as between views
Column-level data lineage	Links between columns
Use of objects	Object access statistics
Data profiling	Statistics on tables and columns
Deleted objects	Objects removed from the source since the previous ingestion

Metadata is ingested from the ClickHouse® system tables using SQLAlchemy. Within a catalog, metadata has the following structure: database → table/view → column.

Data lineage is traced by parsing SQL definitions of views stored in system.tables.create_table_query.

For data profiling, the user needs the SELECT permissions for the tables.

OpenSearchOpenSearch

Metadata with the following source information is ingested from OpenSearch into the catalog:

Source information	Description
Indexes	Indexes available in the source
Index fields	Fields included in indexes
Field types	Field data types
Index templates	Rules applied when creating indexes
Data profiling	Statistics on indexes and fields
Deleted objects	Indexes removed from the source since the previous ingestion

Metadata is ingested from OpenSearch via the API. Within a catalog, each index is represented as a dataset with fields. The field name and type are determined based on the index schema.

For data profiling, the user needs permissions to read indexes.

Yandex MPP Analytics for PostgreSQL/Greenplum®Yandex MPP Analytics for PostgreSQL/Greenplum®

Metadata with the following source information is ingested from Greenplum® and Yandex MPP Analytics for PostgreSQL into the catalog:

Source information	Description
Databases and schemas	Databases and schemas available in the source
Tables	Tables, their technical names, and parent schema mapping
Views	Views and their SQL definitions
Materialized views	Materialized views and their SQL definitions
External tables	Information about external tables
Columns	Names, data types, and parent schema mapping
Limitations	Rules for maintaining data integrity
Indexes	Table indexes
Comments	Table and column descriptions defined via COMMENT ON
Data lineage	Links between tables, views, and materialized views
Data profiling	Statistics on tables and columns
Deleted objects	Objects removed from the source since the previous ingestion

Within a catalog, metadata has the following structure: database → schema → table/view → column.

There are two methods for data lineage tracing:

• Parsing the SQL definitions of views.
• Analyzing SQL queries.

For data profiling, the user needs the SELECT permissions for the tables.

Yandex StoreDoc/MongoDBYandex StoreDoc/MongoDB

Metadata with the following source information is ingested from MongoDB and Yandex StoreDoc into the catalog:

Source information	Description
Databases	Databases available in the source
Collections	Names, parent database mapping, list of fields and their types, document structure, technical attributes
Fields	Names, nested fields, types, support for storing NULL values, mixed types
Deleted objects	Collections removed from the source since the previous ingestion

Within a catalog, metadata has the following structure: database → collection → field.

In Yandex StoreDoc and MongoDB, there is no single collection schema, as individual documents may have varying structures. If schema inference is enabled, a unified collection schema is generated based on a sample of documents. To build the schema, you need read permissions for the databases metadata is ingested from.

Yandex Data TransferYandex Data Transfer

The metadata ingested from Data Transfer into a catalog constitutes metadata of transfers running within the selected Yandex Cloud folders.

This metadata includes the following transfer information:

• Links between storages, tables, and columns.
• Data transformations.

For each transfer participant without a corresponding source in the catalog, a storage without an active source is created. In this case, the catalog will only store metadata obtained during the transfer. You can later create a source for each storage and configure metadata ingestion.

Yandex WebSQLYandex WebSQL

The metadata ingested from WebSQL into a catalog constitutes the metadata of SQL queries and database objects used in queries. Query results are not ingested. The ingested queries are restricted to databases which have a corresponding source in the catalog.

In the catalog, a user will only see their queries or queries for which they have view permissions.

Yandex DataLensYandex DataLens

A catalog ingests metadata for the following DataLens objects:

• Datasets
• Charts
• Dashboards
• Reports
• Workbooks and collections

The ingested metadata is limited to the pre-selected connections. If no connections are selected, the system will ingest metadata from all available connections.

To ingest metadata, the service account needs the datalens.visitor and datalens.admin roles for the organization. Instead of the datalens.admin role, you can grant granular permissions to access individual DataLens resources (e.g., datalens.workbooks.viewer). In this scenario, the ingested metadata will be limited to the specific resources to which the service account has access.

Learn more about roles in DataLens.

ClickHouse® is a registered trademark of ClickHouse, Inc.

Greenplum® and Greenplum Database® are registered trademarks or trademarks of Broadcom Inc. in the United States and/or other countries.