Data sources

A data source is the connection to a database or service used to load metadata into the catalog. User data itself is not loaded. When profiling is enabled, additional read queries are executed on data or samples to calculate statistics on columns and tables.

The metadata with the following resource information is loaded into the catalog:

Source details	Description
Storage objects	Databases, schemas, tables, views, indexes
Data structure	Column/field names, their data types, and whether they can store NULL values (nullable), nested columns/fields
Comments	Descriptions of tables and columns
Data lineage	Links between tables and views
Data profiling	Statistics on tables and columns
Deleted objects	Objects that were deleted from the source since the previous upload

Source metadataSource metadata

The set of loaded metadata depends on the data source:

• PostgreSQL
• MySQL®
• ClickHouse®
• OpenSearch
• Yandex MPP Analytics for PostgreSQL/Greenplum®
• Yandex StoreDoc/MongoDB
• Yandex Data Transfer
• Yandex WebSQL
• Yandex DataLens
• Trino
• Apache Airflow™

PostgreSQLPostgreSQL

The metadata with the following resource information is loaded into the catalog from PostgreSQL:

Source details	Description
Databases and schemas	Databases and schemas available in the source
Tables	Tables, their technical names and the schema they belong to
Views	Views and their SQL definitions
Stored procedures	Stored procedures and their SQL definitions
Columns	Names, data types, and the schema they belong to
Comments	Table and column descriptions defined via COMMENT ON
Data lineage	Links between tables and views, as well as between views
Column-level data lineage	Links between columns in views
Data profiling	Statistics on tables, rows, and columns
Deleted objects	Objects that were deleted from the source since the previous upload

In the catalog, metadata is structured as: database → schema → table/view → column.

Data lineage can be determined by:

• Parsing SQL definitions of views.

• Analyzing SQL queries.

  SQL query analysis requires PostgreSQL version 13 or higher and the pg_stat_statements extension.

For data profiling, the user needs SELECT privileges on the tables.

MySQL®MySQL®

The metadata with the following resource information is loaded into the catalog from MySQL®:

Source details	Description
Databases	Databases available in the source
Tables	Tables and the database they belong to
Views	Views and their SQL definitions
Stored procedures	Stored procedures and their SQL definitions
Columns	Names and data types
Comments	Comments on tables and columns
Data lineage	Links between tables and views, as well as between views
Column-level data lineage	Links between source columns and view columns
Data profiling	Statistics on tables and columns
Deleted objects	Objects that were deleted from the source since the previous upload

Metadata is loaded from the MySQL® system tables. In the catalog, metadata is structured as: database → table/view → column.

Data lineage is determined by parsing SQL definitions of views.

To collect metadata, the user needs the following privileges:

• SHOW VIEW: For views.
• SELECT for the tables: For data profiling metrics.

Learn more about user privileges in MySQL®.

ClickHouse®ClickHouse®

The metadata with the following resource information is loaded into the catalog from ClickHouse®:

Source details	Description
Databases	Databases available in the source
Tables	User tables
Views	Views and their SQL definitions
Materialized views	Materialized views and their SQL definitions
Dictionaries	Dictionaries and their structure
Columns	Names and data types
Comments	Comments on tables and columns
Data lineage	Links between tables and views, as well as between views
Column-level data lineage	Links between columns
Use of objects	Object access statistics
Data profiling	Statistics on tables and columns
Deleted objects	Objects that were deleted from the source since the previous upload

Metadata is loaded from the ClickHouse® system tables using SQLAlchemy. In the catalog, metadata is structured as: database → table/view → column.

Data lineage is determined by parsing SQL definitions of views stored in system.tables.create_table_query.

For data profiling, the user needs SELECT privileges on the tables.

OpenSearchOpenSearch

The metadata with the following resource information is loaded into the catalog from OpenSearch:

Source details	Description
Indexes	Indexes available in the source
Index fields	Fields included in indexes
Field types	Field data types
Index templates	Rules applied when creating indexes
Data profiling	Statistics on indexes and fields
Deleted objects	Indexes that were deleted from the source since the previous upload

Metadata is loaded from OpenSearch using the API. In the catalog, each index is represented as a dataset with fields. The field name and type are determined based on the index schema.

For data profiling, the user needs read permissions on the indexes.

Yandex MPP Analytics for PostgreSQL/Greenplum®Yandex MPP Analytics for PostgreSQL/Greenplum®

The metadata with the following resource information is loaded into the catalog from Greenplum® and Yandex MPP Analytics for PostgreSQL:

Source details	Description
Databases and schemas	Databases and schemas available in the source
Tables	Tables, their technical names and the schema they belong to
Views	Views and their SQL definitions
Materialized views	Materialized views and their SQL definitions
External tables	Information about external tables
Columns	Names, data types, and the schema they belong to
Limitations	Check the data integrity
Indexes	Table indexes
Comments	Table and column descriptions defined via COMMENT ON
Data lineage	Links between tables, views, and materialized views
Data profiling	Statistics on tables and columns
Deleted objects	Objects that were deleted from the source since the previous upload

In the catalog, metadata is structured as: database → schema → table/view → column.

Data lineage can be determined by:

• Parsing SQL definitions of views.
• Analyzing SQL queries.

For data profiling, the user needs SELECT privileges on the tables.

Yandex StoreDoc/MongoDBYandex StoreDoc/MongoDB

The metadata with the following resource information is loaded into the catalog from Yandex StoreDoc and MongoDB:

Source details	Description
Databases	Databases available in the source
Collections	Names, database they belong to, list of fields and their types, document structure, technical attributes
Fields	Names, nested fields, types, whether they can store NULL values, mixed types
Deleted objects	Collections that were deleted from the source since the previous upload

In the catalog, metadata is structured as: database → collection → field.

In Yandex StoreDoc and MongoDB, there is no unified collection schema, as documents may have different structures. If schema inference is enabled, a merged collection schema is built based on a sample of documents. To build the schema, you need read permissions on the databases from which metadata is loaded.

Yandex Data TransferYandex Data Transfer

Metadata from Data Transfer is loaded into the catalog for transfers running in selected Yandex Cloud folders.

The metadata contains the following information about the transfer:

• Links between storages, tables and columns.
• Data transformation.

A storage without an active source is created for each transfer participant with no source created in the catalog. In this case, only the metadata obtained within the transfer will be available in the catalog. Later, you can create a source for each storage and configure metadata loading.

Yandex WebSQLYandex WebSQL

Metadata from WebSQL is loaded into the catalog for SQL queries and database objects used in queries. Query results are not loaded. Queries are loaded only for databases for which a source was created in the catalog.

In the catalog, a user sees only their own queries or queries for which they have view permissions.

Yandex DataLensYandex DataLens

Metadata from DataLens is loaded into the catalog for the following objects:

• Datasets
• Charts
• Dashboards
• Reports
• Workbooks and collections

Metadata is loaded for selected connections. If no connections are selected, metadata is loaded for all available connections.

To load metadata, the service account needs the datalens.visitor and datalens.admin roles for the organization. Instead of the datalens.admin role, you can grant permissions to access individual DataLens resources (e.g., datalens.workbooks.viewer). In that case, only metadata for resources to which the service account has access is loaded.

Learn more about roles in DataLens.

ClickHouse® is a registered trademark of ClickHouse, Inc.

Greenplum® and Greenplum Database® are registered trademarks or trademarks of Broadcom Inc. in the United States and/or other countries.