Setting up a backend for data sources

Data Catalog uses source connections to read technical metadata, such as definitions of tables, views, materialized views, procedures, indexes, as well as information from the system tables of the source. For advanced statistical estimates, Data Catalog may also execute data read or data slice queries.

For complete and accurate metadata reads, you should configure the Data Catalog source backend, i.e., parameters and access permissions at the level of the database or another service you are going to source metadata from.

General configuration recommendationsGeneral configuration recommendations

For Data Catalog to export the most complete metadata possible, configure additional permissions, accesses, and database settings:

1. To connect to the source, create a dedicated technical user with read-only permissions. Do not grant permissions for data modification, DDL operations, object deletion, or user management.

2. Grant this user access to all objects you need to catalog. If the technical user lacks access to a schema, table, view, or index, Data Catalog may skip it entirely or export incomplete metadata for that object.

   What we recommend:

   • Grant permissions for all schemas and databases you want to see in the catalog.
   • Do not grant permissions for temporary, system, and internal schemas unless they need to be visible to users.
   • Use ingestion filters to explicitly exclude technical objects.

3. For a full metadata export, grant access not just to tables but other objects as well:

   • Views.
   • Materialized views.
   • Stored procedures and functions (if the source supports their export).
   • Field comments and descriptions.
   • System catalogs.
   • Query statistics and logs.

4. Enable profiling separately after you estimate the source load and verify ingestion stability.

   When profiling is enabled, ingestion collects table and column-level statistics and may run additional data read or data slice queries to generate advanced statistical estimates. This results in increased load on the source.

   For production sources, it is best to start in light mode, which is controlled by the size of the sample and the scope of statistical estimates.

Configuring different backend typesConfiguring different backend types

PostgreSQLPostgreSQL

Set up a backend for ingestion of PostgreSQL source metadata:

1. Create a dedicated technical user for the ingestion:

   CREATE USER data_catalog_reader WITH PASSWORD '<password>';
   

   Specify this user in the source connection settings.

2. Grant the user access to relevant databases:

   GRANT CONNECT ON DATABASE <DB_name> TO data_catalog_reader;
   

3. Grant the user access to schemas:

   GRANT USAGE ON SCHEMA <schema_name> TO data_catalog_reader;
   

4. Grant the user access to tables and views:

   GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO data_catalog_reader;
   

5. Grant the following read permissions for the newly created tables to be ingestible as well:

   ALTER DEFAULT PRIVILEGES IN SCHEMA <schema_name>
   GRANT SELECT ON TABLES TO data_catalog_reader;
   

6. In each database subject to data profiling:

   1. Install the statistics collection extension:

      CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
      

   2. Grant the user permissions to read statistics:

      GRANT pg_read_all_stats TO data_catalog_reader;
      

   3. Check that your query history retention period is at least one week long.

   4. Apply the PostgreSQL settings:

      • shared_preload_libraries = 'pg_stat_statements': Restart PostgreSQL after the change.
      • pg_stat_statements.max = 10000: Allows you to increase the number of saved unique queries to collect statistics on rare queries.
      • pg_stat_statements.track = all: Allows tracking nested statements in functions and procedures.

MySQL®MySQL®

Set up a backend for ingestion of MySQL® source metadata:

1. Create a dedicated technical user for the ingestion:

   CREATE USER 'data_catalog_reader'@'%' IDENTIFIED BY '<password>';
   

   Specify this user in the source connection settings.

2. Grant the user access to relevant databases:

   GRANT SELECT ON <DB_name>.* TO 'data_catalog_reader'@'%';
   GRANT SHOW VIEW ON <DB_name>.* TO 'data_catalog_reader'@'%';
   

You need the SELECT privilege for metadata collection and profiling, and SHOW VIEW to read view definitions. Without SHOW VIEW, the ingestion will see the view as an object but will not be able to properly read its definition and build links with other objects.

To display info on procedures stored in the catalog, the user needs access to procedure metadata. Database-level access is usually enough; however, Yandex Managed Service for MySQL® may limit permissions by its service-level policies. Check that the user can see the procedures:

SHOW PROCEDURE STATUS WHERE Db = '<DB_name>';

ClickHouse®ClickHouse®

Set up a backend for ingestion of ClickHouse® source metadata:

1. Create a dedicated technical user for the ingestion:

   CREATE USER data_catalog_reader IDENTIFIED BY '<password>';
   

   Specify this user in the source connection settings.

2. Grant the user access to relevant databases:

   GRANT SELECT ON <DB_name>.* TO data_catalog_reader;
   

3. For metadata reads, grant access to system tables:

   GRANT SELECT ON system.databases TO data_catalog_reader;
   GRANT SELECT ON system.tables TO data_catalog_reader;
   GRANT SELECT ON system.columns TO data_catalog_reader;
   

4. If ClickHouse® system dictionaries are used, grant access to them as well:

   GRANT SELECT ON system.dictionaries TO data_catalog_reader;
   

If RBAC and ClickHouse® profiles are used, you can further restrict the user to read-only mode. This mode allows read commands, including SELECT, SHOW, DESCRIBE, and EXISTS, which are considered to be equivalent to reading from system tables.

OpenSearchOpenSearch

Set up a backend for ingestion of OpenSearch source metadata:

1. Create a role with metadata read permissions. Recommended index-level permissions:

   PUT /_plugins/_security/api/roles/read_indices_data_catalog
   
   {
      "index_permissions": [
         {
           "index_patterns": ["*"],
           "allowed_actions": ["read", "view_index_metadata"]
         }
      ]
   }
   

   For a stricter setup, we recommend limiting names to specific index templates, for example:

   PUT /_plugins/_security/api/roles/read_indices_data_catalog
   
   {
      "index_permissions": [
         {
           "index_patterns": ["prod-*", "analytics-*"],
           "allowed_actions": ["read", "view_index_metadata"]
         }
      ]
   }
   

2. Create a dedicated technical user for ingestions and assign the newly created role to this user:

   PUT /_plugins/_security/api/internalusers/data_catalog_reader
   
   {
     "password": "<password>",
     "opendistro_security_roles": ["read_indices_data_catalog"]
   }
   

   Specify this user in the source connection settings.

Greenplum®Greenplum®

Set up a backend for ingestion of Greenplum® source metadata:

1. Create a dedicated technical user:

   CREATE USER datacatalog_ingest WITH PASSWORD '<password>';
   

   Specify this user in the source connection settings.

2. Grant database access:

   GRANT CONNECT ON DATABASE <DB_name> TO datacatalog_ingest;
   

3. Grant access to relevant schemas:

   GRANT USAGE ON SCHEMA <schema_name> TO datacatalog_ingest;
   

4. Grant read access to tables:

   GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO datacatalog_ingest;
   

   If you need only the structure without profiling or object availability checks, the SELECT privilege for all tables is optional.

5. Grant the following read permissions for the newly created tables to be ingestible as well:

   ALTER DEFAULT PRIVILEGES IN SCHEMA <schema_name>
   GRANT SELECT ON TABLES TO datacatalog_ingest;
   

6. Additional permissions may be required to analyze the most frequent queries. Here is an example:

   GRANT pg_read_all_stats TO datacatalog_ingest;
   

   The pg_stat_statements extension may also be required.

   CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
   

   Warning

   The availability of pg_stat_statements and the pg_read_all_stats role depends on your Greenplum® version and cluster settings.

7. For Data Catalog to automatically display business descriptions, add comments to tables and columns. Here is an example:

   COMMENT ON TABLE mart.orders_daily IS 'Daily aggregated order metrics';
   COMMENT ON COLUMN mart.orders_daily.orders_count IS 'Number of orders per day';
   

Yandex StoreDoc/MongoDBYandex StoreDoc/MongoDB

Set up a backend for ingestion of Yandex StoreDoc/MongoDB source metadata:

1. Create a dedicated technical user:

   db.createUser({ user:"datacatalog_ingest", pwd:"<password>", roles:[]});
   

   Specify this user in the source connection settings.

2. For a standard export, grant the minimum necessary permissions to the technical user:

   • Read access to a limited list of databases (if exporting their metadata only):

     db.grantRolesToUser("datacatalog_ingest", [
     { role: "read", db: "<DB_name>" }
     ]);
     

   • The readAnyDatabase role (if exporting the metadata of all accessible databases):

     db.grantRolesToUser("datacatalog_ingest", [
     { role: "readAnyDatabase", db: "admin" }
     ]);
     

Overview of ingestion requirementsOverview of ingestion requirements

ClickHouse® is a registered trademark of ClickHouse, Inc.

Greenplum® and Greenplum Database® are registered trademarks or trademarks of Broadcom Inc. in the United States and/or other countries.