General questions about Managed Service for Apache Airflow™

How do I fix the IP address range overlap error when creating a cluster?How do I fix the IP address range overlap error when creating a cluster?

When creating a Managed Service for Apache Airflow™ cluster you may get the following error:

user subnet overlaps with service network range 10.248.0.0/13, see documentation for details

This error means that when creating the cluster, you selected a subnet whose IP address range overlaps with the 10.248.0.0/13 address range of the service subnet. In this subnet, Yandex Cloud manages the Managed Service for Apache Airflow™ cluster components.

To fix this error, select another subnet whose IP address range does not overlap with the service subnet range. Learn more about cluster subnet requirements in Network.

How can I fix the no permission error when attaching a service account to a cluster?How can I fix the no permission error when attaching a service account to a cluster?

Error message:

ERROR: rpc error: code = PermissionDenied desc = you do not have permission to access the requested service account or service account does not exist

This error occurs if you link a service account to a cluster while creating or modifying it.

Solution
Assign the iam.serviceAccounts.user role or higher to your Yandex Cloud account.

How to fix the error?How to fix the No module named 'airflow.providers.postgres.operators' error?

When accessing a Managed Service for PostgreSQL cluster from an Managed Service for Apache Airflow™ cluster, you may get this error:

Broken DAG: [/opt/airflow/dags/postgre.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/airflow/dags/postgre.py", line 2, in <module>
    from airflow.providers.postgres.operators.postgres import PostgresOperator
ModuleNotFoundError: No module named 'airflow.providers.postgres.operators'

apache-airflow-providers-postgres 5.13.1 is used by default. The error may occur if using PostgresOperator with newer provider versions.

Solution:

If using a newer provider version, use SQLExecuteQueryOperator instead of PostgresOperator. For more information, see this article.

How to fix the error?How to fix the AirflowException: Unknown hook type "postgres" error?

When accessing a Managed Service for PostgreSQL cluster from an Managed Service for Apache Airflow™ cluster, you may get this error:

Task failed with exception: source="task"
AirflowException: Unknown hook type "postgres"

By default, the apache-airflow-providers-postgres provider is not installed in Managed Service for Apache Airflow™ clusters with Apache Airflow™ versions above 3.0.

Solution:

When creating or updating a Managed Service for Apache Airflow™ cluster, add the apache-airflow-providers-postgres pip package under Dependencies.

How to fix the error?How to fix the No module named 'airflow_clickhouse_plugin' error?

When accessing a Managed Service for ClickHouse® cluster from an Managed Service for Apache Airflow™ cluster, you may get this error:

Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/airflow/dags/dags/clickhouse.py", line 2, in <module>
    from airflow_clickhouse_plugin.hooks.clickhouse import ClickHouseHook
ModuleNotFoundError: No module named 'airflow_clickhouse_plugin'

By default, the airflow-clickhouse-plugin plugin is not installed in Managed Service for Apache Airflow™.

Solution:

When creating or updating a Managed Service for Apache Airflow™ cluster, add the airflow-clickhouse-plugin pip package under Dependencies.

How to fix the error?How to fix the SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] error?

When trying to connect to a Managed Service for ClickHouse® cluster from a Managed Service for Apache Airflow™ cluster, you may get this error:

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

Solution:

1. Download the SSL certificate at https://storage.yandexcloud.net/cloud-certs/CA.pem.

2. Place the certificate in the root of the DAG file bucket.

3. In the connection settings, specify the certificate path in the ca_certs field.

   Value of the secret you created:

   {
     "conn_type": "clickhouse",
     "host": "<ClickHouse®_cluster_host_FQDN>",
     "port": 9440,
     "schema": "default-bd",
     "login": "admin",
     "password": "admin-password",
     "extra": {
         "secure": "True",
         "ca_certs": "/opt/airflow/dags/CA.pem"
     }
   }