Troubleshooting Apache Airflow™

Start diagnostics from the AirflowCluster status and status.conditions.

General diagnosticsGeneral diagnostics

1. Review the cluster state:

   kubectl get airflowcluster <cluster_name> \
     -n <namespace> \
     -o wide
   

2. Get the status conditions and cluster events:

   kubectl describe airflowcluster <cluster_name> \
     -n <namespace>
   

3. Get the cluster UID to search for its resources:

   kubectl get airflowcluster <cluster_name> \
     -n <namespace> \
     -o jsonpath='{.metadata.uid}'
   

4. Check Apache Airflow™ pods by providing the obtained UID below:

   kubectl get pods -n <namespace> \
     -l 'tier=airflow,airflow.stackland.yandex.cloud/owner-uid=<cluster_UID>'
   

Cluster remains in provisioningCluster remains in provisioning

Check the status conditions whose status field has the False value:

For an exact reason, see the reason and message fields of the relevant condition. Make sure that its observedGeneration matches the current metadata.generation.

If you were unable to fix the error by following the instructions below, send your installation administrator the cluster name, namespace, and the reason and message values.

Git errorsGit errors

For a private HTTPS repository, a secret must contain non-empty username and password keys. The following is required for SSH:

• ssh-privatekey in the secret from authSecretRef.
• known_hosts key in ConfigMap from knownHostsConfigMapRef.

Key names are fixed. Also, check the repository URL, existence of the branch, tag, or commit from ref, and the subPath path.

For errors during DAG file uploads, see the git-sync container log:

kubectl get pods -n <namespace> \
  -l 'tier=airflow,component=dag-processor,airflow.stackland.yandex.cloud/owner-uid=<cluster_UID>'
kubectl logs -n <namespace> <pod_name> -c git-sync

Errors during DAG S3 synchronizationErrors during DAG S3 synchronization

If DAG files from S3 do not appear in the web UI, check:

• Bucket name, endpoint, region, and forcePathStyle mode.
• Keys specified in credentialsSecretRef and access permissions for the bucket.
• CA certificate for the endpoint with a private certification authority.
• Existence of DAG files under a prefix from subPath.
• Network access to S3 from the namespace.

View the synchronization log:

kubectl get pods -n <namespace> \
  -l 'tier=airflow,component=dag-processor,airflow.stackland.yandex.cloud/owner-uid=<cluster_UID>'
kubectl logs -n <namespace> <pod_name> -c dag-sync

If pods remain Pending, review their events:

kubectl describe pod <pod_name> -n <namespace>

Keep in mind the S3 source limitations. If pods are not running due to a lack of resources or storage unavailability, send the events to your installation administrator.

The password from the secret does not workThe password from the secret does not work

Find the administrative secret by cluster UID. If the username key has been changed, restore the original username: you cannot update the administrator name.

After updating the password key, wait until the secret is processed and check status.conditions for InitialUserReady. If an error occurs, send the reason and message values to your installation administrator.

If the cluster has switched back to Ready but you cannot log in with the new password, the existing user's password could remain unchanged. To restore access, manually set the password from the secret in Apache Airflow™. To do this, you need permissions for running commands in pods (pods/exec). In case you do not have such permissions, contact your installation administrator.

1. If the cluster is paused, resume it. Find a running API server pod by providing the UID from the Find secret guide in the command below:

   kubectl get pods -n <namespace> \
     -l 'tier=airflow,component=api-server,airflow.stackland.yandex.cloud/owner-uid=<cluster_UID>'
   

2. Run the password change command in the pod you found:

   kubectl exec -it <pod_name> \
     -n <namespace> \
     -c api-server -- \
     airflow users reset-password --username <username>
   

   Use the username from the username key of the administrative secret. In the interactive request, enter the password from the password key and confirm it.

3. Log out of the web UI, then log in anew with the password from the secret.

Database errorsDatabase errors

The metadata database is created and configured automatically. If DatabaseReady=False has persisted over a long period of time or Apache Airflow™ throws a database connection error, contact your installation administrator. Provide the cluster name, namespace, and error message.

Package installation errorsPackage installation errors

If DependenciesReady=False, start with reason and message:

• SecretNotFound, InvalidSecret, or InvalidCABundle: Check the secret from spec.dependencies.repositories (credentials, CA, and APT signature key). See the requirements in the Package repositories section.
• ImageBuildPending or ImageReplicationPending: Dependencies are still being prepared for use. Wait until the update is complete. If the status remains unchanged for a long time, contact your installation administrator.
• ImageBuildFailed: Check the package names and versions, repository availability, and the compatibility of dependencies with the Apache Airflow™ version from spec.version.

Fix spec.dependencies or the secret you are using. If the reason is unclear based on message, forward this message to your installation administrator.

In case of ImagePullBackOff in pod events, you should also contact your installation administrator.

Web UI is not publishedWeb UI is not published

Check spec.publicAccess and the cluster status. If spec.publicAccess: false, use internal access or update the setting.

When spec.publicAccess: true, wait for the Ready status and get the address from status.apiServerURL using the Open the web UI guide. If the address does not appear or is unavailable, provide the cluster name, namespace, and error message to your installation administrator.

Deletion is not completedDeletion is not completed

Cluster deletion includes deleting its data and automatically created buckets and may take time. If the Deleting status remains unchanged for a long time, provide the kubectl describe airflowcluster <cluster_name> -n <namespace> command output to your installation administrator.

Do not disable Managed Service for Apache Airflow™ until all clusters are deleted completely.