Creating an Apache Airflow™ cluster

The Apache Airflow™ cluster is created by the AirflowCluster resource in the project namespace. Before creating the cluster, the installation administrator must enable Managed Service for Apache Airflow™.

The metadata database is created automatically. No database connection configuration is required.

Getting startedGetting started

Set up the following:

• Project namespace.
• Git repository with DAG files or S3-compatible storage.
• S3-compatible storage if remote task logs are required.
• Secret containing credentials if a private Git repository or external S3 storage is used.

Create the Secret and ConfigMap referenced by the configuration in the same namespace as AirflowCluster.

The Object Storage component must be enabled for automatically created buckets.

Check available Apache Airflow™ versions:

kubectl get airflowimagecatalog airflow \
  -o jsonpath='{range .spec.images[*]}{.version}{"\n"}{end}'

If the command does not output versions, check the resource structure: kubectl get airflowimagecatalog airflow -o yaml.

Using the CLIUsing the CLI

1. Create an AirflowCluster resource file, e.g., airflow-cluster.yaml and insert one of the following configurations:

   Git and local logs

   Object Storage for DAG and logs

   This example uses a public Git repository and local task logs.

   apiVersion: airflow.stackland.yandex.cloud/v1alpha1
   kind: AirflowCluster
   metadata:
     name: airflow-demo
   spec:
     version: "3.1.8"
     dags:
       source: git
       git:
         repo: https://github.com/example/airflow-dags.git
         ref: main
     publicAccess: true
     deletionProtection: true
   

   This example creates separate buckets for DAG files and task logs. DAG files are loaded from the dags/ prefix in the bucket.

   apiVersion: airflow.stackland.yandex.cloud/v1alpha1
   kind: AirflowCluster
   metadata:
     name: airflow-demo
   spec:
     version: "3.1.8"
     dags:
       source: s3
       s3:
         type: stackland-storage
         stacklandStorage:
           subPath: dags/
     taskLogs:
       type: stackland-storage
       stacklandStorage:
         subPath: logs/
     publicAccess: true
     deletionProtection: true
   

2. Specify the cluster name in metadata.name and the available version in spec.version. For the Git source, replace the repository URL and the ref value with your own.

   Both examples include public access to the web interface (spec.publicAccess: true) and deletion protection (spec.deletionProtection: true). Change these parameters if necessary.

3. Apply the manifest:

   kubectl apply -f airflow-cluster.yaml -n <namespace>
   

   You can also specify the namespace in the manifest metadata.namespace and omit the -n parameter from the command.

4. Substitute the cluster name from metadata.name and wait for the Ready state:

   kubectl get airflowcluster <cluster_name> \
     -n <namespace> \
     -w
   

5. If the cluster remains in Provisioning for a long time or goes Failed, check the status conditions in status.conditions:

   kubectl describe airflowcluster <cluster_name> -n <namespace>
   

   Recommendations are provided in the Troubleshooting section.

Private Git repositoryPrivate Git repository

For HTTPS, create a Secret with fixed username and password keys:

kubectl create secret generic airflow-git-auth \
  -n <namespace> \
  --from-literal=username=<username> \
  --from-literal=password=<token_or_password>

Add a link to spec.dags.git.authSecretRef of the manifest:

spec:
  dags:
    source: git
    git:
      repo: https://git.example.com/team/airflow-dags.git
      ref: main
      authSecretRef:
        name: airflow-git-auth

For SSH, the Secret must contain ssh-privatekey, and the ConfigMap must contain the known_hosts key:

spec:
  dags:
    source: git
    git:
      repo: git@git.example.com:team/airflow-dags.git
      ref: main
      authSecretRef:
        name: airflow-git-ssh
      knownHostsConfigMapRef:
        name: airflow-git-known-hosts

The Git and S3 synchronization interval is 60 seconds and is not configurable.

External S3 storageExternal S3 storage

Create a Secret with access keys:

kubectl create secret generic airflow-s3-credentials \
  -n <namespace> \
  --from-literal=access-key-id=<key_ID> \
  --from-literal=secret-access-key=<secret_key>

For DAG files, specify the following in spec:

spec:
  dags:
    source: s3
    s3:
      type: s3
      s3:
        bucket: airflow-dags
        region: ru-central1
        endpointUrl: https://storage.yandexcloud.net
        credentialsSecretRef:
          name: airflow-s3-credentials
          accessKeyIdPath: access-key-id
          secretAccessKeyPath: secret-access-key
        subPath: dags/

For remote logs, set the taskLogs field in spec:

spec:
  taskLogs:
    type: s3
    s3:
      bucket: airflow-logs
      region: ru-central1
      endpointUrl: https://storage.yandexcloud.net
      credentialsSecretRef:
        name: airflow-s3-credentials
        accessKeyIdPath: access-key-id
        secretAccessKeyPath: secret-access-key
      subPath: logs/

Next stepsNext steps

• Get administrator credentials.
• Open the Apache Airflow™ web interface.
• Modify the cluster settings if you need to configure resources or dependencies.