Updating Apache Airflow™ cluster settings

To update Apache Airflow™ cluster settings, use the AirflowCluster resource.

Using the CLIUsing the CLI

1. Export the current configuration:

   kubectl get airflowcluster <cluster_name> \
     -n <namespace> \
     -o yaml > airflow-cluster.yaml
   

2. Update the spec fields and remove the status, metadata.resourceVersion, metadata.uid, metadata.creationTimestamp, and metadata.managedFields system fields from the file.

3. Apply the configuration:

   kubectl apply -f airflow-cluster.yaml
   

4. Wait for the update to complete:

   kubectl get airflowcluster <cluster_name> \
     -n <namespace> \
     -w
   

   The status.phase field must become Ready, or Paused if the cluster is paused.

What you can updateWhat you can update

Without re-creating the cluster, you can:

• Select a different supported version in spec.version.
• Change the number of replicas or change component resources.
• Modify spec.config in line with applicable constraints.
• Enable or disable public access via spec.publicAccess.
• Update additional packages and their repositories in spec.dependencies.
• Update the Git repository ref and the secret with credentials.
• Enable or disable remote logs by adding or removing spec.taskLogs.
• Pause or resume the cluster via spec.pause.
• Enable or disable deletion protection via spec.deletionProtection.

What you cannot updateWhat you cannot update

After creation, you cannot update:

• DAG source type, Git repository URL, and subPath.
• Object storage type, bucket name, endpoint, subPath, and references to managed storage resources.

To update these parameters, create a new AirflowCluster and perform an individual data migration.

The metadata database is created automatically. You cannot select a different database or update any parameters of the existing one.

Component resourcesComponent resources

The examples below show fragments of spec. Add them into your current configuration, keeping the other parameters unchanged.

Example of updating the replica counts and resources:

spec:
  components:
    scheduler:
      replicas: 2
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: "2"
          memory: 4Gi
    dagProcessor:
      replicas: 2

If the resources field is set, specify all four values: requests.cpu, requests.memory, limits.cpu, and limits.memory.

requests sets the guaranteed resources for a single pod, and limits, the maximum resources. The cpu field defines the number of CPU cores, and memory, the amount of RAM.

spec.components supports scheduler, apiServer, triggerer, and dagProcessor. Each of them accepts both replicas and resources. Pod placement rules are not configurable.

Apache Airflow™ parametersApache Airflow™ parameters

The spec.config field defines settings for airflow.cfg, where the first level represents the section, and the second defines the parameter name. For example, to disable loading demo DAG files:

spec:
  config:
    core:
      load_examples: "False"

All values must be single-line strings, including numbers and booleans. Use literal values without template expressions. Section and parameter names must start with a Latin letter, digit, or underscore and contain only Latin letters, digits, underscores, dots, and hyphens. The DEFAULT section is reserved.

The following parameters cannot be updated via spec.config. This restriction applies regardless of letter case:

Additional dependenciesAdditional dependencies

Here is an example of adding Python and Debian packages:

spec:
  dependencies:
    pip:
      - name: apache-airflow-providers-amazon
    deb:
      - name: libpq5

For a Python package, you can specify version with a version constraint, e.g., ">=2.0,<3.0"; for a Debian package, only name is specified. Installing packages does not change the Apache Airflow™ version specified in spec.version: incompatible dependencies cause an installation error.

After updating packages, repositories, or their associated secrets, wait for the cluster update to complete. The DependenciesReady=True condition for the current configuration confirms that the dependencies are ready. If package installation fails, refer to our troubleshooting guide.

Package repositoriesPackage repositories

If repositories is not set, the default pip and APT settings are used. For your own sources, set spec.dependencies.repositories:

spec:
  dependencies:
    pip:
      - name: apache-airflow-providers-amazon
    deb:
      - name: libpq5
    repositories:
      pip:
        index:
          url: https://pypi.example.org/simple
          credentialsSecretRef:
            name: airflow-pypi-auth
          caSecretRef:
            name: airflow-repository-ca
            key: ca.crt
        extraIndexes:
          - url: https://pypi.org/simple
      deb:
        url: https://apt.example.org/debian
        suite: bookworm
        components: [main]
        credentialsSecretRef:
          name: airflow-apt-auth
        signingKeySecretRef:
          name: airflow-apt-key
          key: archive.asc
        caSecretRef:
          name: airflow-repository-ca
          key: ca.crt

Set up the specified secrets within the AirflowCluster namespace:

• credentialsSecretRef: Non-empty username and password keys. Use only printable ASCII characters without spaces. For a repository without authentication, the reference can be omitted.
• caSecretRef: Key with CA certificates in PEM format. Provide the reference if the repository uses a custom certificate authority.
• signingKeySecretRef: Secret record with the APT repository signing key. This reference is mandatory for repositories.deb.

Repository URLs must use HTTPS and must not contain credentials, query parameters, or fragments. The pip URL must end with /simple. One main and up to 16 additional pip indexes are supported; for indexes sharing the same host, use the same credentialsSecretRef. For APT, one repository is supported.

The repositories.pip section is valid only with a non-empty pip list, and repositories.deb, only with a non-empty deb list. In an installation without internet access, replace all addresses in the example, including extraIndexes, with those available within your network.