Uploading DAG files to a Managed Service for Apache Airflow™ cluster

A Managed Service for Apache Airflow™ cluster automatically loads DAG files from the configured source, such as the following:

• Yandex Object Storage bucket
• External Git repository

Prepare a DAG file. You can find an example in this Apache Airflow™ guide. If a DAG uses additional scripts or modules, configure their import.

Importing DAG files from a bucketImporting DAG files from a bucket

When creating or updating a cluster, select a bucket as the source of DAG files and configure the bucket access permissions for the cluster’s service account.

1. Create a folder, e.g., dags, in the selected bucket and upload your DAG file to it. If the DAG uses additional scripts or modules, upload them along with the DAG.

   You can upload your DAG file to the bucket root, but it uploads quicker to a folder.

2. Open the Apache Airflow™ web interface.

3. Make sure that the new DAG has appeared in the DAGs section. It may take a few minutes to load a DAG file from the bucket.

Importing DAG files from a Git repositoryImporting DAG files from a Git repository

When creating or updating a cluster, select a Git repository as the source of DAG files. Specify the repository URL, branch, DAG files directory, and authentication parameters.

1. Add the DAG file to the Git repository directory specified in the cluster settings. If the DAG uses additional scripts or modules, add them along with the DAG.
2. Push your changes to the remote branch specified in the cluster settings. If you are working with a local repository copy, create a commit and push it to that branch.
3. Open the Apache Airflow™ web interface.
4. Make sure that the new DAG has appeared in the DAGs section. It may take a few minutes to import your DAG file from the Git repository.

Using additional scripts and modulesUsing additional scripts and modules

If a DAG uses additional scripts or modules, upload them along with the DAG file to the bucket or Git repository. In the DAG file, specify the full path to these scripts or modules. Let’s say, you have uploaded all files to the dags folder. In this case, specify the following in the from ... import construct:

from dags.<file_name> import <object>