Uploading DAG files to a Managed Service for Apache Airflow™ cluster

Automation, data processing, and scheduled task execution are implemented using DAG files. DAG files are Python 3 scripts run inside Apache Airflow™. For a DAG file example, check this Apache Airflow™ tutorial.

You can import DAG files from:

• Yandex Object Storage bucket
• External Git repository

You can select the DAG file source type when creating or updating the cluster. This automatically delivers DAGs to your Managed Service for Apache Airflow™ cluster and makes them appear in the Apache Airflow™ web UI.

Importing DAG files from a bucketImporting DAG files from a bucket

1. Create a folder, e.g., dags, in the bucket and upload your DAG file to it. The system will automatically import the DAG file to the cluster.

   You can upload your DAG file to the bucket root, but it uploads quicker to a folder.

   If you need to upload additional scripts or modules used in the DAG to this folder, specify the full path to these scripts or modules in the bucket. Let’s say, you have uploaded all the files to the dags folder. If so, specify the following in from ... import of the DAG file:

   from dags.<file_name> import <object>
   

2. Open the Apache Airflow™ web interface.

3. Make sure that the new DAG has appeared in the DAGs section. It may take a few minutes to load a DAG file from the bucket.

Importing DAG files from a Git repositoryImporting DAG files from a Git repository

1. Upload your DAG file to the Git repository folder specified along with the repository address in the cluster settings. The system will automatically import the DAG file to the cluster.

   If you upload additional scripts or modules used in the DAG to this folder, specify the full path to these scripts or modules. Let’s say, you have uploaded all the files to the dags folder. If so, specify the following in from ... import of the DAG file:

   from dags.<file_name> import <object>
   

2. Open the Apache Airflow™ web interface.

3. Make sure that the new DAG has appeared in the DAGs section. It may take a few minutes to import your DAG file from the Git repository.