Uploading DAG files to a Managed Service for Apache Airflow™ cluster

Automation, data processing, and scheduled task execution are implemented using DAG files. DAG files are Python 3.8 scripts run inside Apache Airflow™. For a DAG file example, see the Apache Airflow™ documentation.

DAG files are uploaded to the Yandex Object Storage bucket specified when creating the cluster. This automatically delivers DAG task graphs into a Managed Service for Apache Airflow™ cluster and makes them appear in the Apache Airflow™ web interface.

To upload DAG files into a cluster:

1. Create a folder, e.g., dags, in the bucket and upload a DAG file to it. This will upload DAG into the cluster.

   You can upload your DAG file to the bucket root, but it uploads quicker to a folder.

   If uploading additional scripts or modules used in DAG to this folder, specify the full path to these scripts or modules in the bucket. For example, you have uploaded all the files to the dags folder. If so, specify the following in from ... import of the DAG file:

   from dags.<file_name> import <object>
   

2. Open the Apache Airflow™ web interface.

3. Make sure that the new DAG file has appeared in the DAGs section. It may take a few minutes to upload a DAG file from the bucket.