Uploading DAG files to a Managed Service for Apache Airflow™ cluster

Automation, data processing, and scheduled task execution are implemented using DAG files. DAG files are Python 3 scripts run inside Apache Airflow™. For a DAG file example, check this Apache Airflow™ tutorial.

DAG files are uploaded to the Yandex Object Storage bucket specified when creating a cluster. This automatically delivers DAGs to a Managed Service for Apache Airflow™ cluster and makes them appear in the Apache Airflow™ web interface.

To upload DAG files to a cluster:

1. Create a folder, e.g., dags, in the bucket and upload a DAG file to it. This will upload your DAG to the cluster.

   You can upload your DAG file to the bucket root, but it uploads quicker to a folder.

   If you need to upload additional scripts or modules used in the DAG to this folder, specify the full path to these scripts or modules in the bucket. Let’s say, you have uploaded all the files to the dags folder. If so, specify the following in from ... import of the DAG file:

   from dags.<file_name> import <object>
   

2. Open the Apache Airflow™ web interface.

3. Make sure that the new DAG has appeared in the DAGs section. It may take a few minutes to load a DAG file from the bucket.