Reading a file from Yandex Object Storage

Use a directed acyclic graph (DAG) to read files from Yandex Object Storage of the service account linked to an Apache Airflow™ cluster.

Prepare the DAG file and run the graphPrepare the DAG file and run the graph

1. Create a local file named read_file_from_dags_bucket.py and copy the following script to it:

   from airflow.settings import DAGS_FOLDER
   from airflow.decorators import dag, task
   
   
   @dag(schedule=None)
   def read_file_from_dags_bucket():
       @task
       def read_file():
           with open(f'{DAGS_FOLDER}/data/airflow.txt') as file:
               content = file.read()
               print(f"file content: {content}")
   
       read_file()
   
   
   read_file_from_dags_bucket()
   

2. Upload the read_file_from_dags_bucket.py DAG file to the bucket you created earlier. This will automatically create a graph with the same name in the Apache Airflow™ web interface.

3. Open the Apache Airflow™ web interface.

4. Make sure a new graph named read_file_from_dags_bucket has appeared in the DAGs section.

   It may take a few minutes to upload a DAG file from the bucket.

5. To run the graph, click in the line with its name.

Check the resultCheck the result

To check the result in the Apache Airflow™ web interface:

1. In the DAGs section, open the read_file_from_dags_bucket graph.
2. Go to the Graph section.
3. Select the read_file job.
4. Go to Logs.
5. Make sure the logs contain the file content: {content} string, where content is the file content. This means the query was successful.