Connecting to a Yandex Object Storage bucket with a bucket policy

In Yandex Managed Service for Apache Airflow™, you can work with an Yandex Object Storage bucket with bucket policies configured. You access the bucket from a separate DNS zone through an internal load balancer used to distribute traffic among NAT instances. The described connection is shown below. An Apache Airflow™ cluster will be used in place of the test-s3-vm VM.

Getting startedGetting started

1. Create a network infrastructure to access the Object Storage bucket as shown on the picture above. For information on how to create such an infrastructure, see this tutorial.
2. Test the new infrastructure.
3. To connect to the bucket you created via Apache Airflow™, edit the bucket access policy. To do this, in the Action parameter, specify operations allowed for Apache Airflow™: "s3:GetObject, s3:ListBucket". After that, apply the changes using the terraform apply command.

Prepare the Apache Airflow™ clusterPrepare the Apache Airflow™ cluster

1. Create a service account named my-account with the vpc.user and managed-airflow.integrationProvider roles.
2. Grant the READ permission for the bucket you created earlier to the my-account service account.
3. Create an Apache Airflow™ cluster and specify the my-account service account in it.

Test the connectionTest the connection

To test the connection to the Object Storage bucket, upload the DAG file to the bucket The DAG should be displayed in the Apache Airflow™ web interface.