Connecting to a Yandex Object Storage bucket with a bucket policy

In Yandex Managed Service for Apache Airflow™, you can work with a Yandex Object Storage bucket with bucket policies configured. You access the bucket from a separate DNS zone through an internal load balancer that distributes traffic across NAT instances. This connection is outlined below. An Apache Airflow™ cluster will be used in place of the test-s3-vm VM.

Getting startedGetting started

1. Create a network infrastructure to access the Object Storage bucket as shown in the picture above. Learn how to create such an infrastructure in this tutorial.
2. Test the new infrastructure.
3. To connect to the bucket you created via Apache Airflow™, edit the bucket access policy. To do this, in the Action parameter, specify operations allowed for Apache Airflow™: "s3:GetObject, s3:ListBucket". After that, apply the changes using the terraform apply command.

Prepare the Apache Airflow™ clusterPrepare the Apache Airflow™ cluster

1. Create a service account named my-account with the vpc.user and managed-airflow.integrationProvider roles.
2. Grant the READ permission for the bucket you created earlier to my-account.
3. Create an Apache Airflow™ cluster and specify my-account in it.

Test the connectionTest the connection

To test the connection to the Object Storage bucket, upload the DAG file to the bucket. The DAG should appear in the Apache Airflow™ web interface.