DAG file sources
A Managed Service for Apache Airflow™ cluster automatically loads DAG files from an external source, such as a Yandex Object Storage bucket or Git repository. A single source is selected for a cluster. Its type and connection settings are specified when creating or updating the cluster.
Object Storage bucket
The bucket stores DAG files as well as additional scripts and modules used in DAG. The files may reside in the bucket root or directory. The cluster automatically downloads them from the selected bucket.
Git repository
For connecting to a Git repository, its address, branch name, and path to DAG file folder are specified. The cluster downloads files from the specified folder in the selected branch.
Warning
To use a Git repository, configure internet access through a NAT gateway from the Managed Service for Apache Airflow™ cluster network. When configuring it, link a NAT gateway route table to all the cluster subnets.
Authentication methods
The authentication method sets the repository address format and required credentials:
Connections over SSH use an address in git@<host>:<path_to_repository>.git format, such as git@github.com:<username>/<repository_name>.git, or a private SSH key for repository access.
When connecting via SSH, the RSA, DSA, ECDSA, Ed25519 private keys of the PKCS#1, PKCS#8, OpenSSL, and OpenSSH formats are supported. The private key must not be password-protected.
Connections over HTTP(S) use an address in https://<host>/<path_to_repository>.git or http://<host>/<path_to_repository>.git format, such as https://github.com/<username>/<repository_name>.git, as well as a username and password pair.
Instead of a password, you can provide an access token if the service the repository resides in supports such authentication. The username as well as its password or token cannot be empty. Credentials are specified in separate connection settings, not in the repository address.
Token authentication
An access token is provided in the password setting when connecting over HTTPS. The token is created in the service the repository resides in and must grant read permissions for the repository with DAG files. The token allows you to restrict access permissions granted by account password.
The username depends on the service and the token type. See below for examples and links to guides for services:
|
Service |
Token type |
Username |
|
GitHub |
GitHub username. The username cannot be empty, despite the fact that the token is used for authentication. |
|
|
GitLab |
Any non-empty string, e.g., GitLab username or |
|
|
GitLab |
Name specified when the token was created. The default format is |
|
|
GitLab |
|
|
|
Bitbucket Cloud |
Case-sensitive Bitbucket username or |
|
|
Bitbucket Cloud |
Access token of a repository or project. |
|
|
Azure DevOps |
Any non-empty string, e.g., |
If the token expired or was revoked, create a new one and update the connection parameters in the cluster settings.
Use cases
- Automating Yandex Data Processing operations
- Automating Yandex Query tasks with Yandex Managed Service for Apache Airflow™