Yandex Managed Service for Apache Airflow™
A service for managing the Apache Airflow™ data processing flow orchestrator in Yandex Cloud infrastructure.
One-click deployment
You can deploy Apache Airflow components in just minutes. The product settings have already been optimized for the selected cluster size.
Secure access
User authorization is carried out in Yandex Cloud infrastructure with an IAM role verification step. The service also supports integration with Yandex Lockbox, as well as with the Apache Airflow secret repository.
Automatic data processing
Process data and prepare results with familiar tools automatically, thanks to the integration of Yandex Cloud services with Yandex Managed Service for Apache Airflow.
Temporary Yandex Data Proc clusters
Use computing resources more efficiently by automatically creating a temporary Yandex Data Proc cluster which is deleted once data processing is completed.
Integration with other Yandex Cloud services
Apache Airflow clusters support integration with other Yandex Cloud services without no additional programming or configuration.
Monitoring
View audit logs in Yandex Cloud Logging. Set up alerts in Yandex Monitoring and track metrics such as task completion time and specific errors.
We handle most of the database maintenance
Independent control
Yandex Cloud control
Getting started
Getting started
Create a Yandex Managed Service for Apache Airflow cluster.
Get started in the Apache Airflow web interface.
FAQ
Apache Airflow has several features that make it a unique and powerful tool for automating tasks, planning and managing workflows (workflow orchestration). The main features that set Apache Airflow apart are:
- Support for a variety of data sources and plugins. Apache Airflow features more than 150 integrations with data storage and processing services, including Yandex Cloud services.
- Scalability. Apache Airflow supports the dynamic creation of computing resources to perform tasks, and is capable of adapting to current loads.
- Open source and an active community. Apache Airflow is an open-source project. The community of developers and users provides support and continuous updates for the tool.
- Dependency identification. Apache Airflow allows you to explicitly define dependencies between tasks, which gives you control over how they are completed. This is useful when orchestrating complex processes.
- Monitoring and logging. Apache Airflow provides tools for monitoring and logging tasks. You can easily monitor tasks’ status and progress, and analyze logs to identify errors and improve performance.
- Customizability. You can customize Apache Airflow to suit your needs by creating your own operators and expanding functionality with custom plugins.
Start using Yandex Managed Service for Apache Airflow
Helpful links
Apache® and Apache Airflow™ are registered trademarks or trademarks of the Apache Software Foundation in the U.S. and/or other countries.