Managed Service for Apache Airflow™ API, gRPC: ClusterService
A set of methods for managing Apache Airflow Cluster resources.
Call | Description |
---|---|
Get | Returns the specified Apache Airflow Cluster resource. |
List | Retrieves a list of Apache Airflow Cluster resources. |
Create | Creates an Apache Airflow cluster. |
Update | Updates the specified Apache Airflow cluster. |
Delete | Deletes the specified Apache Airflow cluster. |
Start | Starts the specified Apache Airflow cluster. |
Stop | Stops the specified Apache Airflow cluster. |
ListOperations | Retrieves the list of Operation resources for the specified cluster. |
Calls ClusterService
Get
Returns the specified Apache Airflow Cluster resource.
rpc Get (GetClusterRequest) returns (Cluster)
GetClusterRequest
Field | Description |
---|---|
cluster_id | string Required. ID of the Apache Airflow Cluster resource to return. The maximum string length in characters is 50. |
Cluster
Field | Description |
---|---|
id | string Unique ID of the Apache Airflow cluster. This ID is assigned by Cloud during cluster creation. |
folder_id | string ID of the folder that the Apache Airflow cluster belongs to. |
created_at | google.protobuf.Timestamp The time when the Apache Airflow cluster was created. |
name | string Name of the Apache Airflow cluster. The name is unique within the folder. 1-64 characters long. |
description | string Description of the Apache Airflow cluster. 0-256 characters long. |
labels | map<string,string> Resource labels as key:value pairs. Maximum of 64 per resource. |
monitoring[] | Monitoring Monitoring systems relevant to the Apache Airflow cluster. |
config | ClusterConfig Configuration of Apache Airflow components. |
health | enum Health Aggregated cluster health.
|
status | enum Status Cluster status.
|
network | NetworkConfig Network related configuration options. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster. |
webserver_url | string Address of Apache Airflow web UI. |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
Monitoring
Field | Description |
---|---|
name | string Name of the monitoring system. |
description | string Description of the monitoring system. |
link | string Link to the monitoring system. |
ClusterConfig
Field | Description |
---|---|
version_id | string Version of Apache that runs on the cluster. |
airflow | AirflowConfig Configuration of the Apache Airflow application itself. |
webserver | WebserverConfig Required. Configuration of webserver instances. |
scheduler | SchedulerConfig Required. Configuration of scheduler instances. |
triggerer | TriggererConfig Configuration of triggerer instances. |
worker | WorkerConfig Required. Configuration of worker instances. |
dependencies | Dependencies The list of additional packages installed in the cluster. |
lockbox | LockboxConfig Configuration of Lockbox Secret Backend. |
AirflowConfig
Field | Description |
---|---|
config | map<string,string> Properties to be passed to Apache Airflow configuration file. |
WebserverConfig
Field | Description |
---|---|
count | int64 The number of webserver instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to webserver instances. |
Resources
Field | Description |
---|---|
resource_preset_id | string ID of the preset for computational resources available to an instance (CPU, memory etc.). |
SchedulerConfig
Field | Description |
---|---|
count | int64 The number of scheduler instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to scheduler instances. |
TriggererConfig
Field | Description |
---|---|
count | int64 The number of triggerer instances in the cluster. Acceptable values are 0 to 512, inclusive. |
resources | Resources Resources allocated to triggerer instances. |
WorkerConfig
Field | Description |
---|---|
min_count | int64 The minimum number of worker instances in the cluster. Acceptable values are 0 to 512, inclusive. |
max_count | int64 The maximum number of worker instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to worker instances. |
Dependencies
Field | Description |
---|---|
pip_packages[] | string Python packages that are installed in the cluster. |
deb_packages[] | string System packages that are installed in the cluster. |
LockboxConfig
Field | Description |
---|---|
enabled | bool The setting allows to enable Lockbox Secret Backend. |
NetworkConfig
Field | Description |
---|---|
subnet_ids[] | string IDs of VPC network subnets where instances of the cluster are attached. |
security_group_ids[] | string User security groups. |
CodeSyncConfig
Field | Description |
---|---|
source | oneof: s3 |
s3 | S3Config |
S3Config
Field | Description |
---|---|
bucket | string The name of the Object Storage bucket that stores DAG files used in the cluster. |
LoggingConfig
Field | Description |
---|---|
enabled | bool Logs generated by the Airflow components are delivered to Cloud Logging. |
destination | oneof: folder_id or log_group_id Destination of log records. |
folder_id | string Logs should be written to default log group for specified folder. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
log_group_id | string Logs should be written to log group resolved by ID. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
min_level | yandex.cloud.logging.v1.LogLevel.Level Minimum log entry level. See [LogLevel.Level] for details. |
List
Retrieves a list of Apache Airflow Cluster resources.
rpc List (ListClustersRequest) returns (ListClustersResponse)
ListClustersRequest
Field | Description |
---|---|
folder_id | string Required. ID of the folder to list Apache Airflow clusters in. The maximum string length in characters is 50. |
page_size | int64 The maximum number of results per page to return. If the number of available results is larger than page_size , the service returns a ListClustersResponse.next_page_token that can be used to get the next page of results in subsequent list requests. The maximum value is 1000. |
page_token | string Page token. To get the next page of results, set page_token to the ListClustersResponse.next_page_token returned by the previous list request. The maximum string length in characters is 100. |
filter | string A filter expression that filters resources listed in the response. The expression must specify:
|
ListClustersResponse
Field | Description |
---|---|
clusters[] | Cluster List of Apache Airflow Cluster resources. |
next_page_token | string This token allows you to get the next page of results for list requests. If the number of results is larger than ListClustersRequest.page_size, use the next_page_token as the value for the ListClustersRequest.page_token parameter in the next list request. Each subsequent list request will have its own next_page_token to continue paging through the results. |
Cluster
Field | Description |
---|---|
id | string Unique ID of the Apache Airflow cluster. This ID is assigned by Cloud during cluster creation. |
folder_id | string ID of the folder that the Apache Airflow cluster belongs to. |
created_at | google.protobuf.Timestamp The time when the Apache Airflow cluster was created. |
name | string Name of the Apache Airflow cluster. The name is unique within the folder. 1-64 characters long. |
description | string Description of the Apache Airflow cluster. 0-256 characters long. |
labels | map<string,string> Resource labels as key:value pairs. Maximum of 64 per resource. |
monitoring[] | Monitoring Monitoring systems relevant to the Apache Airflow cluster. |
config | ClusterConfig Configuration of Apache Airflow components. |
health | enum Health Aggregated cluster health.
|
status | enum Status Cluster status.
|
network | NetworkConfig Network related configuration options. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster. |
webserver_url | string Address of Apache Airflow web UI. |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
Monitoring
Field | Description |
---|---|
name | string Name of the monitoring system. |
description | string Description of the monitoring system. |
link | string Link to the monitoring system. |
ClusterConfig
Field | Description |
---|---|
version_id | string Version of Apache that runs on the cluster. |
airflow | AirflowConfig Configuration of the Apache Airflow application itself. |
webserver | WebserverConfig Required. Configuration of webserver instances. |
scheduler | SchedulerConfig Required. Configuration of scheduler instances. |
triggerer | TriggererConfig Configuration of triggerer instances. |
worker | WorkerConfig Required. Configuration of worker instances. |
dependencies | Dependencies The list of additional packages installed in the cluster. |
lockbox | LockboxConfig Configuration of Lockbox Secret Backend. |
AirflowConfig
Field | Description |
---|---|
config | map<string,string> Properties to be passed to Apache Airflow configuration file. |
WebserverConfig
Field | Description |
---|---|
count | int64 The number of webserver instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to webserver instances. |
Resources
Field | Description |
---|---|
resource_preset_id | string ID of the preset for computational resources available to an instance (CPU, memory etc.). |
SchedulerConfig
Field | Description |
---|---|
count | int64 The number of scheduler instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to scheduler instances. |
TriggererConfig
Field | Description |
---|---|
count | int64 The number of triggerer instances in the cluster. Acceptable values are 0 to 512, inclusive. |
resources | Resources Resources allocated to triggerer instances. |
WorkerConfig
Field | Description |
---|---|
min_count | int64 The minimum number of worker instances in the cluster. Acceptable values are 0 to 512, inclusive. |
max_count | int64 The maximum number of worker instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to worker instances. |
Dependencies
Field | Description |
---|---|
pip_packages[] | string Python packages that are installed in the cluster. |
deb_packages[] | string System packages that are installed in the cluster. |
LockboxConfig
Field | Description |
---|---|
enabled | bool The setting allows to enable Lockbox Secret Backend. |
NetworkConfig
Field | Description |
---|---|
subnet_ids[] | string IDs of VPC network subnets where instances of the cluster are attached. |
security_group_ids[] | string User security groups. |
CodeSyncConfig
Field | Description |
---|---|
source | oneof: s3 |
s3 | S3Config |
S3Config
Field | Description |
---|---|
bucket | string The name of the Object Storage bucket that stores DAG files used in the cluster. |
LoggingConfig
Field | Description |
---|---|
enabled | bool Logs generated by the Airflow components are delivered to Cloud Logging. |
destination | oneof: folder_id or log_group_id Destination of log records. |
folder_id | string Logs should be written to default log group for specified folder. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
log_group_id | string Logs should be written to log group resolved by ID. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
min_level | yandex.cloud.logging.v1.LogLevel.Level Minimum log entry level. See [LogLevel.Level] for details. |
Create
Creates an Apache Airflow cluster.
rpc Create (CreateClusterRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:CreateClusterMetadata
Operation.response:Cluster
CreateClusterRequest
Field | Description |
---|---|
folder_id | string Required. ID of the folder to create Apache Airflow cluster in. The maximum string length in characters is 50. |
name | string Required. Name of the Apache Airflow cluster. The name must be unique within the folder. The maximum string length in characters is 63. Value must match the regular expression [a-zA-Z0-9_-]* . |
description | string Description of the Apache Airflow cluster. The maximum string length in characters is 256. |
labels | map<string,string> Custom labels for the Apache Airflow cluster as key:value pairs. For example, "env": "prod". No more than 64 per resource. The maximum string length in characters for each value is 63. Each value must match the regular expression [-_0-9a-z]* . The maximum string length in characters for each key is 63. Each key must match the regular expression [a-z][-_0-9a-z]* . |
config | ClusterConfig Required. Configuration of Apache Airflow components. |
network | NetworkConfig Network related configuration options. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster. |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
admin_password | string Required. Password of user admin . The string length in characters must be 8-128. |
ClusterConfig
Field | Description |
---|---|
version_id | string Version of Apache that runs on the cluster. |
airflow | AirflowConfig Configuration of the Apache Airflow application itself. |
webserver | WebserverConfig Required. Configuration of webserver instances. |
scheduler | SchedulerConfig Required. Configuration of scheduler instances. |
triggerer | TriggererConfig Configuration of triggerer instances. |
worker | WorkerConfig Required. Configuration of worker instances. |
dependencies | Dependencies The list of additional packages installed in the cluster. |
lockbox | LockboxConfig Configuration of Lockbox Secret Backend. |
AirflowConfig
Field | Description |
---|---|
config | map<string,string> Properties to be passed to Apache Airflow configuration file. |
WebserverConfig
Field | Description |
---|---|
count | int64 The number of webserver instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to webserver instances. |
Resources
Field | Description |
---|---|
resource_preset_id | string ID of the preset for computational resources available to an instance (CPU, memory etc.). |
SchedulerConfig
Field | Description |
---|---|
count | int64 The number of scheduler instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to scheduler instances. |
TriggererConfig
Field | Description |
---|---|
count | int64 The number of triggerer instances in the cluster. Acceptable values are 0 to 512, inclusive. |
resources | Resources Resources allocated to triggerer instances. |
WorkerConfig
Field | Description |
---|---|
min_count | int64 The minimum number of worker instances in the cluster. Acceptable values are 0 to 512, inclusive. |
max_count | int64 The maximum number of worker instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to worker instances. |
Dependencies
Field | Description |
---|---|
pip_packages[] | string Python packages that are installed in the cluster. |
deb_packages[] | string System packages that are installed in the cluster. |
LockboxConfig
Field | Description |
---|---|
enabled | bool The setting allows to enable Lockbox Secret Backend. |
NetworkConfig
Field | Description |
---|---|
subnet_ids[] | string IDs of VPC network subnets where instances of the cluster are attached. |
security_group_ids[] | string User security groups. |
CodeSyncConfig
Field | Description |
---|---|
source | oneof: s3 |
s3 | S3Config |
S3Config
Field | Description |
---|---|
bucket | string The name of the Object Storage bucket that stores DAG files used in the cluster. |
LoggingConfig
Field | Description |
---|---|
enabled | bool Logs generated by the Airflow components are delivered to Cloud Logging. |
destination | oneof: folder_id or log_group_id Destination of log records. |
folder_id | string Logs should be written to default log group for specified folder. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
log_group_id | string Logs should be written to log group resolved by ID. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
min_level | yandex.cloud.logging.v1.LogLevel.Level Minimum log entry level. See [LogLevel.Level] for details. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
CreateClusterMetadata
Field | Description |
---|---|
cluster_id | string ID of the Apache Airflow cluster that is being created. |
Cluster
Field | Description |
---|---|
id | string Unique ID of the Apache Airflow cluster. This ID is assigned by Cloud during cluster creation. |
folder_id | string ID of the folder that the Apache Airflow cluster belongs to. |
created_at | google.protobuf.Timestamp The time when the Apache Airflow cluster was created. |
name | string Name of the Apache Airflow cluster. The name is unique within the folder. 1-64 characters long. |
description | string Description of the Apache Airflow cluster. 0-256 characters long. |
labels | map<string,string> Resource labels as key:value pairs. Maximum of 64 per resource. |
monitoring[] | Monitoring Monitoring systems relevant to the Apache Airflow cluster. |
config | ClusterConfig Configuration of Apache Airflow components. |
health | enum Health Aggregated cluster health.
|
status | enum Status Cluster status.
|
network | NetworkConfig Network related configuration options. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster. |
webserver_url | string Address of Apache Airflow web UI. |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
Monitoring
Field | Description |
---|---|
name | string Name of the monitoring system. |
description | string Description of the monitoring system. |
link | string Link to the monitoring system. |
Update
Updates the specified Apache Airflow cluster.
rpc Update (UpdateClusterRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:UpdateClusterMetadata
Operation.response:Cluster
UpdateClusterRequest
Field | Description |
---|---|
cluster_id | string Required. ID of the Apache Airflow Cluster resource to update. The maximum string length in characters is 50. |
update_mask | google.protobuf.FieldMask Field mask that specifies which fields of the Apache Airflow Cluster resource should be updated. |
name | string New name of the cluster. Value must match the regular expression |[a-z][-a-z0-9]{1,61}[a-z0-9] . |
description | string New description of the Apache Airflow cluster. The maximum string length in characters is 256. |
labels | map<string,string> Custom labels for the Apache Airflow cluster as key:value pairs. For example, "env": "prod". The new set of labels will completely replace the old ones. To add a label, request the current set with the ClusterService.Get method, then send an ClusterService.Update request with the new label added to the set. No more than 64 per resource. The maximum string length in characters for each value is 63. Each value must match the regular expression [-_0-9a-z]* . The string length in characters for each key must be 1-63. Each key must match the regular expression [a-z][-_0-9a-z]* . |
config_spec | UpdateClusterConfigSpec Configuration of Apache Airflow components. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
network_spec | UpdateNetworkConfigSpec Network related configuration options. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
UpdateClusterConfigSpec
Field | Description |
---|---|
airflow | AirflowConfig Configuration of the Apache Airflow application itself. |
webserver | WebserverConfig Configuration of webserver instances. |
scheduler | SchedulerConfig Configuration of scheduler instances. |
triggerer | TriggererConfig Configuration of triggerer instances. |
worker | WorkerConfig Configuration of worker instances. |
dependencies | Dependencies The list of additional packages installed in the cluster. |
lockbox | LockboxConfig Configuration of Lockbox Secret Backend. |
AirflowConfig
Field | Description |
---|---|
config | map<string,string> Properties to be passed to Apache Airflow configuration file. |
WebserverConfig
Field | Description |
---|---|
count | int64 The number of webserver instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to webserver instances. |
Resources
Field | Description |
---|---|
resource_preset_id | string ID of the preset for computational resources available to an instance (CPU, memory etc.). |
SchedulerConfig
Field | Description |
---|---|
count | int64 The number of scheduler instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to scheduler instances. |
TriggererConfig
Field | Description |
---|---|
count | int64 The number of triggerer instances in the cluster. Acceptable values are 0 to 512, inclusive. |
resources | Resources Resources allocated to triggerer instances. |
WorkerConfig
Field | Description |
---|---|
min_count | int64 The minimum number of worker instances in the cluster. Acceptable values are 0 to 512, inclusive. |
max_count | int64 The maximum number of worker instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to worker instances. |
Dependencies
Field | Description |
---|---|
pip_packages[] | string Python packages that are installed in the cluster. |
deb_packages[] | string System packages that are installed in the cluster. |
LockboxConfig
Field | Description |
---|---|
enabled | bool The setting allows to enable Lockbox Secret Backend. |
CodeSyncConfig
Field | Description |
---|---|
source | oneof: s3 |
s3 | S3Config |
S3Config
Field | Description |
---|---|
bucket | string The name of the Object Storage bucket that stores DAG files used in the cluster. |
UpdateNetworkConfigSpec
Field | Description |
---|---|
security_group_ids[] | string User security groups. |
LoggingConfig
Field | Description |
---|---|
enabled | bool Logs generated by the Airflow components are delivered to Cloud Logging. |
destination | oneof: folder_id or log_group_id Destination of log records. |
folder_id | string Logs should be written to default log group for specified folder. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
log_group_id | string Logs should be written to log group resolved by ID. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
min_level | yandex.cloud.logging.v1.LogLevel.Level Minimum log entry level. See [LogLevel.Level] for details. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
UpdateClusterMetadata
Field | Description |
---|---|
cluster_id | string ID of the Apache Airflow Cluster resource that is being updated. |
Cluster
Field | Description |
---|---|
id | string Unique ID of the Apache Airflow cluster. This ID is assigned by Cloud during cluster creation. |
folder_id | string ID of the folder that the Apache Airflow cluster belongs to. |
created_at | google.protobuf.Timestamp The time when the Apache Airflow cluster was created. |
name | string Name of the Apache Airflow cluster. The name is unique within the folder. 1-64 characters long. |
description | string Description of the Apache Airflow cluster. 0-256 characters long. |
labels | map<string,string> Resource labels as key:value pairs. Maximum of 64 per resource. |
monitoring[] | Monitoring Monitoring systems relevant to the Apache Airflow cluster. |
config | ClusterConfig Configuration of Apache Airflow components. |
health | enum Health Aggregated cluster health.
|
status | enum Status Cluster status.
|
network | NetworkConfig Network related configuration options. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster. |
webserver_url | string Address of Apache Airflow web UI. |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
Monitoring
Field | Description |
---|---|
name | string Name of the monitoring system. |
description | string Description of the monitoring system. |
link | string Link to the monitoring system. |
ClusterConfig
Field | Description |
---|---|
version_id | string Version of Apache that runs on the cluster. |
airflow | AirflowConfig Configuration of the Apache Airflow application itself. |
webserver | WebserverConfig Required. Configuration of webserver instances. |
scheduler | SchedulerConfig Required. Configuration of scheduler instances. |
triggerer | TriggererConfig Configuration of triggerer instances. |
worker | WorkerConfig Required. Configuration of worker instances. |
dependencies | Dependencies The list of additional packages installed in the cluster. |
lockbox | LockboxConfig Configuration of Lockbox Secret Backend. |
NetworkConfig
Field | Description |
---|---|
subnet_ids[] | string IDs of VPC network subnets where instances of the cluster are attached. |
security_group_ids[] | string User security groups. |
Delete
Deletes the specified Apache Airflow cluster.
rpc Delete (DeleteClusterRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:DeleteClusterMetadata
Operation.response:google.protobuf.Empty
DeleteClusterRequest
Field | Description |
---|---|
cluster_id | string Required. ID of the Apache Airflow cluster to delete. The maximum string length in characters is 50. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
DeleteClusterMetadata
Field | Description |
---|---|
cluster_id | string ID of the Apache Airflow cluster that is being deleted. |
Start
Starts the specified Apache Airflow cluster.
rpc Start (StartClusterRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:StartClusterMetadata
Operation.response:Cluster
StartClusterRequest
Field | Description |
---|---|
cluster_id | string Required. ID of the Apache Airflow cluster to start. The maximum string length in characters is 50. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
StartClusterMetadata
Field | Description |
---|---|
cluster_id | string ID of the Apache Airflow cluster that is being started. |
Cluster
Field | Description |
---|---|
id | string Unique ID of the Apache Airflow cluster. This ID is assigned by Cloud during cluster creation. |
folder_id | string ID of the folder that the Apache Airflow cluster belongs to. |
created_at | google.protobuf.Timestamp The time when the Apache Airflow cluster was created. |
name | string Name of the Apache Airflow cluster. The name is unique within the folder. 1-64 characters long. |
description | string Description of the Apache Airflow cluster. 0-256 characters long. |
labels | map<string,string> Resource labels as key:value pairs. Maximum of 64 per resource. |
monitoring[] | Monitoring Monitoring systems relevant to the Apache Airflow cluster. |
config | ClusterConfig Configuration of Apache Airflow components. |
health | enum Health Aggregated cluster health.
|
status | enum Status Cluster status.
|
network | NetworkConfig Network related configuration options. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster. |
webserver_url | string Address of Apache Airflow web UI. |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
Monitoring
Field | Description |
---|---|
name | string Name of the monitoring system. |
description | string Description of the monitoring system. |
link | string Link to the monitoring system. |
ClusterConfig
Field | Description |
---|---|
version_id | string Version of Apache that runs on the cluster. |
airflow | AirflowConfig Configuration of the Apache Airflow application itself. |
webserver | WebserverConfig Required. Configuration of webserver instances. |
scheduler | SchedulerConfig Required. Configuration of scheduler instances. |
triggerer | TriggererConfig Configuration of triggerer instances. |
worker | WorkerConfig Required. Configuration of worker instances. |
dependencies | Dependencies The list of additional packages installed in the cluster. |
lockbox | LockboxConfig Configuration of Lockbox Secret Backend. |
AirflowConfig
Field | Description |
---|---|
config | map<string,string> Properties to be passed to Apache Airflow configuration file. |
WebserverConfig
Field | Description |
---|---|
count | int64 The number of webserver instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to webserver instances. |
Resources
Field | Description |
---|---|
resource_preset_id | string ID of the preset for computational resources available to an instance (CPU, memory etc.). |
SchedulerConfig
Field | Description |
---|---|
count | int64 The number of scheduler instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to scheduler instances. |
TriggererConfig
Field | Description |
---|---|
count | int64 The number of triggerer instances in the cluster. Acceptable values are 0 to 512, inclusive. |
resources | Resources Resources allocated to triggerer instances. |
WorkerConfig
Field | Description |
---|---|
min_count | int64 The minimum number of worker instances in the cluster. Acceptable values are 0 to 512, inclusive. |
max_count | int64 The maximum number of worker instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to worker instances. |
Dependencies
Field | Description |
---|---|
pip_packages[] | string Python packages that are installed in the cluster. |
deb_packages[] | string System packages that are installed in the cluster. |
LockboxConfig
Field | Description |
---|---|
enabled | bool The setting allows to enable Lockbox Secret Backend. |
NetworkConfig
Field | Description |
---|---|
subnet_ids[] | string IDs of VPC network subnets where instances of the cluster are attached. |
security_group_ids[] | string User security groups. |
CodeSyncConfig
Field | Description |
---|---|
source | oneof: s3 |
s3 | S3Config |
S3Config
Field | Description |
---|---|
bucket | string The name of the Object Storage bucket that stores DAG files used in the cluster. |
LoggingConfig
Field | Description |
---|---|
enabled | bool Logs generated by the Airflow components are delivered to Cloud Logging. |
destination | oneof: folder_id or log_group_id Destination of log records. |
folder_id | string Logs should be written to default log group for specified folder. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
log_group_id | string Logs should be written to log group resolved by ID. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
min_level | yandex.cloud.logging.v1.LogLevel.Level Minimum log entry level. See [LogLevel.Level] for details. |
Stop
Stops the specified Apache Airflow cluster.
rpc Stop (StopClusterRequest) returns (operation.Operation)
Metadata and response of Operation:
Operation.metadata:StopClusterMetadata
Operation.response:Cluster
StopClusterRequest
Field | Description |
---|---|
cluster_id | string Required. ID of the Apache Airflow cluster to stop. The maximum string length in characters is 50. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any if operation finished successfully. |
StopClusterMetadata
Field | Description |
---|---|
cluster_id | string ID of the Apache Airflow cluster that is being stopped. |
Cluster
Field | Description |
---|---|
id | string Unique ID of the Apache Airflow cluster. This ID is assigned by Cloud during cluster creation. |
folder_id | string ID of the folder that the Apache Airflow cluster belongs to. |
created_at | google.protobuf.Timestamp The time when the Apache Airflow cluster was created. |
name | string Name of the Apache Airflow cluster. The name is unique within the folder. 1-64 characters long. |
description | string Description of the Apache Airflow cluster. 0-256 characters long. |
labels | map<string,string> Resource labels as key:value pairs. Maximum of 64 per resource. |
monitoring[] | Monitoring Monitoring systems relevant to the Apache Airflow cluster. |
config | ClusterConfig Configuration of Apache Airflow components. |
health | enum Health Aggregated cluster health.
|
status | enum Status Cluster status.
|
network | NetworkConfig Network related configuration options. |
code_sync | CodeSyncConfig Parameters of the location and access to the code that will be executed in the cluster. |
deletion_protection | bool Deletion Protection inhibits deletion of the cluster. |
webserver_url | string Address of Apache Airflow web UI. |
service_account_id | string Service account used to access Cloud resources. For more information, see documentation. The maximum string length in characters is 50. |
logging | LoggingConfig Cloud Logging configuration. |
Monitoring
Field | Description |
---|---|
name | string Name of the monitoring system. |
description | string Description of the monitoring system. |
link | string Link to the monitoring system. |
ClusterConfig
Field | Description |
---|---|
version_id | string Version of Apache that runs on the cluster. |
airflow | AirflowConfig Configuration of the Apache Airflow application itself. |
webserver | WebserverConfig Required. Configuration of webserver instances. |
scheduler | SchedulerConfig Required. Configuration of scheduler instances. |
triggerer | TriggererConfig Configuration of triggerer instances. |
worker | WorkerConfig Required. Configuration of worker instances. |
dependencies | Dependencies The list of additional packages installed in the cluster. |
lockbox | LockboxConfig Configuration of Lockbox Secret Backend. |
AirflowConfig
Field | Description |
---|---|
config | map<string,string> Properties to be passed to Apache Airflow configuration file. |
WebserverConfig
Field | Description |
---|---|
count | int64 The number of webserver instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to webserver instances. |
Resources
Field | Description |
---|---|
resource_preset_id | string ID of the preset for computational resources available to an instance (CPU, memory etc.). |
SchedulerConfig
Field | Description |
---|---|
count | int64 The number of scheduler instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to scheduler instances. |
TriggererConfig
Field | Description |
---|---|
count | int64 The number of triggerer instances in the cluster. Acceptable values are 0 to 512, inclusive. |
resources | Resources Resources allocated to triggerer instances. |
WorkerConfig
Field | Description |
---|---|
min_count | int64 The minimum number of worker instances in the cluster. Acceptable values are 0 to 512, inclusive. |
max_count | int64 The maximum number of worker instances in the cluster. Acceptable values are 1 to 512, inclusive. |
resources | Resources Resources allocated to worker instances. |
Dependencies
Field | Description |
---|---|
pip_packages[] | string Python packages that are installed in the cluster. |
deb_packages[] | string System packages that are installed in the cluster. |
LockboxConfig
Field | Description |
---|---|
enabled | bool The setting allows to enable Lockbox Secret Backend. |
NetworkConfig
Field | Description |
---|---|
subnet_ids[] | string IDs of VPC network subnets where instances of the cluster are attached. |
security_group_ids[] | string User security groups. |
CodeSyncConfig
Field | Description |
---|---|
source | oneof: s3 |
s3 | S3Config |
S3Config
Field | Description |
---|---|
bucket | string The name of the Object Storage bucket that stores DAG files used in the cluster. |
LoggingConfig
Field | Description |
---|---|
enabled | bool Logs generated by the Airflow components are delivered to Cloud Logging. |
destination | oneof: folder_id or log_group_id Destination of log records. |
folder_id | string Logs should be written to default log group for specified folder. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
log_group_id | string Logs should be written to log group resolved by ID. Value must match the regular expression ([a-zA-Z][-a-zA-Z0-9_.]{0,63})? . |
min_level | yandex.cloud.logging.v1.LogLevel.Level Minimum log entry level. See [LogLevel.Level] for details. |
ListOperations
Retrieves the list of Operation resources for the specified cluster.
rpc ListOperations (ListClusterOperationsRequest) returns (ListClusterOperationsResponse)
ListClusterOperationsRequest
Field | Description |
---|---|
cluster_id | string Required. ID of the Apache Airflow Cluster resource to list operations for. The maximum string length in characters is 50. |
page_size | int64 The maximum number of results per page to return. If the number of available results is larger than page_size , the service returns a ListClusterOperationsResponse.next_page_token that can be used to get the next page of results in subsequent list requests. The maximum value is 1000. |
page_token | string Page token. To get the next page of results, set page_token to the ListClusterOperationsResponse.next_page_token returned by the previous list request. The maximum string length in characters is 100. |
ListClusterOperationsResponse
Field | Description |
---|---|
operations[] | operation.Operation List of Operation resources for the specified Apache Airflow cluster. |
next_page_token | string This token allows you to get the next page of results for list requests. If the number of results is larger than ListClusterOperationsRequest.page_size, use the next_page_token as the value for the ListClusterOperationsRequest.page_token query parameter in the next list request. Each subsequent list request will have its own next_page_token to continue paging through the results. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty |