Transferring the Managed Service for Apache Airflow™ cluster logs to Yandex Cloud Logging

Written by

Updated at December 4, 2024

Transferring data to the default log group
Sending data to a custom log group

You can set up regular collection of Managed Service for Apache Airflow™ cluster performance logs. Logs will be delivered to a log group in Cloud Logging. You can choose between these two types of log groups:

Log group used by default in the cluster folder.
Custom log group.

Transferring data to the default log group

Make sure the service account has been assigned the managed-airflow.integrationProvider role.
Specify logging settings in the Managed Service for Apache Airflow™ cluster:
Management console

CLI

Terraform

API
1. Go to the folder page and select Managed Service for Apache Airflow™.
2. Select the cluster and click Edit in the top panel.
3. Under Logging, enable the Write logs option.
4. To write logs to the default log group, select Folder in the Destination field.
5. Specify the folder whose log group you want to be using.
6. Select the minimum logging level.
  
  Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
Specify the following logging parameters in the cluster create command:
```
yc managed-airflow cluster update \
   ...
   --log-enabled \
   --log-folder-id <folder_ID> \
   --log-min-level <logging_level>
```
Specify the folder whose log group you want to be using.

Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
Specify the following parameters in the configuration file with the cluster description:
```
resource "yandex_airflow_cluster" "<cluster_name>" {
  ...
  logging = {
    enabled   = true
    folder_id = "<folder_ID>"
    min_level = "<logging_level>"
  }
}
```
Specify the folder whose log group you want to be using.

Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
In the body of the cluster update request (Cluster.Update in the REST API or ClusterService.Update in the gRPC API), specify the following parameters:
```
{
   ...
   "logging": {
      "enabled": true,
      "minLevel": "<logging_level>",
      "folderId": "<folder_ID>"
   }
}
```
Specify the folder whose log group you want to be using.

Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.

Test the transfer of cluster logs to the log group.

Management console

CLI

API

In the management console, go to the relevant folder.
Select Cloud Logging.
Click the row with the default log group.

The page that opens will show the log group records.

To view the records in JSON format, run the command:

yc logging read --group-name=default --format=json

Result:

[
  {
    "uid": "3:74********",
    "resource": {
      "type": "managed-airflow.cluster",
      "id": "c9qv4tnjqdpa********"
    },
    "timestamp": "2024-10-31T11:14:53.740223Z",
    "ingested_at": "2024-10-31T11:14:55.633Z",
    "saved_at": "2024-10-31T11:14:57.231685Z",
    "level": "INFO",
    "message": "10.253.244.40 - - \"GET /health HTTP/1.1\" 200 283 \"-\" \"kube-probe/1.25\"",
    "json_payload": {
      "file": "/home/airflow/.local/lib/python3.8/site-packages/gunicorn/glogging.py",
      "instance": "airflow-c9qv4tnjqdpa********-webserver-68********-q5***",
      "line": 363,
      "resource_id": "c9qv4tnjqdpa********",
      "stream_name": "webserver",
      "thread": "MainThread"
    },
    "stream_name": "webserver"
  }
]

To view log group records, use the LogReadingService/Read gRPC API call.

For more information, see Reading records.

Sending data to a custom log group

Create a log group named airflow-log-group.
Make sure the service account has been assigned the managed-airflow.integrationProvider role.
Specify logging settings in the Managed Service for Apache Airflow™ cluster:
Management console

CLI

Terraform

API
1. Go to the folder page and select Managed Service for Apache Airflow™.
2. Select the cluster and click Edit in the top panel.
3. Under Logging, enable the Write logs option.
4. To write logs to a custom log group, select Log group in the Destination field.
5. Specify the log group, airflow-log-group.
6. Select the minimum logging level.
  
  Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
Specify the following logging parameters in the cluster create command:
```
yc managed-airflow cluster create \
   ...
   --log-enabled \
   --log-group-id <log_group_ID> \
   --log-min-level <logging_level>
```
Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
Specify the following parameters in the configuration file with the cluster description:
```
resource "yandex_airflow_cluster" "<cluster_name>" {
  ...
  logging = {
    enabled      = true
    log_group_id = "<log_group_ID>"
    min_level    = "<logging_level>"
  }
}
```
Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
In the body of the cluster update request (Cluster.Update in the REST API or ClusterService.Update in the gRPC API), specify the following parameters:
```
{
   ...
   "logging": {
      "enabled": true,
      "minLevel": "<logging_level>",
      "logGroupId": "<log_group_ID>"
   }
}
```
Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.