Working with logs
Yandex Cloud Logging collects and displays Yandex Data Processing cluster logs.
To monitor events on the cluster and its individual hosts, specify the relevant log group in its settings. You can do this when creating or updating the cluster. If no log group is selected for the cluster, the default log group in the cluster directory will be used for sending and storing logs.
For more information, see Logs.
Viewing log entries
- Open the folder dashboard.
- Navigate to Yandex Data Processing.
- Click the cluster name.
- Under Configuration, click the name of the cluster log group. The Cloud Logging page will open.
- Click the log group row. This will open the cluster logs.
- Optionally, specify the output settings:
-
-
Getting the Yandex Data Processing job run output:
job_id="<job_ID>" -
Getting the
stdoutlogs for all YARN application containers:application_id="<YARN_app_ID>" AND yarn_log_type="stdout" -
Getting the
stderrlogs of a YARN container:container_id="<YARN_container_ID>" AND yarn_log_type="stderr" -
Getting YARN Resource Manager log entries from the cluster master host:
hostname="<master_host_FQDN>" AND log_type="hadoop-yarn-resourcemanager"
-
-
Message logging levels: From
TRACEtoFATAL. -
Number of messages per page.
-
Message interval (either standard or custom).
-
If you do not have the Yandex Cloud CLI yet, install and initialize it.
See the description of the CLI command for getting logs:
yc logging read --help
Examples:
-
To get logs of the Yandex Data Processing cluster's HDFS NameNode, run this command:
yc logging read \ --group-id=<log_group_ID> \ --resource-ids=<cluster_ID> \ --filter=log_type=hadoop-hdfs-namenode -
To get logs for the last two hours from all Yandex Data Processing clusters assigned to a specific log group, run this command:
yc logging read \ --group-id=<log_group_ID> \ --resource-types=dataproc.cluster \ --since=2h -
To get your cluster's system log over a specific period, run this command:
yc logging read \ --group-id <log_group_ID> \ --resource-ids=<cluster_ID> \ --filter 'syslog' \ --since 'YYYY-MM-DDThh:mm:ssZ' \ --until 'YYYY-MM-DDThh:mm:ssZ'Set the logging period in the
--sinceand--untilparameters. Time format:YYYY-MM-DDThh:mm:ssZ, e.g.,2020-08-10T12:00:00Z. Use the UTC time zone. -
To get the log for metrics sent from a specific host to Yandex Monitoring, run this command:
yc logging read \ --group-id <log_group_ID> \ --resource-ids=<cluster_ID> \ --filter 'telegraf and hostname="<host_FQDN>"' \ --since 'YYYY-MM-DDThh:mm:ssZ' \ --until 'YYYY-MM-DDThh:mm:ssZ'
Note
You can skip --group-id and specify the log group ID directly.
To get the host FQDN:
1. Open the [folder dashboard](https://console.yandex.cloud).
1. Navigate to **Yandex Data Processing**.
1. Click the cluster name.
1. Navigate to the **Hosts** tab.
1. Copy the host FQDN.
Disabling log sending
When creating or updating your cluster, add the dataproc:disable_cloud_logging property set to true.
If you do not have the Yandex Cloud CLI yet, install and initialize it.
The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
When creating or updating your cluster, specify dataproc:disable_cloud_logging=true in the --property parameter or provide an empty string ("") instead of the log group ID in the --log-group-id parameter:
yc dataproc cluster create <cluster_name> \
... \
--log-group-id=""
yc dataproc cluster update <cluster_name_or_ID> \
--property dataproc:disable_cloud_logging=true
Storing logs
Log collection and storage are billed according to the Cloud Logging pricing policy. The default log retention period is three days. To update the retention period, edit the log group settings.
For more information about logs, see this Cloud Logging guide.