Transferring the Metastore cluster logs to Yandex Cloud Logging

Written by

Updated at December 10, 2024

Getting started
Transferring data to the default log group
Sending data to a custom log group

You can set up regular collection of Metastore cluster performance logs. Logs are provided to a log group in Yandex Cloud Logging. You can choose between these two types of log groups:

Log group used by default in the cluster folder
Custom log group

Getting started

Create a service account named metastore-logging-sa.
Assign the managed-metastore.integrationProvider role to the service account.
Set up a NAT gateway in the subnet the cluster will connect to. This will enable the cluster to work with Yandex Cloud services.
Configure the security group.

Transferring data to the default log group

Create a Metastore cluster:
Management console
1. In the management console, select the appropriate folder.
2. Select Yandex MetaData Hub.
3. In the left-hand panel, select the Metastore page.
4. Click Create cluster.
5. Specify metastore-cluster as the cluster name.
6. In the Service account field, select metastore-logging-sa.
7. Under Network settings, select the network and subnet with a NAT gateway configured. Specify the security group you configured.
8. Under Logging:
  1. Enable Write logs.
  2. To write logs to the default log group, select Folder in the Destination field.
  3. Specify the folder whose log group you need to use.
  4. Select the minimum logging level.
    
    Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
9. Click Create.

Test the delivery of cluster logs to the log group.

Management console

CLI

API

In the management console, go to the relevant folder.
Select Cloud Logging.
Click the row with the default log group.

The page that opens will show the log group records.

To view the records in JSON format, run the command:

yc logging read --group-name=default --format=json

Result:

[
  {
    "uid": "23:6d******:3",
    "resource": {
      "type": "managed-metastore.cluster",
      "id": "c9qcv46p3g3a********"
    },
    "timestamp": "2024-11-18T09:45:23.092484Z",
    "ingested_at": "2024-11-18T09:45:24Z",
    "saved_at": "2024-11-18T09:45:25.559195Z",
    "level": "INFO",
    "message": "ugi=metastore\tip=10.253.89.135\tcmd=Done cleaning up thread local RawStore\t",
    "json_payload": {
      "instance": "metastore-server-5f9c7ff***-*****",
      "logger": "org.apache.hadoop.hive.metastore.HiveMetaStore.audit",
      "resource_id": "c9qcv46p3g3a********",
      "stream_name": "metastore-server",
      "thread": "pool-6-thread-79"
    },
    "stream_name": "metastore-server"
  }
]

To view log group records, use the LogReadingService.Read gRPC API call.

For more information, see Reading records.

Sending data to a custom log group

Create a metastore-log-group log group.
Create a Metastore cluster:
Management console
1. In the management console, select the appropriate folder.
2. Select Yandex MetaData Hub.
3. In the left-hand panel, select the Metastore page.
4. Click Create cluster.
5. Specify metastore-cluster as the cluster name.
6. In the Service account field, select metastore-logging-sa.
7. Under Network settings, select the network and subnet with a NAT gateway configured. Specify the security group you configured.
8. Under Logging:
  1. Enable Write logs.
  2. To write logs to a custom log group, select Log group in the Destination field.
  3. Specify the log group, metastore-log-group.
  4. Select the minimum logging level.
    
    Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.
9. Click Create.