Contact UsGet started

Transferring the Metastore cluster logs to Yandex Cloud Logging

Written by
Updated at November 12, 2024

You can set up regular collection of logs with information about Hive Metastore cluster performance. Such logs will be delivered to a custom log group in Yandex Cloud Logging.

To set up delivery of Metastore cluster logs to Cloud Logging:

  1. Create a log group.
  2. Create a service account.
  3. Assign the service account a role for the folder.
  4. Create a Metastore cluster.
  5. Test the delivery of cluster logs to the log group.

If you no longer need the resources you created, delete them.

Create a log group

  1. In the management console, select the folder where you want to create your log group.
  2. Select Cloud Logging.
  3. Click Create group.
  4. Enter metastore-log-group as the name.
  5. Set the retention period for log group records.
  6. Click Create group.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a log group, run this command:

yc logging group create \
   --name metastore-log-group \
   --retention-period <record_retention_period>

This will create a group named metastore-log-group.

The record retention period can only be specified in hours, minutes, or seconds. For example, 1h or 1440m.

  1. If you don't have Terraform, install it and configure the Yandex Cloud provider.

  2. In the configuration file, describe the log group parameters:

    resource "yandex_logging_group" "metastore-log-group" {
  name             = "metastore-log-group"
  folder_id        = "<folder_ID>"
  retention_period = "<record_retention_period>"
}

    You can get the folder ID together with the list of folders.

    The record retention period can only be specified in hours, minutes, or seconds. For example, 1h or 1440m.

    For more information about the yandex_logging_group resource parameters in Terraform, see the provider documentation.

  3. Make sure the configuration files are correct.

    1. In the command line, go to the folder where you created the configuration file.

    2. Run a check using this command:

      terraform plan

    If the configuration is described correctly, the terminal will display a list of created resources and their parameters. If the configuration contains any errors, Terraform will point them out.

  4. Deploy cloud resources.

    1. If the configuration does not contain any errors, run this command:

      terraform apply

    2. Confirm creating the resources: type yes in the terminal and press Enter.

      All the resources you need will then be created in the specified folder. You can check the new resources and their configuration using the management console or this CLI command:

      yc logging group list

To create a log group, use the create REST API method for the LogGroup resource or the LogGroupService/Create gRPC API call and provide the following in the request body:

  • Folder ID in the folderId parameter. You can get the ID with the list of folders.

  • metastore-log-group as the log group name in the name parameter.

  • Retention period for log group records in the retentionPeriod parameter.

    You can specify it in seconds only. Use this format: <number>s. If you specify 0s or no value, the retention period will be unlimited.

Create a service account

Use the service account to deliver logs to Cloud Logging.

To create a service account:

  1. In the management console, select the folder where you created the log group.
  2. Select Identity and Access Management.
  3. Click Create service account.
  4. Enter a name for the service account: metastore-logging-sa.
  5. Click Create.

To create a service account, run the following command:

yc iam service-account create --name metastore-logging-sa

The naming requirements are as follows:

  • The name must be from 3 to 63 characters long.
  • It may contain lowercase Latin letters, numbers, and hyphens.
  • The first character must be a letter and the last character cannot be a hyphen.

To create a service account:

  1. In the configuration file, describe the service account parameters:

    resource "yandex_iam_service_account" "sa" {
  name        = "metastore-logging-sa"
  folder_id   = "<folder_ID>"
}

    For more information about the yandex_iam_service_account resource parameters in Terraform, see the provider documentation.

  2. Make sure the configuration files are correct.

    1. In the command line, go to the folder where you created the configuration file.

    2. Run a check using this command:

      terraform plan

    If the configuration is specified correctly, the terminal will display information about the service account. If the configuration contains any errors, Terraform will point them out.

  3. Deploy cloud resources.

    1. If the configuration does not contain any errors, run this command:

      terraform apply

    2. Confirm creating the service account: type yes in the terminal and press Enter.

      The service account will then be created. You can check the new service account using the management console or this CLI command:

      yc iam service-account list

To create a service account, use the create REST API method for the ServiceAccount resource or the ServiceAccountService/Create gRPC API call and provide the following in the request body:

  • Folder ID in the folderId parameter.
  • metastore-logging-sa as the service account name in the name parameter.

Assign the service account a role for the folder

Assign the managed-metastore.integrationProvider role for the folder to the metastore-logging-sa service account. This role enables a Metastore cluster to work with various Yandex Cloud resources, which includes sending logs to Cloud Logging.

To assign a role:

  1. In the management console, select the appropriate folder.
  2. Go to the Access bindings tab.
  3. Click Configure access.
  4. In the window that opens, enter the service account name,metastore-logging-sa, and select it.
  5. Click Add role and select the managed-metastore.integrationProvider role.
  6. Click Save.

To assign a role, run this command:

yc resource-manager folder add-access-binding <folder_name_or_ID> \
   --role managed-metastore.integrationProvider \
   --subject serviceAccount:<service_account_ID>

You can get the service account ID with the list of service accounts.

To assign a role:

  1. In the configuration file, describe the service account role parameters:

    resource "yandex_resourcemanager_folder_iam_member" "managed-metastore-integration-provider" {
  folder_id   = "<folder_ID>"
  role        = "managed-metastore.integrationProvider"
  member      = "serviceAccount:<service_account_ID>"
}

    You can get the service account ID with the list of service accounts.

  2. Make sure the configuration files are correct.

    1. In the command line, go to the folder where you created the configuration file.

    2. Run a check using this command:

      terraform plan

    If the configuration is described correctly, the terminal will display a list of created resources and their parameters. If the configuration contains any errors, Terraform will point them out.

  3. Deploy cloud resources.

    1. If the configuration does not contain any errors, run this command:

      terraform apply

    2. Confirm creating the resources: type yes in the terminal and press Enter.

      All the resources you need will then be created in the specified folder. You can check the new resource using the management console or this CLI command:

      yc resource-manager folder list-access-bindings <folder_name_or_ID>

To assign a role, use the updateAccessBindings REST API method for the Folder resource or the FolderService/UpdateAccessBindings gRPC API call and provide the following in the request body:

  • ADD value in the accessBindingDeltas.action parameter to add a role.
  • managed-metastore.integrationProvider role in the accessBindingDeltas.accessBinding.roleId parameter.
  • Service account ID in the accessBindingDeltas.accessBinding.subject.id parameter. You can get the ID with the list of service accounts.
  • serviceAccount value in the accessBindingDeltas.accessBinding.subject.type parameter to indicate the role is being assigned to a service account.

Specify the ID of the appropriate folder in the request URL.

Create a Metastore cluster

  1. Set up a NAT gateway in the subnet the cluster will connect to. This will enable the cluster to work with Yandex Cloud services.

  2. Configure security groups.

  3. Create a Metastore cluster:

    1. In the management console, select the appropriate folder.

    2. Select Yandex MetaData Hub.

    3. In the left-hand panel, select the Metastore page.

    4. Click Create cluster.

    5. Specify metastore-cluster as the cluster name.

    6. In the Service account field, select metastore-logging-sa.

    7. Under Network settings, select the network and subnet with a NAT gateway configured.

    8. Under Logging:

      1. Enable Write logs.

      2. To write logs to a custom log group, select Log group in the Destination field.

      3. Specify the log group, metastore-log-group.

      4. Select the minimum logging level.

        Logs of the specified level and higher will be written to the execution log. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default level is INFO.

    9. Click Create.

Test the delivery of cluster logs to the log group

  1. In the management console, go to the relevant folder.
  2. Select Cloud Logging.
  3. Click the row with the metastore-log-group log group.

The page that opens will show the log group records.

To view the records in JSON format, run the command:

yc logging read --group-name=metastore-log-group --format=json

Result:

[
  {
    "uid": "488ece3c-75b8-4d35-95ac-2b49********",
    "resource": {},
    "timestamp": "2024-08-22T02:10:40Z",
    "ingested_at": "2024-08-22T08:49:15.716Z",
    "saved_at": "2024-08-22T08:49:16.176097Z",
    "level": "INFO",
    "message": "My message",
    "json_payload": {
      "request_id": "1234"
    }
  }
]

To view log group records, use the LogReadingService/Read gRPC API call.

For more information, see Reading records.

Delete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the configuration files with the infrastructure created for delivering Metastore cluster logs.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the resources described in the deleted configuration files will be deleted.

Previous
Transferring data between Yandex Data Processing clusters using Metastore
Next
About Yandex MetaData Hub