Transferring Apache Hive™ Metastore cluster logs to Yandex Cloud Logging

You can set up regular collection of Apache Hive™ Metastore cluster performance logs. Logs will be delivered to a log group in Yandex Cloud Logging. You can choose between these two types of log groups:

• Default log group of the cluster folder.
• Custom log group.

Getting startedGetting started

1. Create a service account named metastore-logging-sa.
2. Assign the managed-metastore.integrationProvider role to the service account.
3. Set up a NAT gateway in the subnet the cluster will connect to. This will enable the cluster to work with Yandex Cloud services.
4. Configure the security group.

Transferring data to the default log groupTransferring data to the default log group

1. Create a Apache Hive™ Metastore cluster:

   Management console

   CLI

   REST API

   gRPC API

   1. In the management console, select the folder you need.

   2. Select Yandex MetaData Hub.

   3. In the left-hand panel, select Metastore.

   4. Click Create cluster.

   5. Specify metastore-cluster as the cluster name.

   6. In the Service account field, select metastore-logging-sa.

   7. Under Network settings, select the network and subnet with a NAT gateway configured. Specify the security group you configured.

   8. Under Logging:

      1. Enable Write logs.

      2. To write logs to the default log group, select Folder in the Destination field.

      3. Specify the folder whose log group you want to use.

      4. Select the minimum logging level.

         The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.

   9. Click Create.

   If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

   By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

   Run this command:

   yc managed-metastore cluster create \
      --name metastore-cluster \
      --service-account-id <service_account_ID> \
      --version <Apache Hive™ Metastore_version> \
      --subnet-ids <subnet_IDs> \
      --security-group-ids <security_group_IDs> \
      --resource-preset-id <ID_of_computing_resources> \
      --log-enabled \
      --log-folder-id <folder_ID> \
      --log-min-level <logging_level>
   

   Where:

   • --service-account-id: ID of the metastore-logging-sa service account you created earlier.
   • --subnet-ids: ID of the subnet where you set up the NAT gateway.
   • --security-group-ids: ID of the security group you configured earlier.
   • --log-folder-id: Specify the ID of the folder whose log group you want to use.
   • --log-min-level: Logging level. The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.

   Learn more about creating a cluster.

   1. Get an IAM token for API authentication and save it as an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Create a file named body.json and paste the following code into it:

      {
        "folderId": "<folder_ID>",
        "name": "metastore-cluster",
        "version": "<Apache Hive™ Metastore_version>",
        "configSpec": {
          "resources": {
          "resourcePresetId": "<resource_configuration_ID>"
          }
        },
        "serviceAccountId": "<service_account_ID>",
        "logging": {
          "enabled": true,
          "folderId": "<folder_ID>",
          "minLevel": "<logging_level>"
        },
        "network": {
          "subnetIds": [ "<list_of_subnet_IDs>" ],
          "securityGroupIds": [ "<list_of_security_group_IDs>" ]
        },
      }
      

      Where:

      • serviceAccountId: ID of the metastore-logging-sa service account you created earlier.
      • logging.folderId: Specify the ID of the folder whose log group you want to use.
      • logging.minLevel: Logging level. The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.
      • network.subnetIds: ID of the subnet where you set up the NAT gateway.
      • network.securityGroupIds: ID of the security group you configured earlier.

      Learn more about creating a cluster.

   3. Use the Cluster.Create method and send the following request, e.g., via cURL:

      curl \
          --request POST \
          --header "Authorization: Bearer $IAM_TOKEN" \
          --url 'https://metastore.api.cloud.yandex.net/managed-metastore/v1/clusters' \
          --data '@body.json'
      

   4. View the server response to make sure your request was successful.

   1. Get an IAM token for API authentication and save it as an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Clone the cloudapi repository:

      cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
      

      Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

   3. Create a file named body.json and paste the following code into it:

      {
        "folder_id": "<folder_ID>",
        "name": "metastore-cluster",
        "version": "<Apache Hive™ Metastore_version>",
        "config_spec": {
          "resources": {
            "resource_preset_id": "<resource_configuration_ID>"
          }
        },
        "service_account_id": "<service_account_ID>",
        "logging": {
          "enabled": true,
          "folder_id": "<folder_ID>",
          "min_level": "<logging_level>"
        },
        "network": {
          "subnet_ids": [ "<list_of_subnet_IDs>" ],
          "security_group_ids": [ "<list_of_security_group_IDs>" ]
        },
      }
      

      Where:

      • service_account_id: ID of the metastore-logging-sa service account you created earlier.
      • logging.folder_id: Specify the ID of the folder whose log group you want to use.
      • logging.min_level: Logging level. The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.
      • network.subnet_ids: ID of the subnet where you set up the NAT gateway.
      • network.security_group_ids: ID of the security group you configured earlier.

      Learn more about creating a cluster.

   4. Use the ClusterService.Create call and send the following request, e.g., via gRPCurl:

      grpcurl \
          -format json \
          -import-path ~/cloudapi/ \
          -import-path ~/cloudapi/third_party/googleapis/ \
          -proto ~/cloudapi/yandex/cloud/metastore/v1/cluster_service.proto \
          -rpc-header "Authorization: Bearer $IAM_TOKEN" \
          -d @ \
          metastore.api.cloud.yandex.net:443 \
          yandex.cloud.metastore.v1.ClusterService.Create \
          < body.json
      

   5. View the server response to make sure your request was successful.

2. Test the transfer of cluster logs to the log group:

   Management console

   CLI

   API

   1. In the management console, navigate to the relevant folder.
   2. Select Cloud Logging.
   3. Click the row with the default log group.

   The page that opens will show the log group entries.

   If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

   By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

   To view the entries in JSON format, run this command:

   yc logging read --group-name=default --format=json
   

   Result:

   [
     {
       "uid": "23:6d******:3",
       "resource": {
         "type": "managed-metastore.cluster",
         "id": "c9qcv46p3g3a********"
       },
       "timestamp": "2024-11-18T09:45:23.092484Z",
       "ingested_at": "2024-11-18T09:45:24Z",
       "saved_at": "2024-11-18T09:45:25.559195Z",
       "level": "INFO",
       "message": "ugi=metastore\tip=10.253.89.135\tcmd=Done cleaning up thread local RawStore\t",
       "json_payload": {
         "instance": "metastore-server-5f9c7ff***-*****",
         "logger": "org.apache.hadoop.hive.metastore.HiveMetaStore.audit",
         "resource_id": "c9qcv46p3g3a********",
         "stream_name": "metastore-server",
         "thread": "pool-6-thread-79"
       },
       "stream_name": "metastore-server"
     }
   ]
   

   To view log group entries, use the LogReadingService.Read gRPC API call.

   For more information, see Reading records.

Sending data to a custom log groupSending data to a custom log group

1. Create a log group named metastore-log-group.

2. Create a Apache Hive™ Metastore cluster:

   Management console

   CLI

   REST API

   gRPC API

   1. In the management console, select the folder you need.

   2. Select Yandex MetaData Hub.

   3. In the left-hand panel, select Metastore.

   4. Click Create cluster.

   5. Specify metastore-cluster as the cluster name.

   6. In the Service account field, select metastore-logging-sa.

   7. Under Network settings, select the network and subnet with a NAT gateway configured. Specify the security group you configured.

   8. Under Logging:

      1. Enable Write logs.

      2. To write logs to a custom log group, select Log group in the Destination field.

      3. Specify the metastore-log-group log group.

      4. Select the minimum logging level.

         The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.

   9. Click Create.

   If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

   By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

   Run this command:

   yc managed-metastore cluster create \
      --name metastore-cluster \
      --service-account-id <service_account_ID> \
      --version <Apache Hive™ Metastore_version> \
      --subnet-ids <subnet_IDs> \
      --security-group-ids <security_group_IDs> \
      --resource-preset-id <ID_of_computing_resources> \
      --log-enabled \
      --log-group-id <log_group_ID> \
      --log-min-level <logging_level>
   

   Where:

   • --service-account-id: ID of the metastore-logging-sa service account you created earlier.
   • --subnet-ids: ID of the subnet where you set up the NAT gateway.
   • --security-group-ids: ID of the security group you configured earlier.
   • --log-group-id: metastore-log-group log group ID.
   • --log-min-level: Logging level. The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.

   Learn more about creating a cluster.

   1. Get an IAM token for API authentication and save it as an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Create a file named body.json and paste the following code into it:

      {
        "folderId": "<folder_ID>",
        "name": "metastore-cluster",
        "version": "<Apache Hive™ Metastore_version>",
        "configSpec": {
          "resources": {
          "resourcePresetId": "<resource_configuration_ID>"
          }
        },
        "serviceAccountId": "<service_account_ID>",
        "logging": {
          "enabled": true,
          "logGroupId": "<log_group_ID>",
          "minLevel": "<logging_level>"
        },
        "network": {
          "subnetIds": [ "<list_of_subnet_IDs>" ],
          "securityGroupIds": [ "<list_of_security_group_IDs>" ]
        },
      }
      

      Where:

      • serviceAccountId: ID of the metastore-logging-sa service account you created earlier.
      • logging.logGroupId: metastore-log-group log group ID.
      • logging.minLevel: Logging level. The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.
      • network.subnetIds: ID of the subnet where you set up the NAT gateway.
      • network.securityGroupIds: ID of the security group you configured earlier.

      Learn more about creating a cluster.

   3. Use the Cluster.Create method and send the following request, e.g., via cURL:

      curl \
          --request POST \
          --header "Authorization: Bearer $IAM_TOKEN" \
          --url 'https://metastore.api.cloud.yandex.net/managed-metastore/v1/clusters' \
          --data '@body.json'
      

   4. View the server response to make sure your request was successful.

   1. Get an IAM token for API authentication and save it as an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Clone the cloudapi repository:

      cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
      

      Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

   3. Create a file named body.json and paste the following code into it:

      {
        "folder_id": "<folder_ID>",
        "name": "metastore-cluster",
        "version": "<Apache Hive™ Metastore_version>",
        "config_spec": {
          "resources": {
            "resource_preset_id": "<resource_configuration_ID>"
          }
        },
        "service_account_id": "<service_account_ID>",
        "logging": {
          "enabled": true,
          "log_group_id": "<log_group_ID>",
          "min_level": "<logging_level>"
        },
        "network": {
          "subnet_ids": [ "<list_of_subnet_IDs>" ],
          "security_group_ids": [ "<list_of_security_group_IDs>" ]
        },
      }
      

      Where:

      • service_account_id: ID of the metastore-logging-sa service account you created earlier.
      • logging.log_group_id: metastore-log-group log group ID.
      • logging.min_level: Logging level. The execution log will contain logs of this level or higher. The available levels are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. The default is INFO.
      • network.subnet_ids: ID of the subnet where you set up the NAT gateway.
      • network.security_group_ids: ID of the security group you configured earlier.

      Learn more about creating a cluster.

   4. Use the ClusterService.Create call and send the following request, e.g., via gRPCurl:

      grpcurl \
          -format json \
          -import-path ~/cloudapi/ \
          -import-path ~/cloudapi/third_party/googleapis/ \
          -proto ~/cloudapi/yandex/cloud/metastore/v1/cluster_service.proto \
          -rpc-header "Authorization: Bearer $IAM_TOKEN" \
          -d @ \
          metastore.api.cloud.yandex.net:443 \
          yandex.cloud.metastore.v1.ClusterService.Create \
          < body.json
      

   5. View the server response to make sure your request was successful.

3. Test the transfer of cluster logs to the log group:

   Management console

   CLI

   API

   1. In the management console, navigate to the relevant folder.
   2. Select Cloud Logging.
   3. Click the row with the metastore-log-group log group.

   The page that opens will show the log group entries.

   If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

   By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

   To view the entries in JSON format, run this command:

   yc logging read --group-name=metastore-log-group --format=json
   

   Result:

   [
     {
       "uid": "23:6d******:3",
       "resource": {
         "type": "managed-metastore.cluster",
         "id": "c9qcv46p3g3a********"
       },
       "timestamp": "2024-11-18T09:45:23.092484Z",
       "ingested_at": "2024-11-18T09:45:24Z",
       "saved_at": "2024-11-18T09:45:25.559195Z",
       "level": "INFO",
       "message": "ugi=metastore\tip=10.253.89.135\tcmd=Done cleaning up thread local RawStore\t",
       "json_payload": {
         "instance": "metastore-server-5f9c7ff***-*****",
         "logger": "org.apache.hadoop.hive.metastore.HiveMetaStore.audit",
         "resource_id": "c9qcv46p3g3a********",
         "stream_name": "metastore-server",
         "thread": "pool-6-thread-79"
       },
       "stream_name": "metastore-server"
     }
   ]
   

   To view log group entries, use the LogReadingService.Read gRPC API call.

   For more information, see Reading records.

Apache® and Apache Hive™ are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.