Updating a Apache Hive™ Metastore cluster

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To change a Apache Hive™ Metastore cluster’s settings:

1. View the description of the CLI command for updating a cluster:

   yc managed-metastore cluster update --help
   

2. Provide a list of settings to update in the update cluster command:

   yc managed-metastore cluster update <cluster_name_or_ID> \
      --new-name <cluster_name> \
      --description <cluster_description> \
      --labels <label_list> \
      --service-account-id <service_account_ID> \
      --version <Apache_Hive™_Metastore_version> \
      --warehouse-bucket <bucket_name> \
      --warehouse-path <path_within_bucket> \
      --security-group-ids <security_group_IDs> \
      --resource-preset-id <ID_of_computing_resources> \
      --maintenance-window type=<maintenance_type>,`
                           `day=<day_of_week>,`
                           `hour=<hour> \
      --deletion-protection \
      --log-enabled \
      --log-folder-id <folder_ID> \
      --log-min-level <logging_level>
   

   Where:

   • --new-name: New cluster name.

   • --description: Cluster description.

   • --labels: List of labels. Provide labels in <key>=<value> format.

   • --service-account-id: Service account ID.

   • --version: Apache Hive™ Metastore version.

     Available versions: 3.1, 4.0, and 4.2.

     Warning

     Before integrating an Apache Hive™ Metastore cluster with Yandex Managed Service for Apache Spark™, make sure their versions are compatible:

     • Apache Spark™ 3.5.7 supports Apache Hive™ Metastore 3.1.
     • Apache Spark™ 4.0.0 supports Apache Hive™ Metastore versions 3.1 and higher.

     If required, you can upgrade from Apache Hive™ Metastore 3.1 in this sequence: 3.1 → 4.0 → 4.2. However, you cannot downgrade 4.2 to 3.1.

   • --warehouse-bucket: Name of the Object Storage bucket to store the Apache Hive™ Metastore (warehouse) data.

   • --warehouse-path: Path within the bucket to prefix the Apache Hive™ Metastore data. This is an optional setting.

   • --security-group-ids: List of security group IDs.

   • --resource-preset-id: Computing resource configuration.

   • --maintenance-window: Maintenance window settings (including for disabled clusters), where type is the maintenance type:

     • anytime: Any time (default).
     • weekly: On a schedule. For this value, also specify the following:

       • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.

       • hour: Sequence number of UTC hour interval, from 1 to 24.

         For example, 1 stands for the interval from 00:00 to 01:00, and 5, from 04:00 to 05:00.

   • --deletion-protection: Enables cluster protection against accidental deletion.

   • Logging parameters:

     • --log-enabled: Enables logging. Logs generated by Apache Hive™ Metastore components will be sent to Yandex Cloud Logging.

     • --log-folder-id: Folder ID. Logs will be written to the default log group for this folder.

     • --log-group-id: Custom log group ID. Logs will be written to this group.

       Specify one of the two parameters: --log-folder-id or --log-group-id.

     • --log-min-level: Minimum logging level. Possible values: TRACE, DEBUG, INFO (default), WARN, ERROR, and FATAL.

   You can get the cluster ID and name with the list of clusters in the folder.

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Create a file named body.json and paste the following code into it:

   {
     "updateMask": "<list_of_parameters_to_update>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "labels": { "<label_list>" },
     "deletionProtection": <deletion_protection>,
     "version": "<Apache_Hive™_Metastore_version>",
     "configSpec": {
       "resources": {
       "resourcePresetId": "<resource_configuration_ID>"
       },
       "warehouse": {
         "s3": {
           "bucket": "<bucket_name>",
           "path": "<path_within_bucket>"
         }
       }
     },
     "serviceAccountId": "<service_account_ID>",
     "logging": {
       "enabled": <use_of_logging>,
       "folderId": "<folder_ID>",
       "minLevel": "<logging_level>"
     },
     "networkSpec": {
       "securityGroupIds": [ "<list_of_security_group_IDs>" ]
     },
     "maintenanceWindow": {
       "weeklyMaintenanceWindow": {
       "day": "<day_of_week>",
       "hour": "<hour>"
       }
     }
   }
   

   Where:

   • updateMask: Comma-separated string of settings you want to update.

     Warning

     When you update a cluster, all parameters of the object you are modifying will be reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the updateMask parameter.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels provided in "<key>": "<value>" format.

   • deletionProtection: Enables cluster protection against accidental deletion. The possible values are true or false.

   • version: Apache Hive™ Metastore version.

     Available versions: 3.1, 4.0, and 4.2.

     Warning

     Before integrating an Apache Hive™ Metastore cluster with Yandex Managed Service for Apache Spark™, make sure their versions are compatible:

     • Apache Spark™ 3.5.7 supports Apache Hive™ Metastore 3.1.
     • Apache Spark™ 4.0.0 supports Apache Hive™ Metastore versions 3.1 and higher.

     If required, you can upgrade from Apache Hive™ Metastore 3.1 in this sequence: 3.1 → 4.0 → 4.2. However, you cannot downgrade 4.2 to 3.1.

   • configSpec.resources.resourcePresetId: Computing resource configuration.

   • configSpec.warehouse.s3: Apache Hive™ Metastore data warehouse parameters:

     • bucket: Name of the Object Storage bucket to store the Apache Hive™ Metastore data (warehouse).
     • path: Path within the bucket that will be used to prefix the Apache Hive™ Metastore data. This is an optional setting.

   • serviceAccountId: Service account ID.

   • logging: Logging parameters:

     • enabled: Enables logging. Logs generated by Apache Hive™ Metastore components will be sent to Yandex Cloud Logging. The possible values are true or false.

     • folderId: Folder ID. Logs will be written to the default log group for this folder.

     • logGroupId: Custom log group ID. Logs will be written to this group.

       Specify either folderId or logGroupId.

     • minLevel: Minimum logging level. The possible values are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.

   • networkSpec: Network settings:

     • securityGroupIds: List of security group IDs.

   • maintenanceWindow: Maintenance window settings (including for disabled clusters). In maintenanceWindow, provide one of the two parameters:

     • anytime: Maintenance can take place at any time.

     • weeklyMaintenanceWindow: Maintenance takes place once a week at the specified time:

       • day: Day of week, in DDD format, MON, TUE, WED, THU, FRI, SAT, or SUN.
       • hour: Time of day (UTC) in HH format, from 1 to 24.

3. Call the Cluster.Update method, e.g., via the following cURL request:

   curl \
       --request PATCH \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://metastore.api.cloud.yandex.net/managed-metastore/v1/clusters/<cluster_ID>' \
       --data '@body.json'
   

   You can get the cluster ID with the list of clusters in the folder.

4. Check the server response to make sure your request was successful.

1. Get an IAM token for API authentication and place it in an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "cluster_id": "<cluster_ID>",
     "update_mask": "<list_of_parameters_to_update>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "labels": "{ <label_list> }",
     "deletion_protection": <deletion_protection>,
     "version": "<Apache_Hive™_Metastore_version>",
     "config_spec": {
       "resources": {
         "resource_preset_id": "<resource_configuration_ID>"
       },
       "warehouse": {
         "s3": {
           "bucket": "<bucket_name>",
           "path": "<path_within_bucket>"
         }
       }
     },
     "service_account_id": "<service_account_ID>",
     "logging": {
       "enabled": <use_of_logging>,
       "folder_id": "<folder_ID>",
       "min_level": "<logging_level>"
     },
     "network_spec": {
       "security_group_ids": [ "<list_of_security_group_IDs>" ]
     },
     "maintenance_window": {
       "weekly_maintenance_window": {
         "day": "<day_of_week>",
         "hour": "<hour>"
       }
     }
   }
   

   Where:

   • cluster_id: Cluster ID. You can get it with the list of clusters in the folder.

   • update_mask: List of parameters you want to update as an array of strings (paths[]).

     Format for listing settings

     "update_mask": {
         "paths": [
             "<setting_1>",
             "<setting_2>",
             ...
             "<setting_N>"
         ]
     }
     

     Warning

     When you update a cluster, all parameters of the object you are modifying will be reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the update_mask parameter.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels provided in "<key>": "<value>" format.

   • deletion_protection: Enables cluster protection against accidental deletion. The possible values are true or false.

   • version: Apache Hive™ Metastore version.

     Available versions: 3.1, 4.0, and 4.2.

     Warning

     Before integrating an Apache Hive™ Metastore cluster with Yandex Managed Service for Apache Spark™, make sure their versions are compatible:

     • Apache Spark™ 3.5.7 supports Apache Hive™ Metastore 3.1.
     • Apache Spark™ 4.0.0 supports Apache Hive™ Metastore versions 3.1 and higher.

     If required, you can upgrade from Apache Hive™ Metastore 3.1 in this sequence: 3.1 → 4.0 → 4.2. However, you cannot downgrade 4.2 to 3.1.

   • config_spec.resources.resource_preset_id: Computing resource configuration.

   • config_spec.warehouse.s3: Apache Hive™ Metastore data warehouse parameters:

     • bucket: Name of the Object Storage bucket to store the Apache Hive™ Metastore data (warehouse).
     • path: Path within the bucket that will be used to prefix the Apache Hive™ Metastore data. This is an optional setting.

   • service_account_id: Service account ID.

   • logging: Logging parameters:

     • enabled: Enables logging. Logs generated by Apache Hive™ Metastore components will be sent to Yandex Cloud Logging. The possible values are true or false.

     • folder_id: Folder ID. Logs will be written to the default log group for this folder.

     • log_group_id: Custom log group ID. Logs will be written to this group.

       Specify one of the two parameters: folder_id or log_group_id.

     • min_level: Minimum logging level. The possible values are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.

   • network_spec: Network settings:

     • security_group_ids: List of security group IDs.

   • maintenance_window: Maintenance window settings (including for disabled clusters). In maintenance_window, provide one of the two parameters:

     • anytime: Maintenance can take place at any time.

     • weekly_maintenance_window: Maintenance takes place once a week at the specified time:

       • day: Day of week, in DDD format, MON, TUE, WED, THU, FRI, SAT, or SUN.
       • hour: Time of day (UTC) in HH format, from 1 to 24.

4. Call the ClusterService.Update method, e.g., via the following gRPCurl request:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/metastore/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       metastore.api.cloud.yandex.net:443 \
       yandex.cloud.metastore.v1.ClusterService.Update \
       < body.json
   

5. Check the server response to make sure your request was successful.