Creating a Apache Hive™ Metastore cluster

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a Apache Hive™ Metastore cluster:

1. See the description of the CLI command for creating a cluster:

   yc managed-metastore cluster create --help
   

2. Specify the cluster properties in the creation command:

   yc managed-metastore cluster create \
      --name <cluster_name> \
      --description <cluster_description> \
      --labels <label_list> \
      --service-account-id <service_account_ID> \
      --version <Apache_Hive™_Metastore_version> \
      --subnet-ids <subnet_IDs> \
      --security-group-ids <security_group_IDs> \
      --resource-preset-id <ID_of_computing_resources> \
      --maintenance-window type=<maintenance_type>,`
                           `day=<day_of_week>,`
                           `hour=<hour> \
      --deletion-protection \
      --log-enabled \
      --log-folder-id <folder_ID> \
      --log-min-level <logging_level>
   

   Where:

   • --name: Cluster name.

   • --description: Cluster description.

   • --labels: List of labels. Provide labels in <key>=<value> format.

   • --service-account-id: Service account ID.

   • --version: Apache Hive™ Metastore version.

     Available versions: 3.1 and 4.0.

     Warning

     To integrate the Apache Hive™ Metastore cluster with Yandex Managed Service for Trino and Yandex Managed Service for Apache Spark™, you need version 3.1.

     If required, you can upgrade 3.1 to 4.0, but you cannot downgrade 4.0 to 3.1.

   • --subnet-ids: List of subnet IDs.

   • --security-group-ids: List of security group IDs.

   • --resource-preset-id: Computing resource configuration.

   • --maintenance-window: Maintenance window settings (including for disabled clusters), where type is the maintenance type:

     • anytime: At any time (default).
     • weekly: On a schedule. For this value, also specify the following:
       • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
       • hour: Hour of day (UTC), from 1 to 24.

   • --deletion-protection: Enables cluster protection against accidental deletion.

   • Logging parameters:

     • --log-enabled: Enables logging. Logs generated by Apache Hive™ Metastore components will be sent to Yandex Cloud Logging.

     • --log-folder-id: Folder ID. Logs will be written to the default log group for this folder.

     • --log-group-id: Custom log group ID. Logs will be written to this group.

       Specify one of the two parameters: --log-folder-id or --log-group-id.

     • --log-min-level: Minimum logging level. Possible values: TRACE, DEBUG, INFO (default), WARN, ERROR, and FATAL.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a Apache Hive™ Metastore cluster:

1. In the configuration file, describe the resources you are creating:

   • Apache Hive™ Metastore cluster: Cluster description.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   
   resource "yandex_metastore_cluster" "<cluster_name>" {
     name                = "<cluster_name>"
     subnet_ids          = [yandex_vpc_subnet.<subnet_name>.id]
     security_group_ids  = [<list_of_security_group_IDs>]
     service_account_id  = "<service_account_ID>"
     deletion_protection = <protect_cluster_from_deletion>
     version             = "<version>"
   
     cluster_config = {
       resource_preset_id = "<class_of_computing_resources>"
     }
   
     maintenance_window = {
       type = "<maintenance_type>"
       day  = "<day_of_week>"
       hour = <hour>
     }
   
     logging = {
       enabled   = <enable_logging>
       folder_id = "<folder_ID>"
       min_level = "<logging_level>"
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • name: Cluster name.

   • subnet_ids: List of subnet IDs.

   • security_group_ids: List of security group IDs.

   • service_account_id: Service account ID.

   • deletion_protection: Enables cluster protection against accidental deletion. The possible values are true or false.

   • version: Apache Hive™ Metastore version.

     Available versions: 3.1 and 4.0.

     Warning

     To integrate the Apache Hive™ Metastore cluster with Yandex Managed Service for Trino and Yandex Managed Service for Apache Spark™, you need version 3.1.

     If required, you can upgrade 3.1 to 4.0, but you cannot downgrade 4.0 to 3.1.

   • cluster_config.resource_preset_id: Computing resource configuration.

   • maintenance_window: Maintenance window settings, including those for disabled clusters.

     • type: Maintenance type. The possible values include:
       • ANYTIME: Any time.
       • WEEKLY: On a schedule.
     • day: Day of week for the WEEKLY type, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: UTC hour for the WEEKLY type, from 1 to 24.

   • logging: Logging parameters:

     • enable: Enables logging. Logs generated by Apache Hive™ Metastore components will be sent to Yandex Cloud Logging. The possible values are true or false.

     • folder_id: Folder ID. Logs will be written to the default log group for this folder.

     • group_id: Custom log group ID. Logs will be written to this group.

       Specify one of the two parameters: folder_id or group_id.

     • min_level: Minimum logging level. Possible values: TRACE, DEBUG, INFO (default), WARN, ERROR, and FATAL.

2. Validate your configuration.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Confirm resource changes.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see this provider guide.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Create a file named body.json and paste the following code into it:

   {
     "folderId": "<folder_ID>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "labels": { "<label_list>" },
     "deletionProtection": <deletion_protection>,
     "version": "<Apache_Hive™_Metastore_version>",
     "configSpec": {
       "resources": {
       "resourcePresetId": "<resource_configuration_ID>"
       }
     },
     "serviceAccountId": "<service_account_ID>",
     "logging": {
       "enabled": <use_of_logging>,
       "folderId": "<folder_ID>",
       "minLevel": "<logging_level>"
     },
     "network": {
       "subnetIds": [ "<list_of_subnet_IDs>" ],
       "securityGroupIds": [ "<list_of_security_group_IDs>" ]
     },
     "maintenanceWindow": {
       "weeklyMaintenanceWindow": {
       "day": "<day_of_week>",
       "hour": "<hour>"
       }
     }
   }
   

   Where:

   • folderId: Folder ID. You can get it with the list of folders in the cloud.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels provided in "<key>": "<value>" format.

   • deletionProtection: Enables cluster protection against accidental deletion. The possible values are true or false.

   • version: Apache Hive™ Metastore version.

     Available versions: 3.1 and 4.0.

     Warning

     To integrate the Apache Hive™ Metastore cluster with Yandex Managed Service for Trino and Yandex Managed Service for Apache Spark™, you need version 3.1.

     If required, you can upgrade 3.1 to 4.0, but you cannot downgrade 4.0 to 3.1.

   • configSpec.resources.resourcePresetId: Computing resource configuration.

   • serviceAccountId: Service account ID.

   • logging: Logging parameters:

     • enabled: Enables logging. Logs generated by Apache Hive™ Metastore components will be sent to Yandex Cloud Logging. The possible values are true or false.

     • folderId: Folder ID. Logs will be written to the default log group for this folder.

     • logGroupId: Custom log group ID. Logs will be written to this group.

       Specify either folderId or logGroupId.

     • minLevel: Minimum logging level. The possible values are TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.

   • network: Network settings:

     • subnetIds: List of subnet IDs.
     • securityGroupIds: List of security group IDs.

   • maintenanceWindow: Maintenance window settings (including for disabled clusters). In maintenanceWindow, provide one of the two parameters:

     • anytime: Maintenance can take place at any time.

     • weeklyMaintenanceWindow: Maintenance takes place once a week at the specified time:

       • day: Day of week, in DDD format, MON, TUE, WED, THU, FRI, SAT, or SUN.
       • hour: Time of day (UTC) in HH format, from 1 to 24.

3. Use the Cluster.Create method and send the following request, e.g., via cURL:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://metastore.api.cloud.yandex.net/managed-metastore/v1/clusters' \
       --data '@body.json'
   

4. View the server response to make sure your request was successful.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "folder_id": "<folder_ID>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "labels": "{ <label_list> }",
     "deletion_protection": <deletion_protection>,
     "version": "<Apache_Hive™_Metastore_version>",
     "config_spec": {
       "resources": {
         "resource_preset_id": "<resource_configuration_ID>"
       }
     },
     "service_account_id": "<service_account_ID>",
     "logging": {
       "enabled": <use_of_logging>,
       "folder_id": "<folder_ID>",
       "min_level": "<logging_level>"
     },
     "network": {
       "subnet_ids": [ "<list_of_subnet_IDs>" ],
       "security_group_ids": [ "<list_of_security_group_IDs>" ]
     },
     "maintenance_window": {
       "weekly_maintenance_window": {
         "day": "<day_of_week>",
         "hour": "<hour>"
       }
     }
   }
   

   Where:

   • folder_id: Folder ID. You can get it with the list of folders in the cloud.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels provided in "<key>": "<value>" format.

   • deletion_protection: Enables cluster protection against accidental deletion. The possible values are: true or false.

   • version: Apache Hive™ Metastore version.

     Available versions: 3.1 and 4.0.

     Warning

     To integrate the Apache Hive™ Metastore cluster with Yandex Managed Service for Trino and Yandex Managed Service for Apache Spark™, you need version 3.1.

     If required, you can upgrade 3.1 to 4.0, but you cannot downgrade 4.0 to 3.1.

   • config_spec.resources.resource_preset_id: Computing resource configuration.

   • service_account_id: Service account ID.

   • logging: Logging parameters:

     • enabled: Enables logging. Logs generated by Apache Hive™ Metastore components will be sent to Yandex Cloud Logging. The possible values are: true or false.

     • folder_id: Folder ID. Logs will be written to the default log group for this folder.

     • log_group_id: Custom log group ID. Logs will be written to this group.

       Specify one of the two parameters: folder_id or log_group_id.

     • min_level: Minimum logging level. The possible values are: TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.

   • network: Network settings:

     • subnet_ids: List of subnet IDs.
     • security_group_ids: List of security group IDs.

   • maintenance_window: Maintenance window settings (including for disabled clusters). In maintenance_window, provide one of the two parameters:

     • anytime: Maintenance can take place at any time.

     • weekly_maintenance_window: Maintenance takes place once a week at the specified time:

       • day: Day of week, in DDD format, MON, TUE, WED, THU, FRI, SAT, or SUN.
       • hour: Time of day (UTC) in HH format, from 1 to 24.

4. Use the ClusterService.Create call and send the following request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/metastore/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       metastore.api.cloud.yandex.net:443 \
       yandex.cloud.metastore.v1.ClusterService.Create \
       < body.json
   

5. View the server response to make sure your request was successful.