Updating a subcluster

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To change the number of hosts for a subcluster:

1. View the description of the CLI command for updating a subcluster:

   yc dataproc subcluster update --help
   

2. Run this command to update the subcluster, specifying the new number of hosts:

   yc dataproc subcluster update <subcluster_name_or_ID> \
      --cluster-name=<cluster_name> \
      --hosts-count=<number_of_hosts>
   

   You can get the subcluster ID and name with the list of subclusters in the cluster, and the cluster name, with the list of clusters in the folder.

1. Open the current Terraform configuration file with the infrastructure plan.

   To learn how to create this file, refer to Creating a cluster.

2. In the description of the Yandex Data Processing cluster, edit the hosts_count value under subcluster_spec for your data storage or data processing subcluster:

   resource "yandex_dataproc_cluster" "data_cluster" {
     ...
     cluster_config {
       ...
       subcluster_spec {
         name        = "<subcluster_name>"
         ...
         hosts_count = <number_of_hosts_in_subcluster>
       }
     }
   }
   

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To change the host class for a subcluster:

1. View the description of the CLI command for updating a subcluster:

   yc dataproc subcluster update --help
   

2. Get the list of available host classes (the ZONE IDS column lists the availability zones you can select each class in):

   yc dataproc resource-preset list
   

   Result:

   +-----------+--------------------------------+-------+----------+
   |    ID     |            ZONE IDS            | CORES |  MEMORY  |
   +-----------+--------------------------------+-------+----------+
   | b3-c1-m4  | ru-central1-a, ru-central1-b,  |     2 | 4.0 GB   |
   |           | ru-central1-c                  |       |          |
   | ...                                                           |
   +-----------+--------------------------------+-------+----------+
   

3. Run this command to update the subcluster, specifying the new host class:

   yc dataproc subcluster update <subcluster_name_or_ID> \
      --cluster-name=<cluster_name> \
      --resource-preset=<host_class>
   

   You can request the subcluster name or ID with the list of cluster subclusters, and the cluster name, with the list of folder clusters.

1. Open the current Terraform configuration file with the infrastructure plan.

   To learn how to create this file, refer to Creating a cluster.

2. In the description of the Yandex Data Processing cluster, edit the resource_preset_id value under subcluster_spec.resources for your subcluster:

   resource "yandex_dataproc_cluster" data_cluster {
     ...
     cluster_config {
       ...
       subcluster_spec {
         name = "<subcluster_name>"
         ...
         resources {
           resource_preset_id = "<subcluster_host_class>"
         ...
       }
     }
   }
   

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm resource changes.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see this provider guide.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To configure autoscaling for subclusters:

1. View the description of the CLI command for updating a subcluster:

   yc dataproc subcluster update --help
   

2. Run this command to update the subcluster, specifying the autoscaling settings:

   yc dataproc subcluster update <subcluster_name_or_ID> \
      --cluster-name=<cluster_name> \
      --hosts-count=<minimum_number_of_hosts> \
      --max-hosts-count=<maximum_number_of_hosts> \
      --enable-preemptible=<using_preemptible_VMs> \
      --warmup-duration=<VM_warmup_period> \
      --stabilization-duration=<stabilization_period> \
      --measurement-duration=<load_measurement_interval> \
      --cpu-utilization-target=<target_CPU_utilization> \
      --autoscaling-decommission-timeout=<decommissioning_timeout>
   

   Where:

   • --hosts-count: Minimum number of hosts (VMs) in a subcluster. The minimum value is 1, and the maximum value is 32.
   • --max-hosts-count: Maximum number of hosts (VMs) in a subcluster. The minimum value is 1, and the maximum value is 100.
   • --enable-preemptible: Specifies if preemptible VMs are used. It can be either true or false.
   • --warmup-duration: Time required to warm up a VM, in <value>s format. The minimum value is 0s, and the maximum value is 600s.
   • --stabilization-duration: Period, in seconds, during which the required number of VMs cannot be decreased, in <value>s format. The minimum value is 60s and the maximum value is 1800s.
   • --measurement-duration: Period, in seconds, for which average utilization is calculated for each VM, in <value>s format. The minimum value is 60s (1 minute), and the maximum value is 600s (10 minutes).
   • --cpu-utilization-target: Target CPU utilization, in %. Use this setting to enable scaling based on CPU utilization. Otherwise, yarn.cluster.containersPending will be used for scaling based on the number of pending resources. The minimum value is 10, and the maximum value is 100.
   • --autoscaling-decommission-timeout: Decommissioning timeout, in seconds. The minimum value is 0, and the maximum value is 86400 (24 hours).

   You can get the subcluster ID and name with the list of subclusters in the cluster, and the cluster name, with the list of clusters in the folder.

To configure autoscaling for subclusters:

1. Open the current Terraform configuration file with the infrastructure plan.

   To learn how to create this file, refer to Creating a cluster.

2. In the description of the Yandex Data Processing cluster, add a section named subcluster_spec.autoscaling_config with the required autoscaling settings for your subcluster:

   resource "yandex_dataproc_cluster" "data_cluster" {
     ...
     cluster_config {
       ...
       subcluster_spec {
         name = "<subcluster_name>"
         role = "COMPUTENODE"
         ...
         autoscaling_config {
         max_hosts_count        = <maximum_number_of_VMs_in_group>
         measurement_duration   = <load_measurement_interval>
         warmup_duration        = <warmup_period>
         stabilization_duration = <stabilization_period>
         preemptible            = <use_of_preemptible_VMs>
         cpu_utilization_target = <target_CPU_utilization>
         decommission_timeout   = <decommissioning_timeout>
       }
     }
   

   Where:

   • max_hosts_count: Maximum number of hosts (VMs) in a subcluster. The minimum value is 1, and the maximum value is 100.
   • measurement_duration: Period, in seconds, for which average utilization is calculated for each VM, in <value>s format. The minimum value is 60s (1 minute), and the maximum value is 600s (10 minutes).
   • warmup_duration: Time required to warm up a VM, in <value>s format. The minimum value is 0s, and the maximum value is 600s.
   • stabilization_duration: Period, in seconds, during which the required number of VMs cannot be decreased, in <value>s format. The minimum value is 60s and the maximum value is 1800s.
   • preemptible: Indicates if preemptible VMs are used. It can be either true or false.
   • cpu_utilization_target: Target CPU utilization, in %. Use this setting to enable scaling based on CPU utilization. Otherwise, yarn.cluster.containersPending will be used for scaling based on the number of pending resources. The minimum value is 10, and the maximum value is 100.
   • decommission_timeout: Decommissioning timeout, in seconds. The minimum value is 0, and the maximum value is 86400 (24 hours).

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm resource changes.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see this provider guide.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To change the storage size for a subcluster:

1. View the description of the CLI command for updating a subcluster:

   yc dataproc subcluster update --help
   

2. Run this command to update the subcluster, specifying the storage size you need:

   yc dataproc subcluster update <subcluster_name_or_ID> \
      --cluster-name=<cluster_name> \
      --disk-size=<storage_size_in_GB>
   

   You can request the subcluster name or ID with the list of cluster subclusters, and the cluster name, with the list of folder clusters.

If all these conditions are met, Yandex Data Processing starts expanding the storage.

To expand the subcluster storage:

1. Open the current Terraform configuration file with the infrastructure plan.

   To learn how to create this file, refer to Creating a cluster.

2. In the description of the Yandex Data Processing cluster, edit the disk_size value under subcluster_spec.resources for your subcluster:

   resource "yandex_dataproc_cluster" "data_cluster" {
     ...
     cluster_config {
       ...
       subcluster_spec {
         name = "<subcluster_name>"
         ...
         resources {
           disk_size = <storage_size_in_GB>
           ...
         }
       }
     }
   }
   

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm resource changes.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see this provider guide.

1. Open the current Terraform configuration file with the infrastructure plan.

   To learn how to create this file, refer to Creating a cluster.

2. Edit the security_group_ids value in the cluster description:

   resource "yandex_dataproc_cluster" "data_cluster" {
     ...
     security_group_ids = [ "<list_of_cluster_security_group_IDs>" ]
   }
   

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm resource changes.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see this provider guide.