Contact UsGet started
© 2024 Direct Cursus Technology L.L.C.

Updating subclusters

Written by
Updated at December 16, 2024

You can perform the following actions for any subcluster:

You can also change subclusters' availability zone. In which case they will be migrated to a different availability zone together with the Yandex Data Processing cluster. The process depends on the cluster type:

Changing the number of hosts

You can change the number of hosts in data storage and processing subclusters:

  1. Go to the folder page and select Yandex Data Processing.
  2. Click the cluster name and open the Subclusters tab.
  3. Click for the subcluster you need and select Edit.
  4. Enter or select the required number of hosts in the Hosts field.
  5. (Optional) Specify the decommissioning timeout.
  6. Click Save changes.

Yandex Data Processing will run the add host operation.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the number of hosts for a subcluster:

  1. View a description of the CLI update subcluster command:

    yc dataproc subcluster update --help

  2. Set a new number of hosts in the update subcluster command:

    yc dataproc subcluster update <subcluster_name_or_ID> \
   --cluster-name=<cluster_name> \
   --hosts-count=<number_of_hosts>

    You can request a subcluster name or ID with a list of cluster subclusters, and a cluster name with a list of folder clusters.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. In the Yandex Data Processing cluster description, edit the value of the hosts_count parameter under subcluster_spec for the relevant data storage or processing subcluster:

    resource "yandex_dataproc_cluster" "data_cluster" {
  ...
  cluster_config {
    ...
    subcluster_spec {
      name        = "<subcluster_name>"
      ...
      hosts_count = <number_of_hosts_in_subcluster>
    }
  }
}

Changing the host class

Warning

Changing host properties through the Yandex Compute Cloud interfaces may result in host failure. To change the cluster host settings, use the Yandex Data Processing interfaces, such as the management console, CLI, Terraform, or API.

You can change the computing power of hosts in a separate subcluster. It depends on the driver deploy mode:

  • In deployMode=cluster mode, when the driver is deployed on one of the cluster's compute hosts, 4–8 CPU cores and 16 GB RAM are sufficient for the subcluster with the master host.
  • In deployMode=client mode, when the driver is deployed on the cluster's master host, the computing power depends on the job logic and the number of concurrently running jobs.

For more information about driver deploy modes and computing resource consumption, see Resource allocation.

To change the host class for a subcluster:

  1. In the management console, select the folder with the cluster whose subcluster you want to change.
  2. Select Yandex Data Processing and the required cluster.
  3. Go to Subclusters.
  4. Click for the subcluster you need and select Edit.
  5. Select the required platform and configuration under Host class.
  6. (Optional) Specify the decommissioning timeout.
  7. Click Save changes.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the host class for a subcluster:

  1. View a description of the CLI update subcluster command:

    yc dataproc subcluster update --help

  2. Request a list of available host classes (the ZONE IDS column specifies the availability zones where you can select the appropriate class):

    yc dataproc resource-preset list

    Result:

    +-----------+--------------------------------+-------+----------+
|    ID     |            ZONE IDS            | CORES |  MEMORY  |
+-----------+--------------------------------+-------+----------+
| b3-c1-m4  | ru-central1-a, ru-central1-b,  |     2 | 4.0 GB   |
|           | ru-central1-c                  |       |          |
| ...                                                           |
+-----------+--------------------------------+-------+----------+

  3. Specify the class in the update subcluster command:

    yc dataproc subcluster update <subcluster_name_or_ID> \
   --cluster-name=<cluster_name> \
   --resource-preset=<host_class>

    You can request a subcluster name or ID with a list of cluster subclusters, and a cluster name with a list of folder clusters.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. In the Yandex Data Processing cluster description, change the resource_preset_id parameter value under subcluster_spec.resources of the relevant subcluster:

    resource "yandex_dataproc_cluster" data_cluster {
  ...
  cluster_config {
    ...
    subcluster_spec {
      name = "<subcluster_name>"
      ...
      resources {
        resource_preset_id = "<subcluster_host_class>"
      ...
    }
  }
}

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see the provider documentation.

Yandex Data Processing will run the update subcluster operation. Note that all the hosts in the updated subcluster will be restarted.

Changing the autoscaling rule for data processing subclusters

You can configure the autoscaling rule in data processing subclusters.

Make sure the cloud quota is sufficient to increase the VM resources. Open the Quotas page for your cloud and check that the Compute Cloud section still has space available in the following lines:

  • Total HDD capacity
  • Total SSD capacity
  • Number of disks
  • Number of vCPUs for instances
  • Number of instances

To enable autoscaling, make sure the Yandex Data Processing cluster service account has the following roles assigned:

To configure autoscaling for subclusters:

  1. Go to the folder page and select Yandex Data Processing.
  2. Select a cluster and open the Subclusters tab.
  3. Click for the subcluster you need and select Edit.
  4. Under Scaling, enable Autoscaling if it is disabled.
  5. Set autoscaling parameters.
  6. The default metric used for autoscaling is yarn.cluster.containersPending. To enable scaling based on CPU utilization, disable the Default scaling setting and specify the target CPU utilization level.
  7. Click Save changes.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To configure autoscaling for subclusters:

  1. View a description of the CLI update subcluster command:

    yc dataproc subcluster update --help

  2. Set the autoscaling parameters in the update subcluster command:

    yc dataproc subcluster update <subcluster_name_or_ID> \
   --cluster-name=<cluster_name> \
   --hosts-count=<minimum_number_of_hosts> \
   --max-hosts-count=<maximum_number_of_hosts> \
   --enable-preemptible=<use_preemptible_VMs> \
   --warmup-duration=<VM_warmup_period> \
   --stabilization-duration=<stabilization_period> \
   --measurement-duration=<load_measurement_interval> \
   --cpu-utilization-target=<target_CPU_utilization_level> \
   --autoscaling-decommission-timeout=<decommissioning_timeout>

    Where:

    • --hosts-count: Minimum number of hosts (VMs) in a subcluster. The minimum value is 1, and the maximum value is 32.
    • --max-hosts-count: Maximum number of hosts (VMs) in a subcluster. The minimum value is 1, and the maximum value is 100.
    • --enable-preemptible: Indicates if preemptible VMs are used. It can either be true or false.
    • --warmup-duration: Time required to warm up a VM instance, in <value>s format. The minimum value is 0s, and the maximum value is 600s.
    • --stabilization-duration: Period, in seconds, during which the required number of VMs cannot be decreased, in <value>s format. The minimum value is 60s and the maximum value is 1800s.
    • --measurement-duration: Period, in seconds, for which the average utilization is calculated for each VM, in <value>s format. The minimum value is 60s (1 minute), and the maximum value is 600s (10 minutes).
    • --cpu-utilization-target: Target CPU utilization level, in %. Use this setting to enable scaling based on CPU utilization. Otherwise, yarn.cluster.containersPending will be used for scaling based on the number of pending resources. The minimum value is 10, and the maximum value is 100.
    • --autoscaling-decommission-timeout: Decommissioning timeout in seconds. The minimum value is 0, and the maximum value is 86400 (24 hours).

    You can request a subcluster name or ID with a list of cluster subclusters, and a cluster name with a list of folder clusters.

To configure autoscaling for subclusters:

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. In the Yandex Data Processing cluster description, add a section named subcluster_spec.autoscaling_config with the required autoscaling settings for the relevant subcluster:

    resource "yandex_dataproc_cluster" "data_cluster" {
  ...
  cluster_config {
    ...
    subcluster_spec {
      name = "<subcluster_name>"
      role = "COMPUTENODE"
      ...
      autoscaling_config {
      max_hosts_count        = <maximum_number_of_VMs_in_group>
      measurement_duration   = <load_measurement_interval>
      warmup_duration        = <warmup_period>
      stabilization_duration = <stabilization_period>
      preemptible            = <use_preemptible_VMs>
      cpu_utilization_target = <target_CPU_utilization_level>
      decommission_timeout   = <decommissioning_timeout>
    }
  }

    Where:

    • max_hosts_count: Maximum number of hosts (VMs) in a subcluster. The minimum value is 1, and the maximum value is 100.
    • measurement_duration: Period, in seconds, for which the average utilization is calculated for each VM, in <value>s format. The minimum value is 60s (1 minute), and the maximum value is 600s (10 minutes).
    • warmup_duration: Time required to warm up a VM instance, in <value>s format. The minimum value is 0s, and the maximum value is 600s.
    • stabilization_duration: Period, in seconds, during which the required number of VMs cannot be decreased, in <value>s format. The minimum value is 60s and the maximum value is 1800s.
    • preemptible: Indicates if preemptible VMs are used. It can either be true or false.
    • cpu_utilization_target: Target CPU utilization level, in %. Use this setting to enable scaling based on CPU utilization. Otherwise, yarn.cluster.containersPending will be used for scaling based on the number of pending resources. The minimum value is 10, and the maximum value is 100.
    • decommission_timeout: Decommissioning timeout in seconds. The minimum value is 0, and the maximum value is 86400 (24 hours).

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see the provider documentation.

Increasing storage size

Warning

Changing host properties through the Yandex Compute Cloud interfaces may result in host failure. To change the cluster host settings, use the Yandex Data Processing interfaces, such as the management console, CLI, Terraform, or API.

You can increase the amount of storage available to each host in a particular subcluster.

Note

Currently, you cannot reduce storage size. If necessary, re-create the Yandex Data Processing subcluster.

Make sure the cloud quota is sufficient to increase the VM resources. Open the Quotas page for your cloud and check that the Compute Cloud section still has space available in the following lines:

  • Total HDD capacity.
  • Total SSD capacity.
  • Number of disks.

To change the storage size for a subcluster:

  1. In the management console, select the folder with the cluster whose subcluster you want to change.
  2. Select Yandex Data Processing and the required cluster.
  3. Go to Subclusters.
  4. Click for the subcluster you need and select Edit.
  5. Enter or select the required amount of storage under Size of storage.
  6. Click Save changes.

Yandex Data Processing will run the update subcluster operation.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the storage size for a subcluster:

  1. View a description of the CLI update subcluster command:

    yc dataproc subcluster update --help

  2. Specify the required storage size in the update subcluster command:

    yc dataproc subcluster update <subcluster_name_or_ID> \
   --cluster-name=<cluster_name> \
   --disk-size=<storage_size_in_GB>

    You can request a subcluster name or ID with a list of cluster subclusters, and a cluster name with a list of folder clusters.

If all these conditions are met, Yandex Data Processing launches the operation to increase storage space.

To increase the subcluster storage size:

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. In the Yandex Data Processing cluster description, change the disk_size parameter value under subcluster_spec.resources of the relevant subcluster:

    resource "yandex_dataproc_cluster" "data_cluster" {
  ...
  cluster_config {
    ...
    subcluster_spec {
      name = "<subcluster_name>"
      ...
      resources {
        disk_size = <storage_size_in_GB>
        ...
      }
    }
  }
}

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see the provider documentation.

Changing security groups

  1. Go to the folder page and select Yandex Data Processing.
  2. Click the cluster name and open the Hosts tab.
  3. Click the host name.
  4. Under Network, click and select Edit.
  5. Select the appropriate security groups.
  6. Click Save.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Change the value of the security_group_ids parameter in the cluster description:

    resource "yandex_dataproc_cluster" "data_cluster" {
  ...
  security_group_ids = [ "<list_of_cluster_security_group_IDs>" ]
}

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see the provider documentation.

Warning

You may need to additionally set up security groups to connect to the cluster.

Previous
Managing subclusters
Next
Connecting to component interfaces