Discuss with expertTry it for free
© 2026 Direct Cursus Technology L.L.C.

Managing subclusters Yandex Data Processing

Written by
Updated at June 23, 2026

In addition to updating the settings of a particular subcluster, you can create new and delete existing subclusters.

Warning

Each cluster can only have one subcluster with a master host, so you cannot create or delete subclusters with this role. You cannot delete data storage subclusters either.

Getting a list of subclusters in a cluster

  1. Open the folder dashboard.
  2. Navigate to Yandex Data Processing.
  3. Click the name of your cluster and select the Subclusters tab.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To get a list of Yandex Data Processing subclusters in a cluster, run the following command:

yc dataproc subcluster list --cluster-name=<cluster_name>

You can get the cluster name with the list of clusters in the folder.

Creating a subcluster

The number of hosts in Yandex Data Processing clusters is limited by quotas on the number of vCPUs and the amount of RAM available to virtual machines in your cloud. To view the available resources, navigate to Quotas and find Compute Cloud.

  1. In the management console, select the relevant folder.
  2. Navigate to Yandex Data Processing and select the cluster.
  3. Navigate to Subclusters.
  4. Click Create subcluster.
  5. Specify the subcluster settings:

    1. Hosts: Select the number of hosts.

    2. Roles: Select the subcluster roles depending on the services to deploy on the hosts:

      • COMPUTENODE: Role for processing data. In subclusters with this role, you can deploy YARN NodeManager and Spark libraries.
      • DATANODE: Role for storing data. In subclusters with this role, you can deploy YARN NodeManager, Spark libraries, HBase RegionServer, and HDFS Datanode.

    3. Under Host class, select a platform and computing resources available to the host.

    4. Under Storage size, specify the storage type and size.

    5. Under Network settings:

      • Select Network ID format.

      • Specify Subnet (subnet of the network where your cluster resides).

      • Optionally, enable Public access to allow access to subcluster hosts from the internet.

        You will not be able to change this setting after the subcluster is created.

        Tip

        You can delete data processing subclusters and recreate them with the required configuration.

    6. Optionally, enable Autoscaling.

  6. Click Add subcluster.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To create a subcluster:

  1. See the description of the CLI command for creating a subcluster:

    yc dataproc subcluster create --help

  2. Specify the subcluster settings in the create command below (the example does not show all available settings):

    yc dataproc subcluster create <subcluster_name> \
  --cluster-name=<cluster_name> \
  --role=<subcluster_role> \
  --resource-preset=<host_class> \
  --disk-type=<storage_type> \
  --disk-size=<storage_size_in_GB> \
  --subnet-name=<subnet_name> \
  --hosts-count=<number_of_hosts>

    Where:

    • --cluster-name: Cluster name. You can get the cluster name with the list of clusters in the folder.
    • --role: Subcluster role, datanode or computenode.
    • --resource-preset: Host class.
    • --disk-type: Storage type, which can be network-ssd, network-hdd, or network-ssd-nonreplicated.
    • --disk-size: Storage size, in GB.
    • --subnet-name: Subnet name.
    • --hosts-count: Number of subcluster hosts. The minimum value is 1, and the maximum value is 32.

  1. Open the current Terraform configuration file with the infrastructure plan.

    For more on how to create this file, see Creating a cluster.

  2. In the Yandex Data Processing cluster description, add a subcluster_spec section containing the settings for the new subcluster:

    resource "yandex_dataproc_cluster" "data_cluster" {
  ...
  cluster_config {
    ...
    subcluster_spec {
      name = "<subcluster_name>"
      role = "<subcluster_role>"
      resources {
        resource_preset_id = "<host_class>"
        disk_type_id       = "<storage_type>"
        disk_size          = <storage_size_in_GB>
      }
      subnet_id   = "<subnet_ID>"
      hosts_count = <number_of_hosts_in_subcluster>
      ...
    }
  }
}

    Where role is the subcluster role, COMPUTENODE or DATANODE.

  3. Make sure the settings are correct.

    1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

    2. Run this command:

      terraform validate

      Terraform will show any errors found in your configuration files.

  4. Confirm updating the resources.

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see this provider guide.

Deleting a subcluster

Warning

You cannot delete data storage subclusters.

To delete a subcluster:

  1. In the management console, select the relevant folder.
  2. Navigate to Yandex Data Processing and select the cluster.
  3. Navigate to Subclusters.
  4. Click for the subcluster in question and select Delete.
  5. Optionally, specify the decommissioning timeout.
  6. In the window that opens, click Delete.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To delete a subcluster in a Yandex Data Processing cluster, run this command:

yc dataproc subcluster delete <subcluster_name_or_ID> \
  --cluster-name=<cluster_name>

You can request the subcluster name or ID with the list of cluster subclusters, and the cluster name, with the list of folder clusters.

  1. Open the current Terraform configuration file with the infrastructure plan.

    For more on how to create this file, see Creating a cluster.

  2. Delete the subcluster_spec section of the relevant subcluster from the Yandex Data Processing cluster description.

  3. Make sure the settings are correct.

    1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

    2. Run this command:

      terraform validate

      Terraform will show any errors found in your configuration files.

  4. Type yes and press Enter.

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information about the resources you can create with Terraform, see this provider guide.

Previous
Deleting a cluster
Next
Updating subclusters