Creating a Yandex Data Processing cluster

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a Yandex Data Processing cluster:

1. Check whether the folder has any subnets for the Yandex Data Processing cluster hosts:

   yc vpc subnet list
   

   If there are no subnets in the folder, create the required subnets in Yandex Virtual Private Cloud.

2. View the description of the CLI command to create a Yandex Data Processing cluster:

   yc dataproc cluster create --help
   

3. Specify Yandex Data Processing cluster parameters in the create command (the list of supported parameters in the example is not exhaustive):

   yc dataproc cluster create <cluster_name> \
     --bucket=<bucket_name> \
     --zone=<availability_zone> \
     --service-account-name=<service_account_name> \
     --version=<image_version> \
     --services=<list_of_components> \
     --ssh-public-keys-file=<path_to_public_SSH_key> \
     --subcluster name=<name_of_subcluster_with_master_host>,`
                  `role=masternode,`
                  `resource-preset=<host_class>,`
                  `disk-type=<storage_type>,`
                  `disk-size=<storage_size_in_GB>,`
                  `subnet-name=<subnet_name>,`
                  `assign-public-ip=<public_access_to_subcluster_host> \
     --subcluster name=<name_of_subcluster_for_storing_data>,`
                  `role=datanode,`
                  `resource-preset=<host_class>,`
                  `disk-type=<storage_type>,`
                  `disk-size=<storage_size_in_GB>,`
                  `subnet-name=<subnet_name>,`
                  `hosts-count=<number_of_hosts>,`
                  `assign-public-ip=<public_access_to_subcluster_host> \
     --deletion-protection \
     --ui-proxy=<access_to_component_web_interfaces> \
     --log-group-id=<log_group_ID> \
     --security-group-ids=<list_of_security_group_IDs>
   

   Note

   The Yandex Data Processing cluster name must be unique within the folder. It may contain Latin letters, numbers, hyphens, and underscores. The name may be up to 63 characters long.

   Where:

   • --bucket: Name of an Object Storage bucket where job dependencies and results will be stored. The service account of the Yandex Data Processing cluster must have READ and WRITE permissions for this bucket.

   • --zone: Availability zone where the Yandex Data Processing cluster hosts will reside.

   • --service-account-name: Name of the Yandex Data Processing cluster service account. Make sure to assign the following roles to the Yandex Data Processing cluster service account:

     • dataproc.agent: To enable the service account to get info on cluster host states, jobs, and log groups.
     • dataproc.provisioner: To enable the service account to work with an autoscaling instance group. This will enable subcluster autoscaling.

   • --version: Image version.

     Using an image of version 2.0.39 or higher, you can create a lightweight cluster without HDFS and subclusters for data storage. In this case, be sure to add one or more data processing subclusters and specify a bucket name.

     Tip

     To use the most recent image version, specify 2.0 in the --version parameter.

   • --services: List of components that you want to use in the Yandex Data Processing cluster. If you do not specify this parameter, these default components will be used: yarn, tez, and spark.

   • --ssh-public-keys-file: Full path to the file with the public part of the SSH key for access to the Yandex Data Processing cluster hosts.For information about how to generate and use SSH keys, see the Yandex Compute Cloud documentation.

   • --subcluster: Parameters of Yandex Data Processing subclusters:

     • name: Yandex Data Processing subcluster name.

     • role: Yandex Data Processing subcluster role, which can be masternode, datanode, or computenode.

     • resource-preset: Host class.

     • disk-type: Storage type, which can be network-ssd, network-hdd, or network-ssd-nonreplicated.

     • disk-size: Storage size in GB.

     • subnet-name: Subnet name.

     • hosts-count: Number of hosts in the Yandex Data Processing subclusters for data storage or processing. The minimum value is 1, and the maximum value is 32.

     • assign-public-ip: Access to Yandex Data Processing subcluster hosts from the internet. It can either be true or false. If access is enabled, you can only connect to the Yandex Data Processing cluster using SSL. For more information, see Connecting to a Yandex Data Processing cluster.

       Warning

       After you create a Yandex Data Processing cluster, you cannot request or disable public access to the subcluster. However, you can delete the Yandex Data Processing subcluster for data processing and create it again with the public access settings you need.

   • --deletion-protection: Protection against accidental deletion of the Yandex Data Processing cluster.

     Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

   • --ui-proxy: Access to Yandex Data Processing component web interfaces. It can either be true or false.

   • target_group_id: Log group ID.

   • --security-group-ids: List of security group IDs.

   To create a Yandex Data Processing cluster with multiple data storage or processing subclusters, provide the required number of --subcluster arguments in the cluster creation command:

   yc dataproc cluster create <cluster_name> \
     ...
     --subcluster <subcluster_parameters> \
     --subcluster <subcluster_parameters> \
     ...
   

4. To enable autoscaling in Yandex Data Processing subclusters for data processing, specify the following parameters:

   yc dataproc cluster create <cluster_name> \
     ...
     --subcluster name=<subcluster_name>,`
                  `role=computenode`
                  `...`
                  `hosts-count=<minimum_number_of_hosts>`
                  `max-hosts-count=<maximum_number_of_hosts>,`
                  `preemptible=<use_preemptible_VMs>,`
                  `warmup-duration=<VM_warmup_period>,`
                  `stabilization-duration=<stabilization_period>,`
                  `measurement-duration=<load_measurement_interval>,`
                  `cpu-utilization-target=<target_CPU_utilization_level>,`
                  `autoscaling-decommission-timeout=<decommissioning_timeout>
   

   Where:

   • hosts-count: Minimum number of hosts (VMs) in the Yandex Data Processing subcluster. The minimum value is 1, and the maximum value is 32.
   • max-hosts-count: Maximum number of hosts (VMs) in the Yandex Data Processing subcluster. The minimum value is 1, and the maximum value is 100.
   • preemptible: Indicates if preemptible VMs are used. It can either be true or false.
   • warmup-duration: Time required to warm up a VM instance, in <value>s format. The minimum value is 0s, and the maximum value is 600s.
   • stabilization-duration: Period, in seconds, during which the required number of VMs cannot be decreased, in <value>s format. The minimum value is 60s and the maximum value is 1800s.
   • measurement-duration: Period, in seconds, for which the average utilization is calculated for each VM, in <value>s format. The minimum value is 60s (1 minute), and the maximum value is 600s (10 minutes).
   • cpu-utilization-target: Target CPU utilization level, in %. Use this setting to enable scaling based on CPU utilization. Otherwise, yarn.cluster.containersPending will be used for scaling based on the number of pending resources. The minimum value is 10, and the maximum value is 100.
   • autoscaling-decommission-timeout: Decommissioning timeout in seconds. The minimum value is 0, and the maximum value is 86400 (24 hours).

5. To create a Yandex Data Processing cluster based on dedicated host groups, specify their IDs as a comma-separated list in the --host-group-ids parameter:

   yc dataproc cluster create <cluster_name> \
     ...
     --host-group-ids=<dedicated_host_group_IDs>
   

   Alert

   You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

6. To configure Yandex Data Processing cluster hosts using initialization scripts, specify them in one or multiple --initialization-action parameters:

   yc dataproc cluster create <cluster_name> \
     ...
     --initialization-action uri=<initialization_script_URI>,`
                             `timeout=<script_timeout>,`
                             `args=["arg1","arg2","arg3",...]
   

   Where:

   • uri: Link to the initialization script in the https://, http://, hdfs://, or s3a:// scheme.
   • (Optional) timeout: Script timeout, in seconds. If your initialization script runs longer than this time period, it will be terminated.
   • (Optional) args: Arguments for the initialization script, provided as a comma-separated list.

Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

For more information about the provider resources, see the documentation on the Terraform website or the mirror.

If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

To create a Yandex Data Processing cluster:

1. Using the command line, navigate to the folder that will contain the Terraform configuration files with an infrastructure plan. Create the directory if it does not exist.

2. If you don't have Terraform, install it and configure the Yandex Cloud provider.

3. Create a configuration file describing the cloud network and subnets.

   The Yandex Data Processing cluster resides in a cloud network. If you already have a suitable network, you do not need to describe it again.

   Yandex Data Processing cluster hosts reside in subnets of the selected cloud network. If you already have suitable subnets, you do not need to describe them again.

   Example structure of a configuration file that describes a cloud network with a single subnet:

   resource "yandex_vpc_network" "test_network" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "test_subnet" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = yandex_vpc_network.test_network.id
     v4_cidr_blocks = ["<subnet>"]
   }
   

4. Create a configuration file describing the service account to access the Yandex Data Processing cluster, as well as the static key and the Object Storage bucket to store jobs and results.

   resource "yandex_iam_service_account" "data_proc_sa" {
     name        = "<service_account_name>"
     description = "<service_account_description>"
   }
   
   resource "yandex_resourcemanager_folder_iam_member" "dataproc-agent" {
     folder_id = "<folder_ID>"
     role      = "dataproc.agent"
     member    = "serviceAccount:${yandex_iam_service_account.data_proc_sa.id}"
   }
   
   resource "yandex_resourcemanager_folder_iam_member" "dataproc-provisioner" {
     folder_id = "<folder_ID>"
     role      = "dataproc.provisioner"
     member    = "serviceAccount:${yandex_iam_service_account.data_proc_sa.id}"
   }
   
   resource "yandex_iam_service_account_static_access_key" "sa_static_key" {
     service_account_id = yandex_iam_service_account.data_proc_sa.id
   }
   
   resource "yandex_storage_bucket" "data_bucket" {
     depends_on = [
       yandex_resourcemanager_folder_iam_member.dataproc-provisioner
     ]
   
     bucket     = "<bucket_name>"
     access_key = yandex_iam_service_account_static_access_key.sa_static_key.access_key
     secret_key = yandex_iam_service_account_static_access_key.sa_static_key.secret_key
   }
   

5. Create a configuration file describing the Yandex Data Processing cluster and its subclusters.

   If required, here you can also specify the properties of the Yandex Data Processing cluster components, jobs, and the environment.

   Below is an example of a configuration file structure that describes a Yandex Data Processing cluster consisting of a subcluster with a master host, a data storage subcluster, and a data processing subcluster:

   resource "yandex_dataproc_cluster" "data_cluster" {
     bucket              = "<bucket_name>"
     name                = "<cluster_name>"
     description         = "<cluster_description>"
     service_account_id  = yandex_iam_service_account.data_proc_sa.id
     zone_id             = "<availability_zone>"
     security_group_ids  = ["<list_of_security_group_IDs>"]
     deletion_protection = <cluster_deletion_protection>
   
     cluster_config {
       version_id = "<image_version>"
   
       hadoop {
         services   = ["<list_of_components>"]
         # Example of the list: ["HDFS", "YARN", "SPARK", "TEZ", "MAPREDUCE", "HIVE"].
         properties = {
           "<component_property>" = <value>
           ...
         }
         ssh_public_keys = [
           file("${file("<path_to_public_SSH_key>")}")
         ]
       }
   
       subcluster_spec {
         name = "<name_of_subcluster_with_master_host>"
         role = "MASTERNODE"
         resources {
           resource_preset_id = "<host_class>"
           disk_type_id       = "<storage_type>"
           disk_size          = <storage_size_in_GB>
         }
         subnet_id   = yandex_vpc_subnet.test_subnet.id
         hosts_count = 1
       }
   
       subcluster_spec {
         name = "<name_of_subcluster_for_storing_data>"
         role = "DATANODE"
         resources {
           resource_preset_id = "<host_class>"
           disk_type_id       = "<storage_type>"
           disk_size          = <storage_size_in_GB>
         }
         subnet_id   = yandex_vpc_subnet.test_subnet.id
         hosts_count = <number_of_hosts_in_subcluster>
       }
   
       subcluster_spec {
         name = "<data_processing_subcluster_name>"
         role = "COMPUTENODE"
         resources {
           resource_preset_id = "<host_class>"
           disk_type_id       = "<storage_type>"
           disk_size          = <storage_size_in_GB>
         }
         subnet_id   = yandex_vpc_subnet.test_subnet.id
         hosts_count = <number_of_hosts_in_subcluster>
       }
     }
   }
   

   Where deletion_protection means deletion protection for the Yandex Data Processing cluster. It can either be true or false.

   Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.

   Using an image of version 2.0.39 or higher, you can create a lightweight cluster without HDFS and subclusters for data storage. In this case, be sure to add one or more data processing subclusters and specify a bucket name.

   Tip

   To use the most recent image version, specify 2.0 in the version_id parameter.

   To access web interfaces of Yandex Data Processing components, add the ui_proxy field set to true to the Yandex Data Processing cluster description:

   resource "yandex_dataproc_cluster" "data_cluster" {
     ...
     ui_proxy = true
     ...
   }
   

   To configure autoscaling parameters in Yandex Data Processing subclusters for data processing, add the autoscaling_config section with the settings you need to the subcluster_spec description of the relevant subcluster:

   subcluster_spec {
     name = "<subcluster_name>"
     role = "COMPUTENODE"
     ...
     autoscaling_config {
       max_hosts_count        = <maximum_number_of_VMs_in_group>
       measurement_duration   = <load_measurement_interval>
       warmup_duration        = <warmup_period>
       stabilization_duration = <stabilization_period>
       preemptible            = <use_preemptible_VMs>
       cpu_utilization_target = <target_CPU_utilization_level>
       decommission_timeout   = <decommissioning_timeout>
     }
   }
   

   Where:

   • max_hosts_count: Maximum number of hosts (VMs) in the Yandex Data Processing subcluster. The minimum value is 1, and the maximum value is 100.
   • measurement_duration: Period, in seconds, for which the average utilization is calculated for each VM, in <value>s format. The minimum value is 60s (1 minute), and the maximum value is 600s (10 minutes).
   • warmup_duration: Time required to warm up a VM instance, in <value>s format. The minimum value is 0s, and the maximum value is 600s.
   • stabilization_duration: Period, in seconds, during which the required number of VMs cannot be decreased, in <value>s format. The minimum value is 60s and the maximum value is 1800s.
   • preemptible: Indicates if preemptible VMs are used. It can either be true or false.
   • cpu_utilization_target: Target CPU utilization level, in %. Use this setting to enable scaling based on CPU utilization. Otherwise, yarn.cluster.containersPending will be used for scaling based on the number of pending resources. The minimum value is 10, and the maximum value is 100.
   • decommission_timeout: Decommissioning timeout in seconds. The minimum value is 0, and the maximum value is 86400 (24 hours).

   For more information about the resources you can create with Terraform, see the provider documentation.

6. Check that the Terraform configuration files are correct:

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

7. Create a Yandex Data Processing cluster:

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

   All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

To create a Yandex Data Processing cluster, use the create API method and include the following in the request:

• ID of the folder to host the Yandex Data Processing cluster, in the folderId parameter.

• Yandex Data Processing cluster name in the name parameter.

• Yandex Data Processing cluster configuration in the configSpec parameter, including:

  • Image version in the configSpec.versionId parameter.

    Using an image of version 2.0.39 or higher, you can create a lightweight cluster without HDFS and subclusters for data storage. In this case, be sure to add one or more data processing subclusters and specify a bucket name.

    Tip

    To use the most recent image version, specify 2.0.

  • List of components in the configSpec.hadoop.services parameter.

  • Public part of the SSH key in the configSpec.hadoop.sshPublicKeys parameter.

  • Settings of the Yandex Data Processing subclusters in the configSpec.subclustersSpec parameter.

• Availability zone of the Yandex Data Processing cluster in the zoneId parameter.

• ID of the Yandex Data Processing cluster service account in the serviceAccountId parameter. Make sure to assign the following roles to the service account:

  • dataproc.agent: To enable the service account to get info on cluster host states, jobs, and log groups.
  • dataproc.provisioner: To enable the service account to work with an autoscaling instance group. This will enable subcluster autoscaling.

• Bucket name in the bucket parameter.

• IDs of the Yandex Data Processing cluster security groups in the hostGroupIds parameter.

• Yandex Data Processing cluster deletion protection settings in the deletionProtection parameter.

  Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

To assign a public IP address to all hosts of a Yandex Data Processing subcluster, provide the true value in the configSpec.subclustersSpec.assignPublicIp parameter.

To create a Yandex Data Processing cluster based on dedicated host groups, provide a list of their IDs in the hostGroupIds parameter.

To configure Yandex Data Processing cluster hosts using initialization scripts, specify them in one or multiple configSpec.hadoop.initializationActions parameters.