Creating an Apache Spark™ cluster

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a Yandex Managed Service for Apache Spark™ cluster:

1. In the configuration file, describe the resources you are creating:

   • Yandex Managed Service for Apache Spark™ cluster: Cluster description.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_spark_cluster" "<cluster_name>" {
     description         = "<cluster_description>"
     name                = "<cluster_name>"
     folder_id           = "<folder_ID>"
     service_account_id  = "<service_account_ID>"
     deletion_protection = <protect_cluster_from_deletion>
   
     labels = {
       <label_list>
     }
   
     network = {
       subnet_ids         = ["<list_of_subnet_IDs>"]
       security_group_ids = ["<list_of_security_group_IDs>"]
     }
   
     config = {
       resource_pools = {
         driver = {
           resource_preset_id = "<host_class>"
           size               = <fixed_number_of_instances>
         }
         executor = {
           resource_preset_id = "<host_class>"
           size               = <fixed_number_of_instances>
         }
       }
     }
   
     logging = {
       enabled      = <enable_logging>
       folder_id    = "<folder_ID>"
     }
   
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • description: Cluster description. This is an optional parameter.

   • name: Cluster name.

   • folder_id: Folder ID. This is an optional parameter. If the value is missing, the cluster will reside in the folder specified in the provider settings.

   • service_account_id: Service account ID.

   • deletion_protection: Cluster protection from accidental deletion, true or false. This is an optional parameter.

   • labels: List of labels. This is an optional parameter. Provide labels in <key> = "<value>" format.

   • subnet_ids: Subnet IDs list.

   • security_group_ids: List of security group IDs.

   • driver: Host configuration to run Apache Spark™ drivers. In this section, specify:

     • Host class in the resource_preset_id parameter.
     • Number of instances. Specify a fixed number in the size parameter or the minimum and maximum number for autoscaling in the min_size and max_size parameters.

   • executor: Host configuration to run Apache Spark™ executors. In this section, specify:

     • Host class in the resource_preset_id parameter.
     • Number of instances. Specify a fixed number in the size parameter or the minimum and maximum number for autoscaling in the min_size and max_size parameters.

   • logging: Logging parameters. Logs generated by Apache Spark™ components will be sent to Yandex Cloud Logging. To enable logging:

     • Set the enabled = true value.

     • Specify one of two log storage locations:

       • folder_id: Folder ID. Logs will be written to the default log group for this folder.
       • log_group_id: Custom log group ID. Logs will be written to this group.

2. If necessary, configure additional DBMS settings:

   • To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

     resource "yandex_spark_cluster" "<cluster_name>" {
       ...
       maintenance_window {
         type = <maintenance_type>
         day  = <day_of_week>
         hour = <hour>
       }
       ...
     }
     

     Where:

     • type: Maintenance type. The possible values include:
       • ANYTIME: Anytime
       • WEEKLY: On a schedule
     • day: Day of week for the WEEKLY type, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: UTC hour for the WEEKLY type, from 1 to 24.

   • To enable Apache Spark™ History Server, add a section named history_server to the cluster configuration description:

     resource "yandex_spark_cluster" "<cluster_name>" {
     ...
       config = {
       ...
         history_server = {
           enabled = true
         }
       }
     }
     

   • To connect an Apache Hive™ Metastore server to your cluster, add the metastore section to the cluster configuration description:

     resource "yandex_spark_cluster" "<cluster_name>" {
     ...
       config = {
       ...
         metastore = {
           cluster_id = "<metastore_cluster_ID>"
         }
       }
     }
     

   • To connect additional deb and pip packages for running Apache Spark™ jobs, add the dependencies section to the cluster configuration description:

     resource "yandex_spark_cluster" "<cluster_name>" {
     ...
       config = {
       ...
         dependencies = {
           deb_packages = ["<list_of_deb_packages>"]
           pip_packages = ["<list_of_pip_packages>"]
         }
       }
     }
     

     Where deb_packages and pip_packages are package names. Their format depends on the installation command: apt install for deb packages and pip install for pip packages.

3. Validate your configuration.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Create a Yandex Managed Service for Apache Spark™ cluster.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   This will create all the resources you need in the specified folder. You can check the new resources and their configuration using the management console.

For more information, see this Terraform provider article.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "folder_id": "<folder_ID>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "labels": { <label_list> },
     "config": {
       "resource_pools": {
         "driver": {
           "resource_preset_id": "<resource_ID>",
           "scale_policy": {
             "fixed_scale": {
               "size": "<number_of_instances>"
             }
           }
         },
         "executor": {
           "resource_preset_id": "<resource_ID>",
           "scale_policy": {
             "auto_scale": {
               "min_size": "<minimum_number_of_instances>",
               "max_size": "<maximum_number_of_instances>"
             }
           }
         }
       },
       "history_server": {
         "enabled": <use_of_Apache_Spark_History_Server>
       },
       "dependencies": {
         "pip_packages": [ <list_of_pip_packages> ],
         "deb_packages": [ <list_of_deb_packages> ]
       },
       "metastore": {
         "cluster_id": "<cluster_ID>"
       }
     },
     "network": {
       "subnet_ids": [ <list_of_subnet_IDs> ],
       "security_group_ids": [ <list_of_security_group_IDs> ]
     },
     "deletion_protection": <deletion_protection>,
     "service_account_id": "<service_account_ID>",
     "logging": {
       "enabled": <use_of_logging>,
       "folder_id": "<folder_ID>"
     }
   }
   

   Where:

   • folder_id: Folder ID. You can request it with the list of folders in the cloud.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels. Provide labels in "<key>": "<value>" format.

   • config: Cluster configuration:

     • resource_pools: Resource pool configuration:

       • driver: Host configuration to run Apache Spark™ drivers.

         • resource_preset_id: Driver host class.

         • scale_policy: Host group scaling policy for the driver:

           • fixed_scale: Fixed scaling policy.

             • size: Number of driver hosts.

           • auto_scale: Automatic scaling policy.

             • min_size: Minimum number of driver hosts.
             • max_size: Maximum number of driver hosts.

             Specify either fixed_scale or auto_scale.

       • executor: Host configuration to run Apache Spark™ executors.

         • resource_preset_id: Executor host class.

         • scale_policy: Host group scaling policy for the executor:

           • fixed_scale: Fixed scaling policy.

             • size: Number of executor hosts.

           • auto_scale: Automatic scaling policy.

             • min_size: Minimum number of executor hosts.
             • max_size: Maximum number of executor hosts.

             Specify either fixed_scale or auto_scale.

       • history_server: History server parameters.

         • enabled: Flag to enable history server. It allows using the service to monitor Spark History Server applications.

       • dependencies: Lists of packages enabling you to install additional libraries and applications on the cluster.

         • pip_packages: List of pip packages.
         • deb_packages: List of deb packages.

         The package name format and version are defined by the install command: pip install for pip packages and apt install for deb packages.

       • metastore: Metastore parameters.

         • cluster_id: Apache Hive™ Metastore cluster ID.

     • network: Network settings:

       • subnet_ids: List of subnet IDs.
       • security_group_ids: List of security group IDs.

     • deletion_protection: Enables cluster protection against accidental deletion. The possible values are: true or false.

       Even if it is enabled, one can still connect to the cluster manually and delete it.

     • service_account_id: Service account ID.

     • logging: Logging parameters:

       • enabled: Enables logging. Logs generated by Spark applications will go to Yandex Cloud Logging. The possible values are: true or false.
       • folder_id: Folder ID. Logs will be written to the default log group for this folder.
       • log_group_id: Custom log group ID. Logs will be written to this group.

       Specify either folder_id or log_group_id.

4. Use the ClusterService/Create call and send the following request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/spark/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       spark.api.cloud.yandex.net:443 \
       yandex.cloud.spark.v1.ClusterService.Create \
       < body.json
   

5. View the server response to make sure your request was successful.