Updating an Apache Spark™ cluster

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To change the cluster name and description:

1. View the description of the CLI command for updating a cluster:

   yc managed-spark cluster update --help
   

2. To change cluster name and description, run this command:

   yc managed-spark cluster update <cluster_name_or_ID> \
     --new-name <cluster_name> \
     --description <cluster_description> 
   

   Where:

   • --new-name: Cluster name, unique within the cloud.
   • --description: Cluster description.

   You can get the cluster name and ID with the list of clusters in the folder.

1. Open the current Terraform configuration file with the infrastructure plan.

   To learn how to create this file, refer to Creating a cluster.

2. Edit the description parameter in the cluster's description:

   resource "yandex_spark_cluster" "<cluster_name>" {
     ...
     description = "<cluster_description>"
     ...
   }
   

   Where description is the cluster's description.

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm updating the resources.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "cluster_id": "<cluster_ID>",
     "update_mask": {
       "paths": [
         <list_of_settings_to_update>
       ]
     },
     "name": "<cluster_name>",
     "description": "<cluster_description>"
   }
   

   Where:

   • cluster_id: Cluster ID.

     You can get the cluster ID with the list of clusters in the folder.

   • update_mask: List of parameters to update as an array of strings (paths[]).

     Format for listing settings

     "update_mask": {
       "paths": [
         "<setting_1>",
         "<setting_2>",
         ...
         "<setting_N>"
       ]
     }
     

     Warning

     When you update a cluster, all parameters of the object you are modifying will be reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the update_mask parameter.

   • name: Cluster name, unique within the cloud.

   • description: Cluster description.

4. Call the ClusterService.Update method, e.g., via the following gRPCurl request:

   grpcurl \
     -format json \
     -import-path ~/cloudapi/ \
     -import-path ~/cloudapi/third_party/googleapis/ \
     -proto ~/cloudapi/yandex/cloud/spark/v1/cluster_service.proto \
     -rpc-header "Authorization: Bearer $IAM_TOKEN" \
     -d @ \
     spark.api.cloud.yandex.net:443 \
     yandex.cloud.spark.v1.ClusterService.Update \
     < body.json
   

5. Check the server response to make sure your request was successful.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To update a service account:

1. View the description of the CLI command for updating a cluster:

   yc managed-spark cluster update --help
   

2. Update your service account using this command:

   yc managed-spark cluster update <cluster_name_or_ID> \
     --service-account-id <service_account_ID>
   

   Where --service-account-id is the ID of the service account for access to Yandex Cloud services. Make sure to assign the managed-spark.integrationProvider role to this service account:

   You can get the cluster name and ID with the list of clusters in the folder.

1. Open the current Terraform configuration file with the infrastructure plan.

   Learn how to create this file in Creating a cluster.

2. Edit the service_account_id parameter in the cluster's description:

   resource "yandex_spark_cluster" "<cluster_name>" {
     ...
     service_account_id  = "<service_account_ID>"
     ...
   }
   

   Where service_account_id is the service account ID.

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm updating the resources.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "cluster_id": "<cluster_ID>",
     "update_mask": {
       "paths": [
         "service_account_id"
       ]
     },
     "service_account_id": "<service_account_ID>"
   }
   

   Where:

   • cluster_id: Cluster ID.

     You can get the cluster ID with the list of clusters in the folder.

   • update_mask: List of parameters to update as an array of strings (paths[]).

     Format for listing settings

     "update_mask": {
       "paths": [
         "<setting_1>",
         "<setting_2>",
         ...
         "<setting_N>"
       ]
     }
     

     Warning

     When you update a cluster, all parameters of the object you are modifying will be reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the update_mask parameter.

   • service_account_id: Service account ID.

4. Call the ClusterService.Update method, e.g., via the following gRPCurl request:

   grpcurl \
     -format json \
     -import-path ~/cloudapi/ \
     -import-path ~/cloudapi/third_party/googleapis/ \
     -proto ~/cloudapi/yandex/cloud/spark/v1/cluster_service.proto \
     -rpc-header "Authorization: Bearer $IAM_TOKEN" \
     -d @ \
     spark.api.cloud.yandex.net:443 \
     yandex.cloud.spark.v1.ClusterService.Update \
     < body.json
   

5. Check the server response to make sure your request was successful.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To update security groups:

1. View the description of the CLI command for updating a cluster:

   yc managed-spark cluster update --help
   

2. Update security groups using this command:

   yc managed-spark cluster update <cluster_name_or_ID> \
     --security-group-ids <list_of_security_group_IDs>
   

   Where --security-group-ids is the list of security group IDs.

   You can get the cluster name and ID with the list of clusters in the folder.

1. Open the current Terraform configuration file with the infrastructure plan.

   Learn how to create this file in Creating a cluster.

2. Edit the security_group_ids parameter in the cluster's description:

   resource "yandex_spark_cluster" "<cluster_name>" {
     ...
     security_group_ids  = [<list_of_security_group_IDs>]
     ...
   }
   

   Where security_group_ids is the list of security group IDs.

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm updating the resources.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "cluster_id": "<cluster_ID>",
     "update_mask": {
       "paths": [
         "network_spec.security_group_ids"
       ]
     },
     "network_spec": {
       "security_group_ids": [ <list_of_security_group_IDs> ]
     }
   }
   

   Where:

   • cluster_id: Cluster ID.

     You can get the cluster ID with the list of clusters in the folder.

   • update_mask: List of parameters to update as an array of strings (paths[]).

     Format for listing settings

     "update_mask": {
       "paths": [
         "<setting_1>",
         "<setting_2>",
         ...
         "<setting_N>"
       ]
     }
     

     Warning

     When you update a cluster, all parameters of the object you are modifying will be reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the update_mask parameter.

   • network_spec: Network settings:

     • security_group_ids: List of security group IDs.

4. Call the ClusterService.Update method, e.g., via the following gRPCurl request:

   grpcurl \
     -format json \
     -import-path ~/cloudapi/ \
     -import-path ~/cloudapi/third_party/googleapis/ \
     -proto ~/cloudapi/yandex/cloud/spark/v1/cluster_service.proto \
     -rpc-header "Authorization: Bearer $IAM_TOKEN" \
     -d @ \
     spark.api.cloud.yandex.net:443 \
     yandex.cloud.spark.v1.ClusterService.Update \
     < body.json
   

5. Check the server response to make sure your request was successful.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To update the driver and executor configuration:

1. View the description of the CLI command for updating a cluster:

   yc managed-spark cluster update --help
   

2. Update the driver and executor configuration by running this command:

   yc managed-spark cluster update <cluster_name_or_ID> \
     --driver-preset-id <driver_resource_ID> \
     --driver-fixed-size <number_of_driver_instances> \
     --executor-preset-id <executor_resource_ID> \
     --executor-fixed-size <number_of_executor_instances> \
   

   Where:

   • --driver-preset-id: Driver host class.
   • --driver-fixed-size: Fixed number of driver hosts.
   • --driver-min-size: Minimum number of driver hosts for autoscaling.
   • --driver-max-size: Maximum number of driver hosts for autoscaling.

   Specify either a fixed number of hosts (--driver-fixed-size) or minimum and maximum number of hosts (--driver-min-size and --driver-max-size) for autoscaling.

   • --executor-preset-id: Executor host class.
   • --executor-fixed-size: Fixed number of executor hosts.
   • --executor-min-size: Minimum number of executor hosts for autoscaling.
   • --executor-max-size: Maximum number of executor hosts for autoscaling.

   Specify either a fixed number of hosts (--executor-fixed-size) or minimum and maximum number of hosts (--executor-min-size and --executor-max-size) for autoscaling.

1. Open the current Terraform configuration file with the infrastructure plan.

   Learn how to create this file in Creating a cluster.

2. Update the driver and executor configuration under driver and executor respectively:

   resource "yandex_spark_cluster" "<cluster_name>" {
     ...
     resource_pools = {
       driver = {
         resource_preset_id = "<host_class>"
         size               = <fixed_number_of_instances>
       }
       executor = {
         resource_preset_id = "<host_class>"
         size               = <fixed_number_of_instances>
       }
     }
     ...
   }
   

   Where:

   • driver: Host configuration to run Apache Spark™ drivers. In this section, specify:

     • resource_preset_id: Host class.
     • size: Fixed number of instances.
     • min_size: Minimum number of hosts if autoscaling is on.
     • max_size: Maximum number of hosts if autoscaling is on.

   • executor: Host configuration to run Apache Spark™ executors. In this section, specify:

     • resource_preset_id: Host class.
     • size: Fixed number of instances.
     • min_size: Minimum number of hosts if autoscaling is on.
     • max_size: Maximum number of hosts if autoscaling is on.

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm updating the resources.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "cluster_id": "<cluster_ID>",
     "update_mask": {
       "paths": [ 
         <list_of_settings_to_update>
         ]
     },
     "config_spec": {
       "resource_pools": {
         "driver": {
           "resource_preset_id": "<driver_resource_ID>",
           "scale_policy": {
             "fixed_scale": {
               "size": "<number_of_driver_instances>"
             }
           }
         },
         "executor": {
           "resource_preset_id": "<executor_resource_ID>",
           "scale_policy": {
             "auto_scale": {
               "min_size": "<minimum_number_of_executor_instances>",
               "max_size": "<maximum_number_of_executor_instances>"
             }
           }
         }
       }
     }
   }
   

   Where:

   • cluster_id: Cluster ID.

     You can get the cluster ID with the list of clusters in the folder.

   • update_mask: List of parameters to update as an array of strings (paths[]).

     Format for listing settings

     "update_mask": {
       "paths": [
         "<setting_1>",
         "<setting_2>",
         ...
         "<setting_N>"
       ]
     }
     

     Warning

     When you update a cluster, all parameters of the object you are modifying will be reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the update_mask parameter.

   • config_spec: Cluster configuration:

     • resource_pools: Resource pool configuration:

       • driver: Host configuration to run Apache Spark™ drivers.

         • resource_preset_id: Driver host class.

         • scale_policy: Host group scaling policy for the driver:

           • fixed_scale: Fixed scaling policy.

             • size: Number of driver hosts.

           • auto_scale: Automatic scaling policy.

             • min_size: Minimum number of driver hosts.
             • max_size: Maximum number of driver hosts.

           Specify either fixed_scale or auto_scale.

       • executor: Host configuration to run Apache Spark™ executors.

         • resource_preset_id: Executor host class.

         • scale_policy: Host group scaling policy for the executor:

           • fixed_scale: Fixed scaling policy.

             • size: Number of executor hosts.

           • auto_scale: Automatic scaling policy.

             • min_size: Minimum number of executor hosts.
             • max_size: Maximum number of executor hosts.

           Specify either fixed_scale or auto_scale.

4. Call the ClusterService.Update method, e.g., via the following gRPCurl request:

   grpcurl \
     -format json \
     -import-path ~/cloudapi/ \
     -import-path ~/cloudapi/third_party/googleapis/ \
     -proto ~/cloudapi/yandex/cloud/spark/v1/cluster_service.proto \
     -rpc-header "Authorization: Bearer $IAM_TOKEN" \
     -d @ \
     spark.api.cloud.yandex.net:443 \
     yandex.cloud.spark.v1.ClusterService.Update \
     < body.json
   

5. Check the server response to make sure your request was successful.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

To change advanced cluster settings:

1. View the description of the CLI command for updating a cluster:

   yc managed-spark cluster update --help
   

2. Update additional cluster settings using this command:

   yc managed-spark cluster update <cluster_name_or_ID> \
     --history-server-enabled <use_Spark_History_Server> \
     --metastore-cluster-id <Apache_Hive™_Metastore_cluster_ID> \
     --pip-packages <list_of_pip_packages> \
     --deb-packages <list_of_deb_packages> \
     --log-enabled \
     --log-folder-id <folder_ID> \
     --maintenance-window type=<maintenance_type>,`
                         `day=<day_of_week>,`
                         `hour=<hour> \
     --deletion-protection
   

   Where:

   • --history-server-enabled: Enables the Spark History Server monitoring service.

   • --metastore-cluster-id: Apache Hive™ Metastore cluster ID. This setting connects the Apache Hive™ Metastore metadata storage.

   • --pip-packages: List of pip packages.

   • --deb-packages: List of deb packages.

     Use lists of packages to install additional libraries and applications in the cluster.

     You can set version restrictions for the installed packages, e.g.:

     --pip-packages pandas==2.1.1,scikit-learn>=1.0.0,clickhouse-driver~=0.2.0
     

     The package name format and version are defined by the install command: pip install for pip packages and apt install for deb packages.

   • --log-enabled: Enables logging.

   • --log-folder-id: Folder ID. Logs will be written to the default log group for this folder.

   • --log-group-id: Custom log group ID. Logs will be written to this group.

     Specify either --log-folder-id or --log-group-id.

   • --maintenance-window: Maintenance window settings that apply to both running and stopped clusters. The type setting defines the maintenance type:

     • To allow maintenance at any time, select At any time (default).
     • To specify a preferred maintenance start time, select By schedule and specify a day of the week and a UTC time interval. For example, you can choose the cluster's least busy time.

     Both active and stopped clusters are subject to maintenance operations. These may include installing Apache Spark™ updates and patches for hosts as well as other maintenance works.

   • --deletion-protection: Enables cluster protection against accidental deletion.

     Even with deletion protection on, one can still connect to the cluster manually and delete it.

   You can get the cluster name and ID with the list of clusters in the folder.

To change the cluster settings:

1. Open the current Terraform configuration file with the infrastructure plan.

   To learn how to create this file, refer to Creating a cluster.

2. To change cluster settings, change the required field values in the configuration file.

   Here is an example of the configuration file structure:

   resource "yandex_spark_cluster" "my_spark_cluster" {
   
     deletion_protection = <protect_cluster_from_deletion>
   
     config = {
       ...
       history_server = {
         enabled = <use_Spark_History_Server>
       } 
       metastore = {
         cluster_id = "<Apache_Hive™_Metastore_cluster_ID>"
       }
       dependencies = {
         deb_packages = ["<list_of_deb_packages>"]
         pip_packages = ["<list_of_pip_packages>"]
       }
     }
   
     maintenance_window = {
       type = "<maintenance_type>"
       day  = "<day_of_week>"
       hour = "<sequence_number_of_hour_interval>"
     }
   
     logging = {
       enabled      = <enable_logging>
       folder_id    = "<folder_ID>"
     }
   
   }
   

   Where:

   • deletion_protection: Cluster deletion protection, true or false.

   • maintenance_window: Maintenance window settings (including for disabled clusters). In this section, specify:

     • type: Maintenance type. The possible values include:

       • ANYTIME: Any time.
       • WEEKLY: On a schedule.

     • day: Day of week for the WEEKLY maintenance type, MON, TUE, WED, THU, FRI, SAT, or SUN.

     • hour: UTC hour interval for the WEEKLY maintenance type, from 1 to 24.

       For example, 1 stands for the interval from 00:00 to 01:00, and 5, from 04:00 to 05:00.

   • history_server: Connecting Apache Spark™ History Server. To use the service, set the enabled parameter to true.

   • metastore: Connecting a Apache Hive™ Metastore metadata storage. Specify a Apache Hive™ Metastore cluster ID in the cluster_id parameter.

   • dependencies: Additional deb and pip packages for running Apache Spark™ jobs. In this section, specify:

     • deb_packages: Names of deb packages. Their format depends on the apt install installation command.
     • pip_packages: Names of pip packages. Their format depends on the pip install installation command.

   • logging: Logging parameters. Logs generated by Apache Spark™ components will be sent to Yandex Cloud Logging. To enable logging:

     • Set it to enabled = true.

     • Specify one of two log storage locations:

       • folder_id: Folder ID. Logs will be written to the default log group for this folder.
       • log_group_id: Custom log group ID. Logs will be written to this group.

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Confirm updating the resources.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information, see this Terraform provider guide.

To change the cluster settings:

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "cluster_id": "<cluster_ID>",
     "update_mask": {
       "paths": [ "<list_of_settings_to_update>" ]
     },
     "config_spec": {
       "history_server": {
         "enabled": <use_Spark_History_Server>
       },
       "metastore": {
         "cluster_id": "<Apache_Hive™_Metastore_cluster_ID>"
       },
       "environment_id": "<environment_ID>"
     },
     "deletion_protection": <deletion_protection>,
     "logging": {
       "enabled": <use_of_logging>,
       "log_group_id": "<log_group_ID>",
       "folder_id": "<folder_ID>"
     },
     "maintenance_window": {
       "weekly_maintenance_window": {
         "day": "<day_of_week>",
         "hour": "<sequence_number_of_hour_interval>"
       }
     }
   }
   

   Where:

   • cluster_id: Cluster ID. You can get it with the list of clusters in the folder.

   • update_mask: List of settings you want to update as an array of strings (paths[]).

     Format for listing settings

     "update_mask": {
         "paths": [
             "<setting_1>",
             "<setting_2>",
             ...
             "<setting_N>"
         ]
     }
     

     Warning

     When you update a cluster, all its parameters will reset to their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the update_mask parameter.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels provided in "<key>": "<value>" format.

   • config_spec: Cluster configuration:

     • history_server: History server parameters.

       • enabled: Flag to enable history server. It allows using the service to monitor Spark History Server applications.

     • metastore: Parameters of the cluster’s metadata storage.

       • cluster_id: Apache Hive™ Metastore cluster ID.

     • dependencies: Lists of packages to install in the cluster:

       • pip_packages: List of pip packages.
       • deb_packages: List of deb packages.

       The package name format and version are defined by the install commands: pip install for pip packages and apt install for deb packages.

     • spark_version: Apache Spark™ version.

     • environment_id: Base or custom environment ID. Call the EnvironmentService/List method to get the custom environment ID or EnvironmentService/ListBase to get the base environment ID.

       To connect an environment instead of using the deprecated spark_version and dependencies parameters, add config_spec.environment_id, config_spec.spark_version, and config_spec.dependencies to update_mask.paths. Provide only environment_id to config_spec. This will clear the Apache Spark™ version and the package lists you specified earlier manually.

     Warning

     Specify either environment_id or spark_version and dependencies in the request. Updating the version and adding packages without creating an environment is deprecated and will soon be unavailable. Use a base environment or create a custom one containing the packages you need.

     • network: Network settings:

       • security_group_ids: List of security group IDs.

     • deletion_protection: Enables cluster protection against accidental deletion. The possible values are true or false.

       Even with deletion protection on, one can still connect to the cluster manually and delete it.

     • logging: Logging parameters:

       • enabled: Enables logging. The possible values are true or false. Logs generated by Apache Spark™ components will be sent to Yandex Cloud Logging. The possible values are true or false.
       • folder_id: Folder ID. Logs will be written to the default log group for this folder.
       • log_group_id: Custom log group ID. Logs will be written to this group.

       Specify either folder_id or log_group_id.

     • maintenance_window: Maintenance window settings, applying to both running and stopped clusters. Provide one of these two properties:

       • anytime: Maintenance takes place at any time.

       • weekly_maintenance_window: Maintenance takes place once a week at the specified time:

         • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.

         • hour: UTC hour interval, from 1 to 24.

           For example, 1 stands for the interval from 00:00 to 01:00, and 5, from 04:00 to 05:00.

4. Call the ClusterService/Update method, e.g., via the following gRPCurl request:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/spark/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       spark.api.cloud.yandex.net:443 \
       yandex.cloud.spark.v1.ClusterService.Update \
       < body.json
   

5. Check the server response to make sure your request was successful.