Autoscaling in Yandex Managed Service for Apache Spark™

Autoscaling is changing the number of Apache Spark™ cluster hosts depending on current load.

Host pools for drivers and executors get scaled independently:

• If there is not enough free CPUs or RAM to run a new app, i.e., a new driver, Yandex Managed Service for Apache Spark™ starts a new host in the host pool for drivers. If the number of running apps decreases, and an idling host appears in the host pool for drivers, such a host gets deleted.

• If there is not enough free CPUs or RAM to host all executors of every running app, Yandex Managed Service for Apache Spark™ starts new hosts in the host pool for executors. If the number of running executors decreases, and an idling host appears in the host pool for executors, such a host gets deleted.

Autoscaling advantages:

• The cluster becomes more productive when running demanding jobs; whereas, during idling, excessive hosts get deleted so you pay less for them.
• Time per job is less dependent on the number of concurrent jobs.
• Dynamic allocation becomes more efficient: host pool for executors adjusts to the changing app needs.

Autoscaling managementAutoscaling management

You can enable and disable autoscaling for each of the two host pools when creating and updating an Apache Spark™ cluster.

For each host pool, you can set a range for the number of hosts in that pool:

• The minimum number cannot be less than zero.
• The maximum number cannot be more than 100 or less than the minimum number.

What the minimum number of pool hosts does:

• The higher the minimum number of hosts in the pool for drivers, the more apps of the same type can be run concurrently without waiting for additional hosts to start.
• The higher the minimum number of hosts in the pool for executors, the greater the load that can be channeled to the cluster without waiting for additional hosts to start.
• As the minimum number of hosts in the pools increases, you pay more for the cluster when idling or under low load.

What the maximum number of pool hosts does:

• The higher the maximum number of hosts in the pool for drivers, the more apps of the same type can be run concurrently.
• The higher the maximum number of hosts in the pool for executors, the higher the maximum cluster performance.
• As the maximum number of hosts in the pools increases, you pay more for the cluster under maximum load.

Recommendations on the choice of autoscaling parametersRecommendations on the choice of autoscaling parameters

The choice autoscaling parameters depends on the particular cluster use case. The table values are approximate.

The number of concurrent same-type jobs on a cluster and their processing speed depend on the following:

• Number and class of hosts in each pool.
• Parameters of each job.
• Amount of cluster resources reserved for internal needs.
• Time spent on process planning.
• Time spent on pool scaling.
• Use of shuffle operations.