Diagnosing and troubleshooting Spark application performance issues

If your Spark applications are slow:

• Check their execution to identify the cause of performance issues.
• Try using one of the methods for troubleshooting common issues.

Initial performance diagnostics of Spark applicationsInitial performance diagnostics of Spark applications

If your Spark application underperforms, run the initial diagnostics:

• Check the application queue to make sure your application is not getting blocked by others.

• View detailed application info and check the jobs' statuses and actual start and end times on the Event Timeline chart.

• Check the resources allocated to the application:

  • Make sure the application has enough executors and all available executors are not idle.
  • Make sure the resources within each executor are used in a balanced way.

• Check the SQL query plans and the execution time for each operation.

• Check the application logs for any warnings about failures.

Troubleshooting common performance issuesTroubleshooting common performance issues

High garbage collection timeHigh garbage collection time

If you checked the resources allocated to the application and found out that the GC Time in the total Task Time is high:

• Make sure the executor has enough memory allocated.
• Configure the garbage collector manually. Check out this Apache Spark guide.

Multiple executors are competing for CPU resourcesMultiple executors are competing for CPU resources

When allocating executors, the YARN scheduler using the default settings ignores available CPU resources on the node. This may slow down compute-intensive jobs.

To avoid this, enable the alternative resource-aware scheduling algorithm for executors by setting the following cluster-level property:

capacity-scheduler:yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

For more information about the YARN scheduler, see this Hadoop guide.

Recurring heartbeat errors when running jobsRecurring heartbeat errors when running jobs

When running Spark jobs, executors regularly send heartbeat messages to the driver with info on the executor status and operation progress. If the driver does not get any heartbeat messages from an executor during a certain interval, it considers this executor unhealthy and requests the YARN resource manager to forcibly terminate it. In this case, the driver logs will contain a message of the following type:

23/02/23 20:22:09 WARN TaskSetManager: Lost task 28.0 in stage 13.0 (TID 242) 
        (rc1c-dataproc-*****.mdb.yandexcloud.net executor 5): ExecutorLostFailure 
        (executor 5 exited caused by one of the running tasks) 
        Reason: Executor heartbeat timed out after 138218 ms

Such errors may be due to cluster networking issues. In practice, however, heartbeat timeouts most often occur because an executor runs out of memory. Consequently, job logs may fail to register such errors as java.lang.OutOfMemoryError because logging itself fails due to the memory shortage.

If you regularly get heartbeat errors when running jobs and there are no other signs of network errors, increase the amount of RAM allocated per concurrent operation. To do this, change your cluster's component properties:

• Reduce the number of CPU cores per executor in the spark.executor.cores parameter.
• Increase the amount of RAM allocated per executor in the spark.executor.memory parameter.

For more information about these parameters, see this Spark guide.