Component properties

The properties of cluster components, jobs, and environments are stored in the following format:

<key>:<value>

The key can either be a simple string or contain a prefix indicating that it belongs to a specific component:

<key_prefix>:<key_body>:<value>

For example:

hdfs:dfs.replication : 2
hdfs:dfs.blocksize : 1073741824
spark:spark.driver.cores : 1

Updating component propertiesUpdating component properties

You can update the component properties in the following ways:

• At the cluster level when creating or updating it. The properties provided this way apply to any new cluster jobs by default.
• At the level of an individual job when creating it. The properties provided this way only apply to this job and override the cluster-level properties set for it.

Available component propertiesAvailable component properties

The available properties are listed in the official documentation for the components:

Settings for running jobs are specified in special properties:

• dataproc:version: Version of the dataproc-agent that runs jobs, sends the property of a cluster state, and proxies the UI. This poperty is used for debugging. Its default value is latest.

• dataproc:max-concurrent-jobs: Number of concurrent jobs. The default value is auto (calculated based on the min-free-memory-to-enqueue-new-job and job-memory-footprint properties).

• dataproc:min-free-memory-to-enqueue-new-job: Minimum size of free memory to run a job, in bytes. The default value is 1073741824 (1 GB).

• dataproc:job-memory-footprint: Memory size to run a job on the cluster's master host, used to estimate the maximum number of jobs per cluster. The default value is 536870912 (512 MB).

• dataproc:spark_executors_per_vm: Maximum number of containers per computing host when running Spark jobs. The default values are:

  • 1: For lightweight clusters.
  • 2: For clusters with HDFS.

• dataproc:spark_driver_memory_fraction: Computing host memory fraction reserved for the driver when running Spark jobs. The default value is 0.25.

JVM settings for Spark applications set in Yandex Data Processing by defaultJVM settings for Spark applications set in Yandex Data Processing by default

Generally, the following default settings are applied on the Yandex Data Processing clusters to improve JVM performance:

• spark:spark.driver.extraJavaOptions:
  • -XX:+UseConcMarkSweepGC
  • -XX:CMSInitiatingOccupancyFraction=70
  • -XX:MaxHeapFreeRatio=70
  • -XX:+CMSClassUnloadingEnabled
  • -XX:OnOutOfMemoryError='kill -9 %p'
• spark:spark.executor.extraJavaOptions:
  • -verbose:gc
  • -XX:+PrintGCDetails
  • -XX:+PrintGCDateStamps
  • -XX:+UseConcMarkSweepGC
  • -XX:CMSInitiatingOccupancyFraction=70
  • -XX:MaxHeapFreeRatio=70
  • -XX:+CMSClassUnloadingEnabled
  • -XX:OnOutOfMemoryError='kill -9 %p'

If you want to update JVM settings, provide them in a single space-separated string. For example, for the spark:spark.driver.extraJavaOptions cluster property:

-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:MaxHeapFreeRatio=80

Spark settings for integration with Yandex Object StorageSpark settings for integration with Yandex Object Storage

The following settings are available for Apache Spark:

For more information, see the Apache Hadoop documentation.

Installing Python packagesInstalling Python packages

To install additional Python packages, you can use the conda or pip package managers. Provide the package name in the cluster properties as follows:

Using Apache Spark Thrift ServerUsing Apache Spark Thrift Server

You can use Apache Spark Thrift Server as a distributed SQL query engine on top of Apache Spark.

To enable it, set dataproc:hive.thrift.impl : spark, and the server will be available on TCP port 10000. The default value is dataproc:hive.thrift.impl : hive. It causes Apache HiveServer2 to launch on TCP port 10000 if the Hive service is being used.

This feature is available starting with image version 2.0.48.