Setting up Delta Lake in single-cluster mode

Yandex Data Processing 2.0 or higher supports using Delta Lake in single-cluster mode.

For more information about Delta Lake, see the Delta Lake in Yandex Data Processing section in the Delta Lake documentation.

Prepare the infrastructurePrepare the infrastructure

1. If you do not have a Yandex Data Processing cluster, create one.

2. If you attached a Yandex Object Storage bucket for data storage to your cluster:

   1. Create a folder named warehouse in the bucket.
   2. Set the spark.sql.warehouse.dir property to: s3a://<bucket_name>/warehouse/.

3. Create a Hive Metastore cluster and connect it to your Yandex Data Processing cluster.

Set up the component properties to work with Delta LakeSet up the component properties to work with Delta Lake

1. Set the following properties at cluster or individual job level:

   • Set spark.sql.extensions to io.delta.sql.DeltaSparkSessionExtension.
   • Set spark.sql.catalog.spark_catalog to org.apache.spark.sql.delta.catalog.DeltaCatalog.

2. Add the Delta Lake libraries to the dependencies of your cluster or individual job (the required library versions depend on the Yandex Data Processing version):

   Yandex Data Processing 2.0.x

   Yandex Data Processing 2.1.0 or 2.1.3

   Yandex Data Processing 2.1.4 and higher

   Use one of these methods:

   • Download the delta-core_2.12-0.8.0.jar library file, save it to your Object Storage bucket, and provide the file URL in the spark.jars property:

     spark.jars=s3a://<bucket_name>/<path_to_file>

     Make sure the cluster service account has read access to the bucket.

   • Set up cluster access to the Maven repository and set the spark.jars.packages property to io.delta:delta-core_2.12:0.8.0.

     You can set up Maven access in two ways:

     • In your cluster's security group, allow network access to the Maven Central repository.
     • Configure an alternative Maven repository and allow traffic to it in the cluster security group.

   • Download the delta-core_2.12-0.8.0.jar library file, copy it to all the cluster nodes manually or using initialization scripts, and provide the full file path in the spark.driver.extraClassPath and spark.executor.extraClassPath properties.

   Use one of these methods:

   • Download the delta-core_2.12-2.0.2.jar and delta-storage-2.0.2.jar library files, save them to your Object Storage bucket, and provide the comma-separated file URLs in the spark.jars property:

     spark.jars=s3a://<bucket_name>/<path_to_core_file>,s3a://<bucket_name>/<path_to_storage_file>

     Make sure the cluster service account has read access to the bucket.

   • Set up cluster access to the Maven repository and set the spark.jars.packages property to io.delta:delta-core_2.12:2.0.2,io.delta:delta-storage:2.0.2.

     You can set up Maven access in two ways:

     • In your cluster's security group, allow network access to the Maven Central repository.
     • Configure an alternative Maven repository and allow traffic to it in the cluster security group.

   • Download the delta-core_2.12-2.0.2.jar and delta-storage-2.0.2.jar library files, copy them to all the cluster nodes manually or using initialization scripts, and provide the full file path in the spark.driver.extraClassPath and spark.executor.extraClassPath properties.

   Use one of these methods:

   • Download the delta-core_2.12-2.3.0.jar and delta-storage-2.3.0.jar library files, save them to your Object Storage bucket, and provide the comma-separated file URLs in the spark.jars property:

     spark.jars=s3a://<bucket_name>/<path_to_core_file>,s3a://<bucket_name>/<path_to_storage_file>

     Make sure the cluster service account has read access to the bucket.

   • Set up cluster access to the Maven repository and set the spark.jars.packages property to io.delta:delta-core_2.12:2.3.0,io.delta:delta-storage:2.3.0.

     You can set up Maven access in two ways:

     • In your cluster's security group, allow network access to the Maven Central repository.
     • Configure an alternative Maven repository and allow traffic to it in the cluster security group.

   • Download the delta-core_2.12-2.3.0.jar and delta-storage-2.3.0.jar library files, copy them to all the cluster nodes manually or using initialization scripts, and provide the full file path in the spark.driver.extraClassPath and spark.executor.extraClassPath properties.

You can now use Delta Lake in your Yandex Data Processing cluster.

If the above Spark properties are provided at the cluster level, you can use Spark Thrift Server to work with Delta Lake tables.

Delta Lake usage exampleDelta Lake usage example

The use case was tested on a Yandex Data Processing cluster version 2.0 with access to the Maven Central repository.

1. Use SSH to connect to the Yandex Data Processing cluster's master host.

2. Run a Spark session in the cluster by providing the required parameters:

   spark-sql \
       --conf spark.jars.packages=io.delta:delta-core_2.12:0.8.0 \
       --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
       --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
   

3. Within the active session, create a database and switch to it:

   CREATE DATABASE testdelta;
   USE testdelta;
   

4. Create a test table and populate it with data:

   CREATE TABLE tab1(a INTEGER NOT NULL, b VARCHAR(100)) USING DELTA;
   INSERT INTO tab1 VALUES (1,'One'), (2,'Two'), (3,'Three');
   

5. Replace the b column values by adding to them the a column values converted to a string:

   UPDATE tab1 SET b=b || ' ** ' || CAST(a AS VARCHAR(10));
   

6. Check the result:

   SELECT * FROM tab1;
   

   3	Three ** 3
   2	Two ** 2
   1	One ** 1