Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Shared use of Yandex Data Processing tables through Metastore

Written by
Updated at May 5, 2025

You can save data from a Yandex Data Processing cluster to a Yandex Object Storage bucket using a separate Hive Metastore cluster to store table metadata. Thus you can access the saved data from a different Yandex Data Processing cluster that has access to the bucket and is connected to the same Metastore cluster.

To set up shared use of tables with two Yandex Data Processing clusters through Metastore:

  1. Connect Yandex Data Processing to Metastore.
  2. Create a test table.
  3. Get data in the second cluster.

If you no longer need the resources you created, delete them.

If a Yandex Data Processing cluster contains tables that should be available in another Yandex Data Processing cluster, transfer the tables to the appropriate cluster using Metastore.

Warning

If you want to configure an access policy for a bucket and connect to it from a Hive Metastore cluster, you will need some additional infrastructure setup. For more information, see this guide.

The infrastructure support cost includes:

Getting started

Set up your infrastructure:

  1. Create a service account named dataproc-s3-sa and assign the dataproc.agent and dataproc.provisioner roles to it.

  2. In Object Storage, create buckets and configure access to them:

    1. Create a bucket for the input data and grant the READ permission for this bucket to the cluster service account.
    2. Create a bucket for the processing output and grant the cluster service account READ and WRITE permissions for this bucket.

  3. Create a cloud network named dataproc-network.

  4. In the dataproc-network network, create a subnet in any availability zone.

  5. Set up a NAT gateway for the subnet you created.

  6. Create two Yandex Data Processing clusters named dataproc-source and dataproc-target in any suitable host configuration with the following settings:

    • Environment: PRODUCTION
    • Services:
      • SPARK
      • YARN
    • Service account: dataproc-sa.
    • Properties: spark:spark.sql.hive.metastore.sharedPrefixes with the com.amazonaws,ru.yandex.cloud value. Required for PySpark jobs and integration with Metastore.
    • Bucket name: Bucket you created for output data.
    • Network: dataproc-network.

  7. If the cloud network uses security groups, add the following rule for outgoing traffic to the Yandex Data Processing cluster security group:

    • Port range: 9083.
    • Protocol: Any (Any).
    • Source: CIDR.
    • CIDR blocks: 0.0.0.0/0.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the dataproc-to-dataproc.tf configuration file to the same working directory.

    This file describes:

    • Network.
    • Subnet.
    • NAT gateway and route table required for Yandex Data Processing.
    • Security groups required for the Yandex Data Processing clusters.
    • Service account required for the Yandex Data Processing cluster.
    • Service account required to create buckets in Object Storage.
    • Buckets for input and output data.
    • Two Yandex Data Processing clusters.

  6. Specify the following in the dataproc-to-dataproc.tf file:

    • folder_id: Cloud folder ID, same as in the provider settings.
    • input-bucket: Name of the input data bucket.
    • output-bucket: Name of the output data bucket.
    • dp_ssh_key: Absolute path to the public key for the Yandex Data Processing clusters. To learn more, see Connecting to a Yandex Data Processing host via SSH.

  7. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Connect Yandex Data Processing to Metastore

  1. Create a Metastore cluster in the dataproc-network network.

  2. Add the spark:spark.hive.metastore.uris property with the thrift://<Metastore_cluster_IP_address>:9083 value to the Yandex Data Processing cluster settings.

    To find out the Metastore cluster IP address, select Yandex MetaData Hub in the management console and then select the Metastore page in the left-hand panel. Copy the IP address column value for the cluster in question.

Create a test table

In the dataproc-source cluster, create a test table named countries and upload it to Object Storage:

  1. Prepare a script file:

    1. Create a local file named create-table.py and copy the following script to it:

      create-table.py
      from pyspark.sql.types import *
from pyspark.sql import SparkSession

# Creating a Spark session
spark = SparkSession.builder \
    .appName("create-table") \
    .enableHiveSupport() \
    .getOrCreate()

# Creating a data schema
schema = StructType([StructField('Name', StringType(), True),
StructField('Capital', StringType(), True),
StructField('Area', IntegerType(), True),
StructField('Population', IntegerType(), True)])

# Creating a dataframe
df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema)

# Writing the dataframe to a bucket as a _countries_ table
df.write.mode("overwrite").option("path","s3a://<output_bucket_name>/countries").saveAsTable("countries")

    2. In the script, specify the name of the output bucket where the file with the countries table will be saved.

    3. In the input bucket, create a folder named scripts and upload the create-table.py file to it.

  2. Create a PySpark job by specifying the path to the script file in the Main python file field: s3a://<input_bucket_name>/scripts/create-table.py.

  3. Wait for the job to complete and make sure the output bucket's countries folder contains the part-00000-... file.

The data from the created table is now stored in the Object Storage bucket and the table metadata is stored in the Metastore cluster. Now you can delete the dataproc-source cluster.

Get data in the second cluster

Upload the countries table metadata to the dataproc-target cluster and make sure the table is accessible in the cluster for further operations:

  1. Prepare a script file:

    1. Create a local file named obtain-table.py and copy the following script to it:

      obtain-table.py
      from pyspark.sql import SparkSession

# Creating a Spark session
spark = SparkSession.builder \
    .appName("obtain-table") \
    .enableHiveSupport() \
    .getOrCreate()

spark.catalog.listDatabases()

# Getting the _countries_ table metadata from Metastore
df = spark.sql("describe extended countries")

# Requesting data from the _countries_ table
df = spark.sql("select * from countries")

# Testing table-to-bucket transfer
df.repartition(1).write.csv(path='s3a://<output_bucket_name>/csv', header=True, sep=',')

    2. In the script, specify the name of the output bucket the CSV file with the countries table will be saved to.

    3. Upload the obtain-table.py file to the source data bucket’s scripts folder.

  2. Create a PySpark job by specifying the path to the script file in the Main python file field: s3a://<input_bucket_name>/scripts/obtain-table.py.

  3. Wait for the job to complete and make sure the output bucket contains the csv folder with a table in CSV format.

Delete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

  1. Delete the Metastore cluster.

  2. Delete the objects from the buckets.

  3. Delete other resources depending on how they were created:

    1. In the terminal window, go to the directory containing the infrastructure plan.

      Warning

      Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

    2. Delete resources:

      1. Run this command:

        terraform destroy

      2. Confirm deleting the resources and wait for the operation to complete.

      All the resources described in the Terraform manifests will be deleted.

Previous
Automating operations with Yandex Data Processing using Managed Service for Apache Airflow™
Next
Transferring metadata between Yandex Data Processing clusters using Metastore