Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Transferring metadata between Yandex Data Processing clusters using Apache Hive™ Metastore

Written by
Updated at October 15, 2025

You can transfer metadata between Yandex Data Processing clusters with the Hive DBMS. First, you need to export metadata from a cluster, then import it into a different one using Apache Hive™ Metastore.

To transfer metadata between Yandex Data Processing clusters:

  1. Create a test table.
  2. Export data.
  3. Connect Yandex Data Processing to Apache Hive™ Metastore.
  4. Import data.
  5. Check the result.

If you no longer need the resources you created, delete them.

Warning

If you want to configure an access policy for a bucket and connect to it from a Apache Hive™ Metastore cluster, you will need some additional infrastructure setup. For more information, see this guide.

Note

Apache Hive™ Metastore is at the Preview stage.

The infrastructure support cost includes:

Getting started

Set up the infrastructure:

  1. Create a service account named dataproc-s3-sa and assign it the dataproc.agent, dataproc.provisioner, managed-metastore.integrationProvider, and storage.uploader roles.

  2. In Yandex Object Storage, create a bucket named dataproc-bucket. Grant the READ and WRITE permission for this bucket to the service account.

  3. Create a cloud network named dataproc-network.

  4. In this network, create a subnet named dataproc-subnet.

  5. Set up a NAT gateway for the subnet you created.

  6. Create a security group named dataproc-security-group with the following rules:

    Security group rules

    Target service for the rule

    Rule purpose

    Rule settings

    Yandex Data Processing

    For incoming service traffic.
    • Port range: 0-65535
    • Protocol: Any
    • Source: Security group
    • Security group: Self

    Yandex Data Processing

    For incoming traffic, to allow access to NTP servers for time syncing.
    • Port range: 123
    • Protocol: UDP
    • Source: CIDR
    • CIDR blocks: 0.0.0.0/0

    Yandex Data Processing

    For incoming traffic, to connect from the internet via SSH to subcluster hosts with public access.
    • Port range: 22
    • Protocol: TCP
    • Source: CIDR
    • CIDR blocks: 0.0.0.0/0

    Apache Hive™ Metastore

    For incoming client traffic.
    • Port range: 30000-32767
    • Protocol: Any
    • Source: CIDR
    • CIDR blocks: 0.0.0.0/0

    Apache Hive™ Metastore

    For incoming load balancer traffic.
    • Port range: 10256
    • Protocol: Any
    • Source: Load balancer health checks

    Yandex Data Processing

    For outgoing service traffic.
    • Port range: 0-65535
    • Protocol: Any
    • Source: Security group
    • Security group: Self

    Yandex Data Processing

    For outgoing HTTPS traffic.
    • Port range: 443
    • Protocol: TCP
    • Destination: CIDR
    • CIDR blocks: 0.0.0.0/0

    Yandex Data Processing

    For outgoing traffic, to allow access to NTP servers for time syncing.
    • Port range: 123
    • Protocol: UDP
    • Source: CIDR
    • CIDR blocks: 0.0.0.0/0

    Yandex Data Processing

    For outgoing traffic, to allow Yandex Data Processing cluster connections to Apache Hive™ Metastore.
    • Port range: 9083
    • Protocol: Any
    • Source: CIDR
    • CIDR blocks: 0.0.0.0/0

  7. Create two Yandex Data Processing clusters named dataproc-source and dataproc-target with the following settings:

    • Environment: PRODUCTION.

    • Services:

      • HDFS
      • HIVE
      • SPARK
      • YARN
      • ZEPPELIN

    • Service account: dataproc-s3-sa.

    • Availability zone: Zone where dataproc-subnet resides.

    • Properties: spark:spark.sql.hive.metastore.sharedPrefixes with the com.amazonaws,ru.yandex.cloud value. It is required for PySpark jobs and integration with Apache Hive™ Metastore.

    • Bucket name: dataproc-bucket.

    • Network: dataproc-network.

    • Security groups: dataproc-security-group.

    • UI Proxy: Enabled.

    • Subnet for the Yandex Data Processing subclusters: dataproc-subnet.

    • Public access for the master host: Enabled.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the metastore-import.tf configuration file to the same working directory.

    This file describes:

  6. Specify the following in metastore-import.tf:

    • folder_id: Cloud folder ID, same as in the provider settings.
    • dp_ssh_key: Absolute path to the public key for the Yandex Data Processing clusters. Learn more about connecting to a Yandex Data Processing host over SSH here.

  7. Validate your Terraform configuration files using this command:

    terraform validate

    Terraform will display any configuration errors detected in your files.

  8. Create the required infrastructure:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Create a test table

In the dataproc-source cluster, create a test table named countries:

  1. Navigate to the folder dashboard and select Yandex Data Processing.

  2. Open the dataproc-source cluster page.

  3. Click the Zeppelin Web UI link under UI Proxy.

  4. Select Notebook, then select Create new note.

  5. In the window that opens, specify the name for the note and click Create.

  6. To run a PySpark job, paste a Python script into the input line:

    %pyspark

from pyspark.sql.types import *

schema = StructType([StructField('Name', StringType(), True),
StructField('Capital', StringType(), True),
StructField('Area', IntegerType(), True),
StructField('Population', IntegerType(), True)])

df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema)
df.write.mode("overwrite").option("path","s3a://dataproc-bucket/countries").saveAsTable("countries")

  7. Click Run all paragraphs and wait until the job is complete.

  8. Replace the Python code in the input line with this SQL query:

    %sql

SELECT * FROM countries;

  9. Click Run all paragraphs.

    Result:

    |   Name    |  Capital |  Area   | Population |
| --------- | -------- | ------- | ---------- |
| Australia | Canberra | 7686850 | 19731984   |
| Austria   | Vienna   | 83855   | 7700000    |

Export data

To transfer data from one Yandex Data Processing cluster to another, back up the data in the dataproc-source cluster using pg_dump:

  1. Use SSH to connect to the dataproc-source cluster's master host:

    ssh ubuntu@<master_host_FQDN>

    Learn how to get the FQDN.

  2. Create a backup and save it to the metastore_dump.sql file:

    pg_dump --data-only --schema public postgres://hive:hive-p2ssw0rd@localhost/metastore > metastore_dump.sql

  3. Disconnect from the master host.

  4. Download the metastore_dump.sql file to your local current directory:

    scp ubuntu@<master_host_FQDN>:metastore_dump.sql .

  5. Upload the metastore_dump.sql file to the dataproc-bucket bucket.

Connect Yandex Data Processing to Apache Hive™ Metastore

  1. Create a Apache Hive™ Metastore cluster with the following parameters:

    • Service account: dataproc-s3-sa.
    • Network: dataproc-network.
    • Subnet: dataproc-subnet.
    • Security groups: dataproc-security-group.

  2. Add to the dataproc-target cluster settings the spark:spark.hive.metastore.uris property with the following value: thrift://<Apache Hive™ Metastore_cluster_IP_address>:9083.

    To find out the Apache Hive™ Metastore cluster IP address, select Yandex MetaData Hub in the management console and then select the Metastore page in the left-hand panel. Copy the IP address column value for the cluster in question.

Import data

  1. Open the Apache Hive™ Metastore cluster page.
  2. Click Import.
  3. In the window that opens, specify the dataproc-bucket and the metastore_dump.sql file.
  4. Click Import.
  5. Wait for the import to complete. You can check the import status on the Apache Hive™ Metastore cluster page under  Operations.

Check the result

  1. Open the dataproc-target cluster page.

  2. Click the Zeppelin Web UI link under UI Proxy.

  3. Select Notebook, then select Create new note.

  4. In the window that opens, specify the name for the note and click Create.

  5. Run the following SQL query:

    %sql

SELECT * FROM countries;

  6. Click Run all paragraphs.

    Result:

    |   Name    |  Capital |  Area   | Population |
| --------- | -------- | ------- | ---------- |
| Australia | Canberra | 7686850 | 19731984   |
| Austria   | Vienna   | 83855   | 7700000    |

The metadata from the dataproc-source cluster was successfully imported into the dataproc-target cluster.

Delete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

  1. Delete the Apache Hive™ Metastore cluster.

  2. Delete the objects from the bucket.

  3. Delete other resources depending on how they were created:

    1. In the terminal window, go to the directory containing the infrastructure plan.

      Warning

      Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

    2. Delete resources:

      1. Run this command:

        terraform destroy

      2. Confirm deleting the resources and wait for the operation to complete.

      All the resources described in the Terraform manifests will be deleted.

Apache® and Apache Hive™ are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Previous
Shared use of Yandex Data Processing tables through Apache Hive™ Metastore
Next
Working with an Object Storage table from a PySpark job