Transferring data between Yandex Data Processing clusters using Metastore

You can migrate metadata between Yandex Data Processing clusters with a Hive DBMS. First, you need to export metadata from a cluster, then import it into a different one using Hive Metastore.

To transfer metadata between Yandex Data Processing clusters:

1. Create a test table.
2. Export data.
3. Connect Yandex Data Processing to Metastore.
4. Import data.
5. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Prepare the infrastructure:

Create a test tableCreate a test table

In the dataproc-source cluster, create a test table named countries:

1. Go to the folder page and select Yandex Data Processing.

2. Open the dataproc-source cluster page.

3. Click the Zeppelin Web UI link under UI Proxy.

4. Select Notebook, then select Create new note.

5. In the window that opens, specify the name for the note and click Create.

6. To run a PySpark job, paste a Python script into the input line:

   %pyspark
   
   from pyspark.sql.types import *
   
   schema = StructType([StructField('Name', StringType(), True),
   StructField('Capital', StringType(), True),
   StructField('Area', IntegerType(), True),
   StructField('Population', IntegerType(), True)])
   
   df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema)
   df.write.mode("overwrite").option("path","s3a://dataproc-bucket/countries").saveAsTable("countries")
   

7. Click Run all paragraphs and wait until the job is complete.

8. Replace the Python code in the input line with this SQL query:

   %sql
   
   SELECT * FROM countries;
   

9. Click Run all paragraphs.

   Result:

   |   Name    |  Capital |  Area   | Population |
   | --------- | -------- | ------- | ---------- |
   | Australia | Canberra | 7686850 | 19731984   |
   | Austria   | Vienna   | 83855   | 7700000    |
   

Export dataExport data

To transfer data from one Yandex Data Processing cluster to another, back up the data in the dataproc-source cluster using pg_dump:

1. Use SSH to connect to the dataproc-source cluster's master host:

   ssh ubuntu@<master_host_FQDN>
   

   You can learn how to get the FQDN here.

2. Create a backup and save it in the metastore_dump.sql file:

   pg_dump --data-only --schema public postgres://hive:hive-p2ssw0rd@localhost/metastore > metastore_dump.sql
   

3. Disconnect from the master host.

4. Download the metastore_dump.sql file to your local current directory:

   scp ubuntu@<master_host_FQDN>:metastore_dump.sql .
   

5. Upload the metastore_dump.sql file to the dataproc-bucket bucket.

Connect Yandex Data Processing to MetastoreConnect Yandex Data Processing to Metastore

1. Create a Metastore cluster with the following parameters:

   • Service account: dataproc-s3-sa.
   • Network: dataproc-network.
   • Subnet: dataproc-subnet.
   • Security groups: dataproc-security-group.

2. Add to the dataproc-target cluster settings the spark:spark.hive.metastore.uris property with the following value: thrift://<Metastore_cluster_IP_address>:9083.

   To find out the Metastore cluster IP address, select Yandex Data Processing in the management console and then select the Metastore page in the left-hand panel. You will see the cluster IP address under General information.

Import dataImport data

1. Open the Metastore cluster page.
2. Click Import.
3. In the window that opens, specify the dataproc-bucket and the metastore_dump.sql file.
4. Click Import.
5. Wait for the import to complete. You can check the import status on the Metastore cluster page under  Operations.

Check the resultCheck the result

1. Open the dataproc-target cluster page.

2. Click the Zeppelin Web UI link under UI Proxy.

3. Select Notebook, then select Create new note.

4. In the window that opens, specify the name for the note and click Create.

5. Run the following SQL query:

   %sql
   
   SELECT * FROM countries;
   

6. Click Run all paragraphs.

   Result:

   |   Name    |  Capital |  Area   | Population |
   | --------- | -------- | ------- | ---------- |
   | Australia | Canberra | 7686850 | 19731984   |
   | Austria   | Vienna   | 83855   | 7700000    |
   

The metadata from the dataproc-source cluster was successfully imported into the dataproc-target cluster.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Delete the Metastore cluster.

2. Delete the objects from the bucket.

3. Delete other resources depending on how they were created:

   Manually

   Terraform

   1. Yandex Data Processing clusters.
   2. Object Storage bucket.
   3. Route table.
   4. NAT gateway.
   5. Security group.
   6. Subnet.
   7. Network.
   8. Service account.

   1. In the terminal window, go to the directory containing the infrastructure plan.

      Warning

      Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

   2. Delete resources:

      1. Run this command:

         terraform destroy
         

      2. Confirm deleting the resources and wait for the operation to complete.

      All the resources described in the Terraform manifests will be deleted.