Transferring metadata between Yandex Data Processing clusters using Metastore

You can migrate metadata between Yandex Data Processing clusters with a Hive DBMS. First, you need to export metadata from a cluster, then import it into a different one using Hive Metastore.

To transfer metadata between Yandex Data Processing clusters:

1. Create a test table.
2. Export data.
3. Connect Yandex Data Processing to Metastore.
4. Import data.
5. Check the result.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The infrastructure support cost includes:

• Fee for the Yandex Data Processing cluster computing resources and storage (see Yandex Data Processing pricing).
• Fee for data storage and operations in a bucket (see Yandex Object Storage pricing).
• Fee for NAT gateway usage and outbound traffic (see Yandex Virtual Private Cloud pricing).

Getting startedGetting started

Set up your infrastructure:

Create a test tableCreate a test table

In the dataproc-source cluster, create a test table named countries:

1. Go to the folder page and select Yandex Data Processing.

2. Open the dataproc-source cluster page.

3. Click the Zeppelin Web UI link under UI Proxy.

4. Select Notebook, then select Create new note.

5. In the window that opens, specify the name for the note and click Create.

6. To run a PySpark job, paste a Python script into the input line:

   %pyspark
   
   from pyspark.sql.types import *
   
   schema = StructType([StructField('Name', StringType(), True),
   StructField('Capital', StringType(), True),
   StructField('Area', IntegerType(), True),
   StructField('Population', IntegerType(), True)])
   
   df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema)
   df.write.mode("overwrite").option("path","s3a://dataproc-bucket/countries").saveAsTable("countries")
   

7. Click Run all paragraphs and wait until the job is complete.

8. Replace the Python code in the input line with this SQL query:

   %sql
   
   SELECT * FROM countries;
   

9. Click Run all paragraphs.

   Result:

   |   Name    |  Capital |  Area   | Population |
   | --------- | -------- | ------- | ---------- |
   | Australia | Canberra | 7686850 | 19731984   |
   | Austria   | Vienna   | 83855   | 7700000    |
   

Export dataExport data

To transfer data from one Yandex Data Processing cluster to another, back up the data in the dataproc-source cluster using pg_dump:

1. Use SSH to connect to the dataproc-source cluster's master host:

   ssh ubuntu@<master_host_FQDN>
   

   You can learn how to get the FQDN here.

2. Create a backup and save it in the metastore_dump.sql file:

   pg_dump --data-only --schema public postgres://hive:hive-p2ssw0rd@localhost/metastore > metastore_dump.sql
   

3. Disconnect from the master host.

4. Download the metastore_dump.sql file to your local current directory:

   scp ubuntu@<master_host_FQDN>:metastore_dump.sql .
   

5. Upload the metastore_dump.sql file to the dataproc-bucket bucket.

Connect Yandex Data Processing to MetastoreConnect Yandex Data Processing to Metastore

1. Create a Metastore cluster with the following parameters:

   • Service account: dataproc-s3-sa
   • Network: dataproc-network
   • Subnet: dataproc-subnet
   • Security groups: dataproc-security-group

2. Add to the dataproc-target cluster settings the spark:spark.hive.metastore.uris property with the following value: thrift://<Metastore_cluster_IP_address>:9083.

   To find out the Metastore cluster IP address, select Yandex MetaData Hub in the management console and then select the Metastore page in the left-hand panel. Copy the IP address column value for the cluster in question.

Import dataImport data

1. Open the Metastore cluster page.
2. Click Import.
3. In the window that opens, specify the dataproc-bucket and the metastore_dump.sql file.
4. Click Import.
5. Wait for the import to complete. You can check the import status on the Metastore cluster page under  Operations.

Check the resultCheck the result

1. Open the dataproc-target cluster page.

2. Click the Zeppelin Web UI link under UI Proxy.

3. Select Notebook, then select Create new note.

4. In the window that opens, specify the name for the note and click Create.

5. Run the following SQL query:

   %sql
   
   SELECT * FROM countries;
   

6. Click Run all paragraphs.

   Result:

   |   Name    |  Capital |  Area   | Population |
   | --------- | -------- | ------- | ---------- |
   | Australia | Canberra | 7686850 | 19731984   |
   | Austria   | Vienna   | 83855   | 7700000    |
   

The metadata from the dataproc-source cluster was successfully imported into the dataproc-target cluster.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Delete the Metastore cluster.

2. Delete the objects from the bucket.

3. Delete other resources depending on how they were created:

   Manually

   Terraform

   1. Yandex Data Processing clusters.
   2. Object Storage bucket.
   3. Route table.
   4. NAT gateway.
   5. Security group.
   6. Subnet.
   7. Network.
   8. Service account.

   1. In the terminal window, go to the directory containing the infrastructure plan.

      Warning

      Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

   2. Delete resources:

      1. Run this command:

         terraform destroy
         

      2. Confirm deleting the resources and wait for the operation to complete.

      All the resources described in the Terraform manifests will be deleted.