Transferring data between Yandex Data Processing clusters using Metastore

You can migrate metadata between Yandex Data Processing clusters with a Hive DBMS. First, you need to export metadata from a cluster, then import it into a different one using Hive Metastore.

To transfer metadata between Yandex Data Processing clusters:

1. Create a test table.
2. Export data.
3. Connect Yandex Data Processing to Metastore.
4. Import data.
5. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Prepare the infrastructure:

Create a test tableCreate a test table

In the dataproc-source cluster, create a test table named countries:

1. Go to the folder page and select Yandex Data Processing.

2. Open the dataproc-source cluster page.

3. Click the Zeppelin Web UI link under UI Proxy.

4. Select Notebook, then select Create new note.

5. In the window that opens, specify the name for the note and click Create.

6. To run a PySpark job, paste a Python script into the input line:

   %pyspark
   
   from pyspark.sql.types import *
   
   schema = StructType([StructField('Name', StringType(), True),
   StructField('Capital', StringType(), True),
   StructField('Area', IntegerType(), True),
   StructField('Population', IntegerType(), True)])
   
   df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema)
   df.write.mode("overwrite").option("path","s3a://dataproc-bucket/countries").saveAsTable("countries")
   

7. Click Run all paragraphs and wait until the job is complete.

8. Replace the Python code in the input line with this SQL query:

   %sql
   
   SELECT * FROM countries;
   

9. Click Run all paragraphs.

   Result:

   |   Name    |  Capital |  Area   | Population |
   | --------- | -------- | ------- | ---------- |
   | Australia | Canberra | 7686850 | 19731984   |
   | Austria   | Vienna   | 83855   | 7700000    |
   

Export dataExport data

To transfer data from one Yandex Data Processing cluster to another, back up the data in the dataproc-source cluster using pg_dump:

1. Use SSH to connect to the dataproc-source cluster's master host:

   ssh ubuntu@<master_host_FQDN>
   

   You can learn how to get the FQDN here.

2. Create a backup and save it in the metastore_dump.sql file:

   pg_dump --data-only --schema public postgres://hive:hive-p2ssw0rd@localhost/metastore > metastore_dump.sql
   

3. Disconnect from the master host.

4. Download the metastore_dump.sql file to your local current directory:

   scp ubuntu@<master_host_FQDN>:metastore_dump.sql .
   

5. Upload the metastore_dump.sql file to the dataproc-bucket bucket.

Connect Yandex Data Processing to MetastoreConnect Yandex Data Processing to Metastore

1. Create a static access key for the dataproc-s3-sa service account. Save its ID and the secret key.

2. Create a Metastore cluster with the following parameters:

   • Network: dataproc-network
   • Subnet: dataproc-subnet
   • Security groups: dataproc-security-group
   • Access key ID and Secret key: Parameters of the static access key created earlier

3. Add to the dataproc-target cluster settings the spark:spark.hive.metastore.uris property with the following value: thrift://<Metastore_cluster_IP_address>:9083.

   To find out the Metastore cluster IP address, select Yandex Data Processing in the management console and then select the Metastore page in the left-hand panel. You will see the cluster IP address under General information.

Import dataImport data

1. Open the Metastore cluster page.
2. Click Import.
3. In the window that opens, specify the dataproc-bucket and the metastore_dump.sql file.
4. Click Import.
5. Wait for the import to complete. You can check the import status on the Metastore cluster page under  Operations.

Check the resultCheck the result

1. Open the dataproc-target cluster page.

2. Click the Zeppelin Web UI link under UI Proxy.

3. Select Notebook, then select Create new note.

4. In the window that opens, specify the name for the note and click Create.

5. Run the following SQL query:

   %sql
   
   SELECT * FROM countries;
   

6. Click Run all paragraphs.

   Result:

   |   Name    |  Capital |  Area   | Population |
   | --------- | -------- | ------- | ---------- |
   | Australia | Canberra | 7686850 | 19731984   |
   | Austria   | Vienna   | 83855   | 7700000    |
   

The metadata from the dataproc-source cluster was successfully imported into the dataproc-target cluster.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Delete the Metastore cluster.

2. Delete other resources depending on how they were created:

   Manually

   Terraform

   1. Yandex Data Processing clusters
   2. Object Storage bucket
   3. Route table
   4. NAT gateway
   5. Security group
   6. Subnet
   7. Network
   8. Service account

   1. Delete the objects from the bucket.

   2. In the terminal window, go to the directory containing the infrastructure plan.

   3. Delete the metastore-import.tf configuration file.

   4. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   5. Confirm updating the resources.

      1. Run the command to view planned changes:

         terraform plan
         

         If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

         1. Run the command:

            terraform apply
            

         2. Confirm the update of resources.

         3. Wait for the operation to complete.

      All the resources described in the metastore-import.tf configuration file will be deleted.