Shared use of Yandex Data Processing tables through Metastore
You can save data from a Yandex Data Processing cluster to a Yandex Object Storage bucket using a separate Hive Metastore cluster to store table metadata. Thus you can access the saved data from a different Yandex Data Processing cluster that has access to the bucket and is connected to the same Metastore cluster.
To set up shared use of tables with two Yandex Data Processing clusters through Metastore:
If you no longer need the resources you created, delete them.
If a Yandex Data Processing cluster contains tables that should be available in another Yandex Data Processing cluster, transfer the tables to the appropriate cluster using Metastore.
Getting started
Prepare the infrastructure:
-
Create a service account named
dataproc-s3-sa
and assign thedataproc.agent
anddataproc.provisioner
roles to it. -
In Object Storage, create buckets and configure access to them:
- Create a bucket for the input data and grant the cluster service account
READ
permissions for this bucket. - Create a bucket for the processing output and grant the cluster service account
READ and WRITE
permissions for this bucket.
- Create a bucket for the input data and grant the cluster service account
-
Create a cloud network named
dataproc-network
. -
In the
dataproc-network
network, create a subnet in any availability zone. -
Set up a NAT gateway for the subnet you created.
-
Create two Yandex Data Processing clusters named
dataproc-source
anddataproc-target
in any suitable host configuration with the following settings:- Services:
SPARK
YARN
- Service account:
dataproc-sa
. - Properties:
spark:spark.sql.hive.metastore.sharedPrefixes
with thecom.amazonaws,ru.yandex.cloud
value. Required for PySpark jobs and integration with Metastore. - Bucket name: Bucket you created for output data.
- Network:
dataproc-network
.
- Services:
-
If the cloud network uses security groups, add the following rule for outgoing traffic to the Yandex Data Processing cluster security group:
- Port range:
9083
. - Protocol:
Any
(Any
). - Source:
CIDR
. - CIDR blocks:
0.0.0.0/0
.
- Port range:
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
Download the dataproc-to-dataproc.tf
configuration file to the same working directory.This file describes:
- Network.
- Subnet.
- NAT gateway and route table required for Yandex Data Processing.
- Security groups required for the Yandex Data Processing clusters.
- Service account required for the Yandex Data Processing cluster.
- Service account required to create buckets in Object Storage.
- Buckets for input and output data.
- Two Yandex Data Processing clusters.
-
Specify the following in the
dataproc-to-dataproc.tf
file:folder_id
: Cloud folder ID, same as in the provider settings.input-bucket
: Name of the input datа bucket.output-bucket
: Name of the output datа bucket.dp_ssh_key
: Absolute path to the public key for the Yandex Data Processing clusters. For more information, see Connecting to a Yandex Data Processing host via SSH.
-
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
Connect Yandex Data Processing to Metastore
-
Create a Metastore cluster in the
dataproc-network
network. -
Add the
spark:spark.hive.metastore.uris
property with thethrift://<Metastore_cluster_IP_address>:9083
value to the cluster settings.To find out the Metastore cluster IP address, select Yandex Data Processing in the management console
and then select the Metastore page in the left-hand panel. You will see the cluster IP address under General information.
Create a test table
In the dataproc-source
cluster, create a test table named countries
and upload it to Object Storage:
-
Prepare a script file:
-
Create a local file named
create-table.py
and copy the following script to it:create-table.py
from pyspark.sql.types import * from pyspark.sql import SparkSession # Creating a Spark session spark = SparkSession.builder \ .appName("create-table") \ .enableHiveSupport() \ .getOrCreate() # Creating a data schema schema = StructType([StructField('Name', StringType(), True), StructField('Capital', StringType(), True), StructField('Area', IntegerType(), True), StructField('Population', IntegerType(), True)]) # Creating a dataframe df = spark.createDataFrame([('Australia', 'Canberra', 7686850, 19731984), ('Austria', 'Vienna', 83855, 7700000)], schema) # Writing the dataframe to a bucket as a _countries_ table df.write.mode("overwrite").option("path","s3a://<output_bucket_name>/countries").saveAsTable("countries")
-
In the script, specify the name of the output bucket where the file with the
countries
table will be saved. -
In the input bucket, create a folder named
scripts
and upload thecreate-table.py
file to it.
-
1.Create a PySpark job by specifying the path to the script file (s3a://<input_bucket_name>/scripts/create-table.py
) in the Main python file field.
- Wait for the job to complete and make sure the output bucket's
countries
folder contains thepart-00000-...
file.
The data from the created table is now stored in the Object Storage bucket and the table metadata is stored in the Metastore cluster. Now you can delete the dataproc-source
cluster.
Get data in the second cluster
Upload the countries
table metadata to the dataproc-target
cluster and make sure the table is accessible in the cluster for further operations:
-
Prepare a script file:
-
Create a local file named
obtain-table.py
and copy the following script to it:obtain-table.py
from pyspark.sql import SparkSession # Creating a Spark session spark = SparkSession.builder \ .appName("obtain-table") \ .enableHiveSupport() \ .getOrCreate() spark.catalog.listDatabases() # Getting the _countries_ table metadata from Metastore df = spark.sql("describe extended countries") # Requesting data from the _countries_ table df = spark.sql("select * from countries") # Testing table-to-bucket transfer df.repartition(1).write.csv(path='s3a://<output_bucket_name>/csv', header=True, sep=',')
-
In the script, specify the name of the output bucket the CSV file with the
countries
table will be saved to. -
Upload the
obtain-table.py
file to the source data bucket'sscripts
folder.
-
-
Create a PySpark job by specifying the path to the script file (
s3a://<input_bucket_name>/scripts/obtain-table.py
) in the Main python file field. -
Wait for the job to complete and make sure the output bucket contains the
csv
folder with a table in CSV format.
Delete the resources you created
Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:
-
Delete other resources depending on how they were created:
ManuallyTerraform-
Delete the objects from the buckets.
-
In the terminal window, go to the directory containing the infrastructure plan.
-
Delete the
dataproc-to-dataproc.tf
configuration file. -
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the resources described in the
dataproc-to-dataproc.tf
configuration file will be deleted. -
-