Getting started with Hive Metastore

In Yandex MetaData Hub, you can create Hive Metastore clusters and use them to work with Yandex Data Processing clusters.

Getting startedGetting started

1. Go to the management console and log in to Yandex Cloud or sign up if not signed up yet.

2. If you do not have a folder yet, create one:

   1. In the management console, select the appropriate cloud from the list on the left.

   2. At the top right, click Create folder.

   3. Enter the folder name. The naming requirements are as follows:

      • The name must be from 3 to 63 characters long.
      • It may contain lowercase Latin letters, numbers, and hyphens.
      • The first character must be a letter and the last character cannot be a hyphen.

   4. (Optional) Enter a description of the folder.

   5. Select Create a default network. This will create a network with subnets in each availability zone. Within this network, a default security group will be created, inside which all network traffic is allowed.

   6. Click Create.

3. To link your service account to a Metastore cluster, make sure your Yandex Cloud account has the iam.serviceAccounts.user role or higher.

4. Set up a NAT gateway in the subnet to host Metastore and Yandex Data Processing clusters.

5. Create a security group for Metastore and Yandex Data Processing clusters.

6. Add rules Metastore cluster to the security group:

   • For incoming client traffic:

     • Port range: 30000-32767
     • Protocol: Any (Any)
     • Source: CIDR
     • CIDR blocks: 0.0.0.0/0

   • For incoming load balancer traffic:

     • Port range: 10256
     • Protocol: Any (Any)
     • Source: Load balancer healthchecks

7. Add Yandex Data Processing cluster rules to the security group:

   • One rule for inbound and another one for outbound service traffic:

     • Port range: 0-65535
     • Protocol: Any
     • Source/Destination name: Security group
     • Security group: Current

   • A separate rule for outgoing HTTPS traffic to all addresses. This will allow you to use Yandex Object Storage buckets, UI Proxy, and autoscaling of Yandex Data Processing subclusters.

     • Port range: 443
     • Protocol: TCP
     • Destination name: CIDR
     • CIDR blocks: 0.0.0.0/0

   • Rule that allows access to NTP servers for time syncing:

     • Port range: 123
     • Protocol: UDP
     • Destination name: CIDR
     • CIDR blocks: 0.0.0.0/0

8. Create a service account with the dataproc.agent, dataproc.provisioner, and managed-metastore.integrationProvider roles.

9. Create an Object Storage bucket to interact with a Yandex Data Processing cluster.

10. In the network you created earlier, create a Yandex Data Processing cluster. In the settings, set:

    • SPARK and YARN services.
    • Service account you previously created.
    • spark:spark.sql.hive.metastore.sharedPrefixes property to com.amazonaws,ru.yandex.cloud. Required for PySpark jobs and integration with Metastore.
    • Bucket you created earlier.
    • Security group you configured earlier.

Create a Metastore clusterCreate a Metastore cluster

Connect the Metastore cluster to the Yandex Data Processing clusterConnect the Metastore cluster to the Yandex Data Processing cluster

What's nextWhat's next

• Work with tables using Metastore.
• Use Metastore to move data between Yandex Data Processing clusters.
• Store tabular data in Metastore when using Apache Airflow™.
• Export and import Hive metadata in a Metastore cluster.