Getting started with Hive Metastore

In Yandex MetaData Hub, you can create Hive Metastore clusters and use them to work with Yandex Data Processing clusters.

Getting startedGetting started

1. Go to the management console and log in to Yandex Cloud or sign up if not signed up yet.

2. If you do not have a folder yet, create one:

   1. In the management console, select the appropriate cloud in the list on the left.

   2. At the top right, click Create folder.

   3. Enter the folder name. The naming requirements are as follows:

      • The name must be from 3 to 63 characters long.
      • It may contain lowercase Latin letters, numbers, and hyphens.
      • The first character must be a letter and the last character cannot be a hyphen.

   4. (Optional) Enter a description of the folder.

   5. Select Create a default network. This will create a network with subnets in each availability zone. Within this network, a default security group will be created, inside which all network traffic is allowed.

   6. Click Create.

3. Set up a NAT gateway in the subnet to host Metastore and Yandex Data Processing clusters.

4. Create a security group for Metastore and Yandex Data Processing clusters.

5. Add rules Metastore cluster to the security group:

   • For incoming client traffic:

     • Port range: 30000-32767
     • Protocol: Any (Any)
     • Source: CIDR
     • CIDR blocks: 0.0.0.0/0

   • For incoming load balancer traffic:

     • Port range: 10256
     • Protocol: Any (Any)
     • Source: Load balancer healthchecks

6. Add Yandex Data Processing cluster rules to the security group:

   • One rule for inbound and another one for outbound service traffic:

     • Port range: 0-65535
     • Protocol: Any
     • Source/Destination name: Security group
     • Security group: Current

   • A separate rule for outgoing HTTPS traffic to all addresses. This will allow you to use Yandex Object Storage buckets, UI Proxy, and autoscaling of Yandex Data Processing subclusters.

     • Port range: 443
     • Protocol: TCP
     • Destination name: CIDR
     • CIDR blocks: 0.0.0.0/0

   • Rule that allows access to NTP servers for time syncing:

     • Port range: 123
     • Protocol: UDP
     • Destination name: CIDR
     • CIDR blocks: 0.0.0.0/0

7. Create a service account for a Yandex Data Processing cluster with the dataproc.agent and dataproc.provisioner roles.

8. Create an Object Storage bucket to use with the Yandex Data Processing cluster.

9. In the network you created earlier, create a Yandex Data Processing cluster. Configure it as follows:

   • Select SPARK and YARN.
   • Set the spark:spark.sql.hive.metastore.sharedPrefixes property to com.amazonaws,ru.yandex.cloud.

Create a Metastore clusterCreate a Metastore cluster

Connect the Metastore cluster to the Yandex Data Processing clusterConnect the Metastore cluster to the Yandex Data Processing cluster

What's nextWhat's next

• Work with tables using Metastore.
• Use Metastore to move data between Yandex Data Processing clusters.
• Store tabular data in Metastore when using Apache Airflow™.
• Export and import Hive metadata in a Metastore cluster.