Contact UsGet started

Configuring a network for Yandex Data Processing

Written by
Updated at December 16, 2024

In this tutorial, you will learn how to create a Yandex Data Processing cluster and set up subnets and a NAT gateway.

Create resources

You have to create:

  • Network.
  • Subnet.
  • NAT gateway and routing table.
  • Security group for the cluster.
  • Service account for the cluster.
  • Bucket to store job dependencies and results.
  • Yandex Data Processing cluster.

  1. Create a network named data-proc-network with the Create subnets option disabled.

  2. In data-proc-network, create a subnet with the following parameters:

    • Name: data-proc-subnet-a
    • Zone: ru-central1-a
    • CIDR: 192.168.1.0/24

  3. Create a NAT gateway and a routing table named data-proc-route-table in data-proc-network. Associate the routing table with the data-proc-subnet-a subnet.

  4. In the data-proc-network subnet, create a security group named data-proc-security-group with the following rules:

    • One rule for inbound and another one for outbound service traffic:

      • Port range: 0-65535.
      • Protocol: Any.
      • Source/Destination name: Security group
      • Security group: Current.

    • Rule for outgoing HTTPS traffic:

      • Port range: 443.
      • Protocol: TCP.
      • Destination name: CIDR.
      • CIDR blocks: 0.0.0.0/0

    • Rule that allows access to NTP servers for time syncing:

      • Port range: 123
      • Protocol: UDP.
      • Destination name: CIDR.
      • CIDR blocks: 0.0.0.0/0

    Note

    You can add additional rules to a security group to connect to cluster hosts.

  5. Create a service account named data-proc-sa with the following roles:

  6. Create an Yandex Object Storage bucket with restricted access.

  7. Create a Yandex Data Processing cluster with any suitable configuration with the following settings:

    • Service account: data-proc-sa.
    • Bucket ID format: List.
    • Bucket name: Select the created bucket.
    • Network: data-proc-network.
    • Security groups: data-proc-security-group.

  1. If you don't have Terraform, install it and configure the Yandex Cloud provider.

  2. Download the file with the provider settings. Place it in a separate working directory and specify the parameter values.

  3. Download the cluster configuration file to the same working directory.

    The file describes:

    • Network.
    • Subnet.
    • NAT gateway and route table.
    • Security group.
    • Service account to work with cloud resources.
    • Bucket to store job dependencies and results.
    • Yandex Data Processing cluster.

    Note

    You can add additional rules to a security group to connect to cluster hosts.

  4. In the configuration file, specify all the relevant parameters.

  5. Run the terraform init command in the working directory hosting the configuration files. This command initializes the provider specified in the configuration files and enables you to use the provider resources and data sources.

  6. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  7. Create the required infrastructure:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

All the required resources will be created in the specified folder. You can check the new resources and their configuration using the management console.

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

To delete the infrastructure created with Terraform:

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the data-proc-nat-gateway.tf configuration file.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

  5. If you are happy with the planned changes, apply them:

    1. Run this command:

      terraform apply

    2. Confirm updating the resources.

    3. Wait for the operation to complete.

All the resources described in the configuration file will be deleted.

Previous
Configuring a secure GRE tunnel over IPsec
Next
Connecting to Object Storage from VPC