Contact UsGet started

Reconfiguring a network connection when recreating a Yandex Data Processing cluster

Written by
Updated at May 5, 2025

You may need to recreate a cluster to install software updates, transfer the load across clusters, move clusters from one availability zone to another, and perform other operations.

The example below describes how to set up DNS to quickly switch network traffic over to new host FQDNs when recreating a Yandex Data Processing cluster. For the current name of the cluster master host, a network alias (CNAME record) is created in Yandex Cloud DNS. When you recreate the cluster, the CNAME record is changed to the master host's new name.

To set up DNS for your Yandex Data Processing cluster:

  1. Create a DNS zone and a CNAME record.
  2. Delete the cluster and recreate it.

If you no longer need the resources you created, delete them.

The support cost includes:

Getting started

Set up your infrastructure:

  1. Create a network named data-proc-network with the Create subnets option disabled.

  2. In data-proc-network, create a subnet with the following parameters:

    • Name: data-proc-subnet-a
    • Zone: ru-central1-a
    • CIDR: 192.168.1.0/24

  3. Create a NAT gateway and a route table named data-proc-route-table in data-proc-network. Associate the table with the data-proc-subnet-a subnet.

  4. In the data-proc-network network, create a security group named data-proc-security-group with the following rules:

    • One rule for inbound and another one for outbound service traffic:

      • Port range: 0-65535
      • Protocol: Any
      • Source/Destination name: Security group
      • Security group: Current

    • Rule for outgoing HTTPS traffic:

      • Port range: 443
      • Protocol: TCP
      • Destination name: CIDR
      • CIDR blocks: 0.0.0.0/0

  5. Create a service account named data-proc-sa with the following roles:

  6. Create an Yandex Object Storage bucket with restricted access.

  7. Create a Yandex Data Processing cluster of any suitable configuration with the following settings:

    • Environment: PRODUCTION.
    • Service account: data-proc-sa.
    • Bucket ID format: List.
    • Bucket name: Select the bucket you created earlier.
    • Network: data-proc-network.
    • Security groups: data-proc-security-group.

  1. If you do not have Terraform yet, set up and configure it.

  2. Download the file with provider settings. Place it in a separate working directory and specify the parameter values.

  3. Download the data-proc-dns-connect.tf configuration file to the same working directory.

    This file describes:

    • Network.
    • Subnet.
    • DNS zone and CNAME record for the cluster master host.
    • NAT gateway and route table.
    • Security groups.
    • Service account to work with cloud resources.
    • Service account for bucket management.
    • Static access key required to grant the service account the required permissions for the bucket.
    • Bucket to store job dependencies and results.
    • Yandex Data Processing cluster.

  4. In the data-proc-dns-connect.tf file, specify these variables:

    • folder_id: Folder ID.
    • path_to_ssh_public_key: Path to the public SSH key.
    • os_sa_name: Name of the service account for bucket management.
    • bucket: Bucket name.

  5. Run the terraform init command in the working directory with the configuration files. This command initializes the provider specified in the configuration files and enables you to use the provider resources and data sources.

  6. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  7. Create the required infrastructure:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Create a DNS zone and a CNAME record

Create the resources:

  1. Create an internal DNS zone with the following settings:

    • Zone: data-proc-test-user.org..
    • Networks: Select the data-proc-network network.
    • Name: dp-private-zone.

  2. Create a DNS record of the CNAME type with the following settings:

  1. Get the FQDN of the Yandex Data Processing cluster master host.

  2. In the data-proc-dns-connect.tf file, specify this variable:

    • dataproc_fqdn: FQDN of the Yandex Data Processing cluster master host.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Create the required infrastructure:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

Test network access to the cluster by the CNAME record:

dig data-proc-test-user.org.
<...>
;; ANSWER SECTION:
data-proc-test-user.org. 600	IN	CNAME	rc1a-dataproc-m-6ijqng07vul2mu8j.mdb.yandexcloud.net.
rc1a-dataproc-m-6ijqng07vul2mu8j.mdb.yandexcloud.net. 600 IN A 192.168.1.8

Delete the cluster and recreate it

  1. Delete the Yandex Data Processing cluster and create a new one with identical characteristics.
  2. Change the DNS record that you created earlier and specify the FQDN of the master host of the newly created cluster in the Data parameter.

  1. Delete the yandex_dataproc_cluster section in data-proc-dns-connect.tf.

  2. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  3. Apply the changes:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

  4. Add the yandex_dataproc_cluster section to data-proc-dns-connect.tf, with the same contents as in the source file, to create a new Yandex Data Processing cluster.

  5. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  6. Create a cluster:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

  7. Get the FQDN of the master host of the newly created Yandex Data Processing cluster.

  8. In the data-proc-dns-connect.tf file, specify this variable:

    • dataproc_fqdn: FQDN of the cluster master host.

  9. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  10. Apply the changes:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

Check if you still have network access to the cluster by the CNAME record:

dig data-proc-test-user.org.
<...>
;; ANSWER SECTION:
data-proc-test-user.org. 600	IN	CNAME	rc1a-dataproc-m-lsqohjh53rfu659d.mdb.yandexcloud.net.
rc1a-dataproc-m-8kompl81232cdsu8j.mdb.yandexcloud.net. 600 IN A 192.168.1.8

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  1. In the terminal window, go to the directory containing the infrastructure plan.

    Warning

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

  2. Delete resources:

    1. Run this command:

      terraform destroy

    2. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.

Previous
Configuring a network for Yandex Data Processing
Next
Connecting to Object Storage from VPC