Migrating an HDFS Yandex Data Processing cluster to a different availability zone

Written by

Updated at July 16, 2026

Getting started
- Required paid resources
Create a cluster via import in Terraform
Copy the data to the new cluster
Delete the source cluster

Subclusters of each Yandex Data Processing cluster reside in the same cloud network and availability zone. You can migrate a cluster to a different availability zone. The migration process depends on the cluster type:

The following describes how to migrate HDFS clusters.
For information on migrating lightweight clusters, check this guide.

Note

The Intel Broadwell platform is not available for clusters with hosts residing in the ru-central1-d availability zone.

To migrate an HDFS cluster:

To get started, create a subnet in the availability zone to which you are migrating the cluster.

Getting started

Navigate to the management console and log in to Yandex Cloud or create a new account.
On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resources

The cost of infrastructure for migrating to another availability zone includes a fee for Yandex Data Processing clusters: use of computing resources with a Yandex Data Processing markup, use of network drives, retrieval and storage of logs, amount of outgoing traffic (see Yandex Data Processing pricing).

Create a cluster via import in Terraform

To create a Yandex Data Processing cluster in a different availability zone with the same configuration as the source cluster, import the source cluster's configuration into Terraform:

Terraform

If you do not have Terraform yet, install it.
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
In your current working directory, create a .tf file with the following contents:
```
resource "yandex_dataproc_cluster" "old" { }
```
Save the ID of the source cluster to an environment variable:
```
export DATAPROC_CLUSTER_ID=<cluster_ID>
```
You can get the cluster ID with the list of clusters in the folder.

Import the source cluster configuration into Terraform:

terraform import yandex_dataproc_cluster.old ${DATAPROC_CLUSTER_ID}

Get the imported configuration:
```
terraform show
```
Copy it from your terminal and paste it into the .tf file.
Create a new directory imported-cluster and move your configuration file there.
Modify the configuration so that you can use it to create a new cluster:
- Specify the new cluster name in the resource string and the name parameter.
- Delete created_at, host_group_ids, id, and subcluster_spec.id.
- Change the availability zone in the zone_id parameter.
- In subnet_id of the subcluster_spec sections, specify the ID of the new subnet created in the required availability zone.
- Change the SSH key format in the ssh_public_keys parameter. Initial format:
```
ssh_public_keys = [
  <<-EOT
    <key>
  EOT,
]
```
  Required format:
```
ssh_public_keys = [
  "<key>"
]
```
Get the authentication credentials in the imported-cluster directory.
In the same directory, configure and initialize the provider. Instead of manually creating the provider configuration file, you can download it.
Move the configuration file to the imported-cluster directory and specify the arguments. If you have not set the authentication credentials as environment variables, specify them in the configuration file.
Make sure the Terraform configuration files are correct using this command:
```
terraform validate
```
Terraform will display any configuration errors detected in your files.
Create the required infrastructure:
1. Run this command to view the planned changes:
```
terraform plan
```
  If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
2. If everything looks correct, apply the changes:
  1. Run this command:
```
terraform apply
```
  2. Confirm updating the resources.
  3. Wait for the operation to complete.
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Copy the data to the new cluster

Make sure no operations or jobs are being performed on the HDFS files and directories you want to copy.

To view a list of running operations and jobs:
1. Open the management console.
2. Navigate to Yandex Data Processing.
3. Click the source cluster name and select the Operations tab, then select Jobs.
Note

Do not run any operations or jobs modifying the HDFS files and directories you are copying until the migration is complete.
Connect over SSH to the master host of the source cluster.
Get a list of directories and files to copy to the new cluster:
```
hdfs dfs -ls /
```
You can specify the directory you need instead of /.
To test copying data to the new Yandex Data Processing cluster, create test directories:
```
hdfs dfs -mkdir /user/foo &&\
hdfs dfs -mkdir /user/test
```
For demonstration, the example below copies only the /user/foo and /user/test test directories.
Connect over SSH to the master host of the new cluster.
Create a file named srclist:
```
nano srclist
```
Add to it a list of directories to migrate:
```
hdfs://<source_cluster_FQDN>:8020/user/foo
hdfs://<source_cluster_FQDN>:8020/user/test
```
In the command, specify the FQDN of the source cluster master host. Learn how to get an FQDN in this guide.
Place the srclist file to the /user HDFS directory:
```
hdfs dfs -put srclist /user
```
Create a directory to copy the data to. In this example, it is the copy directory nested in /user.
```
hdfs dfs -mkdir /user/copy
```
Copy the data between clusters using DistCp:
```
hadoop distcp -f hdfs://<new_cluster_FQDN>:8020/user/srclist \
hdfs://<new_cluster_FQDN>:8020/user/copy
```
In the command, specify the FQDN of the master host of the new cluster.

As a result, all directories and files specified in srclist will be copied to the /user/copy directory.

If you need to copy a large amount of data, use the -m <maximum_simultaneous_copies> flag in the command to limit network bandwidth consumption. For more information, see this DistCp guide.

You can check the data amount you copy in the HDFS web UI. To open it:
1. Open the management console.
2. Navigate to Yandex Data Processing.
3. Click the source cluster name.
4. On the cluster page, click the HDFS Namenode UI link under UI Proxy.
The DFS Used field shows the source cluster's data amount in HDFS.
Make sure the data is copied:
```
hdfs dfs -ls /user/copy
```

This way you can copy all the data you need. To do this, specify the required directories and files in srclist.

Delete the source cluster

Learn how to do this in this guide.

Migrating an HDFS Yandex Data Processing cluster to a different availability zone

Getting startedGetting started

Required paid resourcesRequired paid resources

Create a cluster via import in TerraformCreate a cluster via import in Terraform

Copy the data to the new clusterCopy the data to the new cluster

Delete the source clusterDelete the source cluster

Was the article helpful?

Getting started

Required paid resources

Create a cluster via import in Terraform

Copy the data to the new cluster

Delete the source cluster