Migrating a Yandex Managed Service for MongoDB cluster from version 4.4 to 6.0 using Yandex Data Transfer
You can migrate a production loaded sharded database deployed in a Managed Service for MongoDB version 4.4 cluster to version 6.0.
To transfer data:
- Prepare the source cluster.
- Prepare the target cluster.
- Set up your transfers.
- Activate the transfers.
- Test the transfer.
If you no longer need the resources you created, delete them.
Getting started
Create a Managed Service for MongoDB version 6.0 target cluster identical to the version 4.4 cluster.
-
Create a Managed Service for MongoDB target cluster with the same configuration as the source cluster and with the following settings:
- Cluster version:
6.0
- Database name:
db1
- Username:
user1
To connect to the cluster from the internet, enable public access to its hosts.
- Cluster version:
-
If using security groups in your cluster, make sure they are configured correctly and allow connecting to the cluster.
-
Assign the
readWrite
role for thedb1
database touser1
. -
Enable cluster sharding and add the required number of shards.
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
In the same working directory, place a
.tf
file with the following contents:resource "yandex_mdb_mongodb_cluster" "old" { }
-
Write the MongoDB version 4.4 cluster ID to an environment variable:
export MONGODB_CLUSTER_ID=<cluster_ID>
You can request the ID with a list of clusters in the folder.
-
Import the MongoDB version 4.4 cluster settings into the Terraform configuration:
terraform import yandex_mdb_mongodb_cluster.old ${MONGODB_CLUSTER_ID}
-
Get the imported configuration:
terraform show
-
Copy it from the terminal and paste it into the
.tf
file. -
Place the file in the new
imported-cluster
directory. -
Modify the copied configuration so that you can create a new cluster from it:
- Specify a new cluster name in the
resource
string and thename
parameter. - Specify version
6.0
in theversion
parameter. - Delete the
created_at
,health
,id
,sharded
, andstatus
parameters. - In the
host
sections, delete thehealth
andname
parameters. - If the
maintenance_window
section specifies thetype = "ANYTIME"
parameter value, delete thehour
parameter. - Delete all
user
sections (if any). You can add database users using the separateyandex_mdb_mongodb_user
resource. - Delete all
database
sections (if any). You can add databases using the separateyandex_mdb_mongodb_database
resource. - (Optional) Make further modifications if you are looking for more customization.
- Specify a new cluster name in the
-
Add the resource to create the database to the file:
resource "yandex_mdb_mongodb_database" "db1" { cluster_id = yandex_mdb_mongodb_cluster.<cluster_name>.id name = "db1" }
Where
<cluster_name>
is the new cluster name specified in theyandex_mdb_mongodb_cluster
resource. -
Add the resource to create
user1
to the file:resource "yandex_mdb_mongodb_user" "user1" { cluster_id = yandex_mdb_mongodb_cluster.<cluster_name>.id name = "user1" password = "<user_password>" permission { database_name = "db1" roles = ["readWrite"] } depends_on = [ yandex_mdb_mongodb_database.db1 ] }
Where
<cluster_name>
is the new cluster name specified in theyandex_mdb_mongodb_cluster
resource. -
In the
imported-cluster
directory, get the authentication data. -
In the same directory, configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in the
imported-cluster
directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file. -
Check that the Terraform configuration files are correct:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
Timeouts
The Terraform provider sets the following timeouts for Managed Service for MongoDB cluster operations:
- Creating a cluster, including by restoring one from a backup: 30 minutes.
- Editing a cluster: 60 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_mongodb_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # An hour and a half
update = "2h" # Two hours
}
}
Prepare the source cluster
-
Create a user with the
readWrite
role for the source database you want to replicate. -
Delete unused collections from the database.
-
Disable unique indexes in the collections. They will be enabled after data migration.
-
Estimate your database workload. If it exceeds 10,000 writes per second, plan several transfers.
- Identify the high-workload collections.
- Distribute your collections between several transfers.
-
Set the oplog storage size with a 15-20% margin over the cluster disk size. This will allow Data Transfer to read changes from the source cluster throughout the data copying process.
For more information about oplog, see the MongoDB documentation
.
Prepare the target cluster
If the source database has sharded collections, prepare the target database. Do not enable unique indexes.
Set up the transfers
-
Create a source endpoint for each scheduled transfer and specify the endpoint parameters:
-
Database type:
MongoDB
. -
Connection type:
Managed Service for MongoDB cluster
. -
Managed Service for MongoDB cluster:
<name_of_MongoDB_source_cluster>
from the drop-down list. -
Authentication source:
<source_cluster_database_name>
. -
User:
<username>
. -
Password:
<password>
. -
Included collections: For each endpoint, specify the list of included collections that you allocated for each transfer.
-
Excluded collections: Specify Time Series collections
, if your database has any. Data Transfer does not support the transfer of such collections.
-
-
Create a target endpoint for each planned transfer and specify endpoint parameters:
-
Database type:
MongoDB
. -
Connection type:
Managed Service for MongoDB cluster
. -
Managed Service for MongoDB cluster:
<name_of_MongoDB_target_cluster>
from the drop-down list. -
Authentication source:
db1
. -
User:
user1
. -
Password:
<password>
. -
Database:
db1
.
If sharded collections have been created in the target database, select either the
Do not purge
orTRUNCATE
purge policy.Selecting the
DROP
policy will result in the service deleting all the data from the target database, including sharded collections, and replacing them with new unsharded ones when a transfer is activated. -
-
Create transfers of the Snapshot and replication type that will use the created endpoints.
To copy large collections (over 1 GB) faster, enable parallel copy in the transfer settings:
- Number of workers:
5
or more - Number of threads:
8
or more
The collection will split into the specified number of parts that will be copied concurrently.
For parallel copy to work, the data type
in the_id
field should be the same for all documents in a collection. If a transfer discovers a type mismatch, the collection will not be partitioned but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection before starting a transfer.Note
If a document with a different data type is added to a collection after a transfer starts, the transfer will move it at the replication stage after the parallel copy operation is completed. However, when re-enabled, the transfer will not be able to partition a collection because the
_id
field's type requirement will not be met for some of the documents in the collection. - Number of workers:
Activate the transfers
- Activate the transfers.
- Wait for the transfer status to change to Replicating.
- Switch the source cluster to read-only.
- If you disabled unique indexes in the source database, enable them in the target database.
- Transfer the load to the target cluster.
- On the transfer monitoring page, wait for the Maximum data transfer delay metric to reach zero for each transfer. This means that all changes that occurred in the source cluster after data was copied are transferred to the target cluster.
- Deactivate the transfers and wait for their status to change to Stopped.
Test the transfer
-
Connect to the
db1
database in the Managed Service for MongoDB target cluster. -
Check whether the data collections have appeared in the
db1
database:show collections db.<collection_name>.find()
Delete the resources you created
Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:
Delete the Managed Service for MongoDB version 6.0
cluster depending on how it was created:
Delete the Managed Service for MongoDB cluster.
If you created your cluster using Terraform:
-
In the terminal window, go to the directory containing the infrastructure plan.
-
Delete the configuration file with the
.tf
extension. -
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the resources described in the deleted configuration file will be deleted.
-