Migrating an Yandex StoreDoc (Managed Service for MongoDB) cluster from 4.4 to 6.0
You can migrate a production loaded sharded database deployed in a Yandex StoreDoc version 4.4 cluster to version 6.0.
To transfer data:
- Prepare the source cluster.
- Prepare the target cluster.
- Set up your transfers.
- Activate the transfers.
- Test your transfer.
If you no longer need the resources you created, delete them.
Required paid resources
- Yandex StoreDoc cluster: Computing resources allocated to hosts, storage size, and backup size (see Yandex StoreDoc pricing).
- Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
- Each transfer: Use of computing resources and number of transferred data rows (see Data Transfer pricing).
Getting started
Create a Yandex StoreDoc version 6.0 target cluster identical to the version 4.4 cluster.
-
Create a Yandex StoreDoc target cluster with the same configuration as the source cluster and with the following settings:
- Cluster version:
6.0. - Database name:
db1. - Username:
user1.
To connect to the cluster from the internet, enable public access to its hosts.
- Cluster version:
-
If using security groups, make sure they are configured correctly and allow connections to your cluster.
-
Grant the
readWriterole for thedb1database touser1. -
Enable cluster sharding and add the required number of shards.
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
In the same working directory, place a
.tffile with the following contents:resource "yandex_mdb_mongodb_cluster" "old" { } -
Write the Yandex StoreDoc version 4.4 cluster ID to an environment variable:
export MONGODB_CLUSTER_ID=<cluster_ID>You can get the ID with the list of clusters in the folder.
-
Import the Yandex StoreDoc version 4.4 cluster settings into the Terraform configuration:
terraform import yandex_mdb_mongodb_cluster.old ${MONGODB_CLUSTER_ID} -
Get the imported configuration:
terraform show -
Copy it from the terminal and paste it into the
.tffile. -
Place the file in the new
imported-clusterdirectory. -
Edit the copied configuration so that you can create a new cluster from it:
- Specify the new cluster name in the
resourcestring and thenameparameter. - Set the
versionparameter to6.0. - Delete
created_at,health,id,sharded, andstatus. - In the
hostsections, deletehealthandname. - If you have
type = "ANYTIME"in themaintenance_windowsection, delete thehourargument. - Delete all
usersections (if any). You can add database users with a separateyandex_mdb_mongodb_userresource. - Delete all
databasesections (if any). You can add databases using the separateyandex_mdb_mongodb_databaseresource. - Optionally, make further changes if you need to customize the configuration.
- Specify the new cluster name in the
-
Add the resource to create the database to the file:
resource "yandex_mdb_mongodb_database" "db1" { cluster_id = yandex_mdb_mongodb_cluster.<cluster_name>.id name = "db1" }Where
<cluster_name>is the new cluster name specified in theyandex_mdb_mongodb_clusterresource. -
Add the resource to the file to create a user named
user1:resource "yandex_mdb_mongodb_user" "user1" { cluster_id = yandex_mdb_mongodb_cluster.<cluster_name>.id name = "user1" password = "<user_password>" permission { database_name = "db1" roles = ["readWrite"] } depends_on = [ yandex_mdb_mongodb_database.db1 ] }Where
<cluster_name>is the new cluster name specified in theyandex_mdb_mongodb_clusterresource. -
Get the authentication credentials in the
imported-clusterdirectory. -
In the same directory, configure and initialize the provider. There is no need to create a provider configuration file manually, as you can download it
. -
Place the configuration file in the
imported-clusterdirectory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file. -
Make sure the Terraform configuration files are correct:
terraform validateTerraform will display any configuration errors detected in your files.
-
Create the required infrastructure:
-
Run this command to view the planned changes:
terraform planIf you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
-
If everything looks correct, apply the changes:
-
Run this command:
terraform apply -
Confirm updating the resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
Timeouts
The Terraform provider sets the following timeouts for Yandex StoreDoc cluster operations:
- Creating a cluster, including by restoring one from a backup: 30 minutes.
- Editing a cluster: 60 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts block to the cluster description, for example:
resource "yandex_mdb_mongodb_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # An hour and a half
update = "2h" # Two hours
}
}
Prepare the source cluster
-
Create a user with the
readWriterole for the source database you want to replicate. -
Delete unused collections from the database.
-
Disable unique indexes in the collections. They will be enabled after data migration.
-
Estimate your database workload. If it exceeds 10,000 writes per second, plan several transfers.
- Identify the high-workload collections.
- Distribute your collections between several transfers.
-
Set the oplog storage size with a 15-20% margin over the cluster disk size. This will allow Data Transfer to read changes from the source cluster throughout the data copying process.
Set up the target cluster
If the source database has sharded collections, prepare the target database. Do not enable unique indexes.
Set up the transfers
-
Create a source endpoint for each scheduled transfer and specify the endpoint parameters:
-
Database type:
MongoDB. -
Connection type:
Yandex StoreDoc cluster. -
Yandex StoreDoc cluster:
<Yandex_StoreDoc_source_cluster_name>from the drop-down list. -
Authentication source:
<source_cluster_database_name>. -
User:
<username>. -
Password:
<password>. -
Included collections: For each endpoint, specify the list of included collections that you allocated for each transfer.
-
Excluded collections: Specify
Time Seriescollections if your database has any. Data Transfer does not support migration of such collections.
-
-
Create a target endpoint for each planned transfer and specify endpoint parameters:
-
Database type:
MongoDB. -
Connection type:
Yandex StoreDoc cluster. -
Yandex StoreDoc cluster:
<Yandex_StoreDoc_target_cluster_name>from the drop-down list. -
Authentication source:
db1. -
User:
user1. -
Password:
<password>. -
Database:
db1.
If sharded collections have been created in the target database, select either the
Do not cleanorTRUNCATEcleanup policy.Selecting the
DROPpolicy will result in the service deleting all the data from the target database, including sharded collections, and replacing them with new unsharded ones when a transfer is activated. -
-
Create transfers of the Snapshot and replication type that will use the created endpoints.
To copy large collections (over 1 GB) faster, enable parallel copy in the transfer settings:
- Number of workers:
5or more - Number of threads:
8or more
The collection will be split into the specified number of parts that will be copied concurrently.
For parallel copy to work, the
_idfield data type must be the same for all documents in the same collection. If a transfer discovers a type mismatch, the collection will not be partitioned but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection before starting a transfer.Note
If a document with a different data type is added to a collection after a transfer starts, the transfer will move it at the replication stage after the parallel copy operation is completed. However, when re-activated, the transfer will not be able to partition a collection because the
_idfield type requirement will not be met for some of the documents in the collection. - Number of workers:
Activate the transfers
- Activate the transfers.
- Wait for the transfer status to change to Replicating.
- Switch the source cluster to read-only.
- If you disabled unique indexes in the source database, enable them in the target database.
- Switch the workload to the target cluster.
- On the transfer monitoring page, wait for the Maximum data transfer delay metric to reach zero for each transfer. This indicates that the target cluster now contains all changes made in the source cluster after the data copy completed.
- Deactivate the transfers and wait for their status to change to Stopped.
Test your transfer
-
Connect to the
db1database in the Yandex StoreDoc target cluster. -
Make sure the data collections have appeared in the
db1database:show collections db.<collection_name>.find()
Delete the resources you created
To reduce the consumption of resources you do not need, delete them:
-
Delete the Yandex StoreDoc cluster version
6.0depending on how it was created:ManuallyUsing TerraformDelete the Yandex StoreDoc cluster.
-
In the terminal window, go to the directory containing the infrastructure plan.
Warning
Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
-
Delete resources:
-
Run this command:
terraform destroy -
Confirm deleting the resources and wait for the operation to complete.
All the resources described in the Terraform manifests will be deleted.
-
-