Contact UsGet started
© 2024 Direct Cursus Technology L.L.C.

Migrating a Yandex Managed Service for MongoDB cluster from version 4.4 to 6.0 using Yandex Data Transfer

Written by
Updated at December 24, 2024

You can migrate a production loaded sharded database deployed in a Managed Service for MongoDB version 4.4 cluster to version 6.0.

To transfer data:

  1. Prepare the source cluster.
  2. Prepare the target cluster.
  3. Set up your transfers.
  4. Activate the transfers.
  5. Test the transfer.

If you no longer need the resources you created, delete them.

Getting started

Create a Managed Service for MongoDB version 6.0 target cluster identical to the version 4.4 cluster.

  1. Create a Managed Service for MongoDB target cluster with the same configuration as the source cluster and with the following settings:

    • Cluster version: 6.0.
    • Database name: db1.
    • Username: user1.

    To connect to the cluster from the internet, enable public access to its hosts.

  2. If using security groups in your cluster, make sure they are configured correctly and allow connecting to the cluster.

  3. Grant the readWrite role for the db1 database to user1.

  4. Enable cluster sharding and add the required number of shards.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. In the same working directory, place a .tf file with the following contents:

    resource "yandex_mdb_mongodb_cluster" "old" { }

  6. Write the MongoDB version 4.4 cluster ID to an environment variable:

    export MONGODB_CLUSTER_ID=<cluster_ID>

    You can request the ID with the list of clusters in the folder.

  7. Import the MongoDB version 4.4 cluster settings into the Terraform configuration:

    terraform import yandex_mdb_mongodb_cluster.old ${MONGODB_CLUSTER_ID}

  8. Get the imported configuration:

    terraform show

  9. Copy it from the terminal and paste it into the .tf file.

  10. Place the file in the new imported-cluster directory.

  11. Modify the copied configuration so that you can create a new cluster from it:

    • Specify the new cluster name in the resource string and the name parameter.
    • Set the version parameter to 6.0.
    • Delete created_at, health, id, sharded, and status.
    • In the host sections, delete health and name.
    • If the maintenance_window section has type = "ANYTIME", delete the hour parameter.
    • Delete all user sections (if any). You can add database users using the separate yandex_mdb_mongodb_user resource.
    • Delete all database sections (if any). You can add databases using the separate yandex_mdb_mongodb_database resource.
    • Optionally, make further changes if you need to customize the configuration.

  12. Add the resource to create the database to the file:

    resource "yandex_mdb_mongodb_database" "db1" {
  cluster_id = yandex_mdb_mongodb_cluster.<cluster_name>.id
  name       = "db1"
}

    Where <cluster_name> is the new cluster name specified in the yandex_mdb_mongodb_cluster resource.

  13. Add the resource to the file to create a user named user1:

    resource "yandex_mdb_mongodb_user" "user1" {
  cluster_id = yandex_mdb_mongodb_cluster.<cluster_name>.id
  name       = "user1"
  password   = "<user_password>"
  permission {
    database_name = "db1"
    roles         = ["readWrite"]
  }
  depends_on = [
    yandex_mdb_mongodb_database.db1
  ]
}

    Where <cluster_name> is the new cluster name specified in the yandex_mdb_mongodb_cluster resource.

  14. Get the authentication credentials in the imported-cluster directory.

  15. In the same directory, configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  16. Place the configuration file in the imported-cluster directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  17. Check that the Terraform configuration files are correct:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  18. Create the required infrastructure:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Timeouts

The Terraform provider sets the following timeouts for Managed Service for MongoDB cluster operations:

  • Creating a cluster, including by restoring one from a backup: 30 minutes.
  • Editing a cluster: 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_mongodb_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # An hour and a half
    update = "2h"    # Two hours
  }
}

Prepare the source cluster

  1. Create a user with the readWrite role for the source database you want to replicate.

  2. Delete unused collections from the database.

  3. Disable unique indexes in the collections. They will be enabled after data migration.

  4. Estimate your database workload. If it exceeds 10,000 writes per second, plan several transfers.

    1. Identify the high-workload collections.
    2. Distribute your collections between several transfers.

  5. Set the oplog storage size with a 15-20% margin over the cluster disk size. This will allow Data Transfer to read changes from the source cluster throughout the data copying process.

    For more information about oplog, see the MongoDB documentation.

Prepare the target cluster

If the source database has sharded collections, prepare the target database. Do not enable unique indexes.

Set up the transfers

  1. Create a source endpoint for each scheduled transfer and specify the endpoint parameters:

    • Database type: MongoDB.

    • Connection type: Managed Service for MongoDB cluster.

    • Managed Service for MongoDB cluster: <MongoDB_source_cluster_name> from the drop-down list.

    • Authentication source: <source_cluster_database_name>.

    • User: <username>.

    • Password: <password>.

    • Included collections: For each endpoint, specify the list of included collections that you allocated for each transfer.

    • Excluded collections: Specify time series collections if your database has any. Data Transfer does not support migration of such collections.

  2. Create a target endpoint for each planned transfer and specify endpoint parameters:

    • Database type: MongoDB.

    • Connection type: Managed Service for MongoDB cluster.

    • Managed Service for MongoDB cluster: <MongoDB_target_cluster_name> from the drop-down list.

    • Authentication source: db1.

    • User: user1.

    • Password: <password>.

    • Database: db1.

    If sharded collections have been created in the target database, select either the Do not clean or TRUNCATE cleanup policy.

    Selecting the DROP policy will result in the service deleting all the data from the target database, including sharded collections, and replacing them with new unsharded ones when a transfer is activated.

  3. Create transfers of the Snapshot and replication type that will use the created endpoints.

    To copy large collections (over 1 GB) faster, enable parallel copy in the transfer settings:

    • Number of workers: 5 or more
    • Number of threads: 8 or more

    The collection will split into the specified number of parts that will be copied concurrently.

    For parallel copy to work, the data type in the _id field should be the same for all documents in a collection. If a transfer discovers a type mismatch, the collection will not be partitioned but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection before starting a transfer.

    Note

    If a document with a different data type is added to a collection after a transfer starts, the transfer will move it at the replication stage after the parallel copy operation is completed. However, when re-activated, the transfer will not be able to partition a collection because the _id field type requirement will not be met for some of the documents in the collection.

Activate the transfers

  1. Activate the transfers.
  2. Wait for the transfer status to change to Replicating.
  3. Switch the source cluster to read-only.
  4. If you disabled unique indexes in the source database, enable them in the target database.
  5. Transfer the load to the target cluster.
  6. On the transfer monitoring page, wait for the Maximum data transfer delay metric to reach zero for each transfer. This means that all changes that occurred in the source cluster after data was copied are transferred to the target cluster.
  7. Deactivate the transfers and wait for their status to change to Stopped.

Test the transfer

  1. Connect to db1 in the Managed Service for MongoDB target cluster.

  2. Make sure the data collections have appeared in the db1 database:

    show collections
db.<collection_name>.find()

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Delete the Managed Service for MongoDB cluster version 6.0 depending on how it was created:

  1. In the terminal window, go to the directory containing the infrastructure plan.

    Warning

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

  2. Delete resources:

    1. Run this command:

      terraform destroy

    2. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.

Previous
Migrating data to Managed Service for MongoDB
Next
Sharding MongoDB collections