Migrating collections from MongoDB to Managed Service for MongoDB

To migrate collections stored in a third-party MongoDB cluster to a Managed Service for MongoDB cluster, you need to transfer the data, write-lock the old database, and transfer the load to a Yandex Cloud cluster.

Required paid resourcesRequired paid resources

The support cost includes:

• Managed Service for MongoDB target cluster fee: Using computing resources allocated to hosts, and its disk space (see MongoDB pricing).
• Per-transfer fee: Using computing resources and the number of transferred data rows (see Data Transfer pricing).
• Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Getting startedGetting started

Make sure that you can connect to the source cluster hosts from the internet.

Migrating data using Yandex Data TransferMigrating data using Yandex Data Transfer

1. Prepare the source cluster.

2. Prepare the target cluster.

3. Create a source endpoint with the following parameters:

   • Database type: MongoDB

   • Endpoint parameters → Connection settings: Custom installation

     Specify the parameters for connecting to the source cluster.

   Note

   The service does not support transferring Time Series collections, so you should exclude these collections in the endpoint settings.

4. Create a target endpoint with the following parameters:

   • Database type: MongoDB

   • Endpoint parameters → Connection settings: Managed Service for MongoDB cluster

     Specify the ID of the target cluster.

5. Create a transfer of the Snapshot and increment type that will use the created endpoints.

   To make large collections (over 1 GB) copy more quickly, enable parallel copy in the transfer settings. Specify two or more workers. The collection will split into the specified number of parts that will be copied concurrently.

   For parallel copy to work, the data type in the _id field should be the same for all documents in a collection. If a transfer discovers a type mismatch, the collection will not be partitioned but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection before starting a transfer.

   Note

   If a document with a different data type is added to a collection after a transfer starts, the transfer will move it at the replication stage after the parallel copy operation is completed. However, when re-enabled, the transfer will not be able to partition a collection because the _id field's type requirement will not be met for some of the documents in the collection.

6. Activate the transfer.

7. Wait for the transfer status to change to Replicating.

8. Switch the source cluster to read-only and transfer the load to the target cluster.

9. On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data copying was completed are transferred to the target cluster.

10. Deactivate the transfer and wait for its status to change to Stopped.

    For more information about transfer statuses, see Transfer lifecycle.

11. Delete the stopped transfer.

12. Delete endpoints for both the source and target.

Example of migrating a collectionExample of migrating a collection

1. Set up the source cluster.
2. Create and configure a target cluster.
3. Set up endpoints and a transfer.
4. Transfer data.
5. Delete the resources you created.

Set up the source clusterSet up the source cluster

1. Connect to the mongos host of the MongoDB source cluster using mongosh.

2. Create a db1 database.

3. Create a user with the db1 owner permissions and log in with that user's credentials:

   use admin
   db.createUser({
     user: "user1",
     pwd: passwordPrompt(),
     roles: [
         {
             role: "dbOwner",
             db: "db1"
         }
     ]
   })
   
   db.auth("user1")
   

4. Fill in collection1 in db1 with random values. 200,000 entries will be added to the collection:

   use db1
   function randomName() {
     // Base 36 uses letters and digits to represent a number:
     return (Math.random()+1).toString(36).substring(2);
   }
   
   var day = 1000 * 60 * 60 * 24;
   function randomDate () {
     return new Date(Date.now() - (Math.floor(Math.random() * day)));
   }
   
   for (var i = 1; i <= 200000; ++i) {
     db.collection1.insertOne({
      name: randomName(),
      creationDate: randomDate(),
      uid: i
     });
   }
   

Create and configure a target clusterCreate and configure a target cluster

Set up endpoints and a transferSet up endpoints and a transfer

Transfer the dataTransfer the data

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Deactivate the transfer and wait for its status to change to Stopped.

   To learn more about the transfer lifecycle, see the Yandex Data Transfer documentation.

2. Delete the stopped transfer.

3. Delete endpoints for both the source and target.

4. Delete the created Managed Service for MongoDB cluster.