Migrating collections from MongoDB to Yandex StoreDoc

To migrate collections from a third-party MongoDB cluster to your Yandex StoreDoc cluster, follow these steps: transfer the data, switch legacy databases to read-only mode, switch the workload to your Yandex Cloud cluster.

Required paid resourcesRequired paid resources

• Yandex StoreDoc cluster: computing resources allocated to hosts, storage and backup size (see Yandex StoreDoc pricing).
• Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• Per transfer: Computing resources used and the number of data rows transferred (see Data Transfer pricing).

Getting startedGetting started

Make sure you can connect to the source cluster’s hosts from the internet.

Migrating data using Yandex Data TransferMigrating data using Yandex Data Transfer

1. Prepare the source cluster.

2. Prepare the target cluster.

3. Create a source endpoint with the following parameters:

   • Database type: MongoDB.

   • Endpoint parameters → Connection settings: Custom installation.

     Configure the source cluster connection settings.

   Note

   Transferring of Time Series collections is not supported, so you should exclude such collections in the endpoint settings.

4. Create a target endpoint with the following parameters:

   • Database type: MongoDB.

   • Endpoint parameters → Connection settings: Yandex StoreDoc cluster.

     Specify the ID of the target cluster.

5. Create a Snapshot and increment-type transfer and configure it to use the previously created endpoints.

   To make large collections (over 1 GB) copy more quickly, enable parallel copy in the transfer settings. Specify two or more workers. The collection will be split into the specified number of parts that will be copied concurrently.

   For parallel copy to work, the _id field data type must be the same for all documents in the same collection. If a transfer discovers a type mismatch, the collection will not be partitioned but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection before starting a transfer.

   Note

   If a document with a different data type is added to a collection after a transfer starts, the transfer will move it at the replication stage after the parallel copy operation is completed. However, when re-activated, the transfer will not be able to partition a collection because the _id field type requirement will not be met for some of the documents in the collection.

6. Activate the transfer.

7. Wait for the transfer status to change to Replicating.

8. Switch the source cluster to "read-only" mode and transfer the load to the target cluster.

9. On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data copying was completed are transferred to the target cluster.

10. Deactivate the transfer and wait for its status to change to Stopped.

    For more information about transfer statuses, see Transfer lifecycle.

11. Delete the stopped transfer.

12. Delete the source and target endpoints.

Collection migration exampleCollection migration example

1. Configure the source cluster.
2. Create and configure the target cluster.
3. Set up the endpoints and transfer.
4. Transfer data.
5. Delete the resources you created.

Configure the source clusterConfigure the source cluster

1. Connect to the mongos host on the MongoDB source cluster via mongosh.

2. Create a database db1.

3. Create a user with owner privileges for db1 and log in as that user:

   use admin
   db.createUser({
     user: "user1",
     pwd: passwordPrompt(),
     roles: [
         {
             role: "dbOwner",
             db: "db1"
         }
     ]
   })
   
   db.auth("user1")
   

4. Populate collection1 in db1 with 200,000 random entries:

   use db1
   function randomName() {
     // Base 36 uses letters and digits to represent a number:
     return (Math.random()+1).toString(36).substring(2);
   }
   
   var day = 1000 * 60 * 60 * 24;
   function randomDate () {
     return new Date(Date.now() - (Math.floor(Math.random() * day)));
   }
   
   for (var i = 1; i <= 200000; ++i) {
     db.collection1.insertOne({
      name: randomName(),
      creationDate: randomDate(),
      uid: i
     });
   }
   

Create and configure the target clusterCreate and configure the target cluster

Set up the endpoints and transferSet up the endpoints and transfer

Transfer the dataTransfer the data

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Deactivate the transfer and wait for its status to change to Stopped.

   To learn more about the transfer lifecycle, see this Yandex Data Transfer guide.

2. Once the transfer is stopped, delete it.

3. Delete the source and target endpoints.

4. Delete the Yandex StoreDoc cluster.