Migrating collections from MongoDB to Yandex StoreDoc

To migrate collections stored in a third-party MongoDB cluster to a Yandex StoreDoc cluster, you need to transfer the data, write-lock the old database, and transfer the load to a Yandex Cloud cluster.

Required paid resourcesRequired paid resources

The support cost for this solution includes:

• Yandex StoreDoc target cluster fee: use of computing resources allocated to hosts and disk space (see Yandex StoreDoc pricing).
• Per-transfer fee: use of computing resources and number of transferred data rows (see Data Transfer pricing).
• Fee for public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Getting startedGetting started

Make sure that you can connect to the source cluster hosts from the internet.

Migrating data using Yandex Data TransferMigrating data using Yandex Data Transfer

1. Prepare the source cluster.

2. Prepare the target cluster.

3. Create a source endpoint with the following parameters:

   • Database type: MongoDB.

   • Endpoint parameters → Connection settings: Custom installation.

     Configure the source cluster connection settings.

   Note

   Transferring of Time Series collections is not supported, so you should exclude such collections in the endpoint settings.

4. Create a target endpoint with the following parameters:

   • Database type: MongoDB.

   • Endpoint parameters → Connection settings: Yandex StoreDoc cluster.

     Specify the ID of the target cluster.

5. Create a Snapshot and increment-type transfer and configure it to use the previously created endpoints.

   To make large collections (over 1 GB) copy more quickly, enable parallel copy in the transfer settings. Specify two or more workers. The collection will be split into the specified number of parts that will be copied concurrently.

   For parallel copy to work, the _id field data type must be the same for all documents in the same collection. If a transfer discovers a type mismatch, the collection will not be partitioned but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection before starting a transfer.

   Note

   If a document with a different data type is added to a collection after a transfer starts, the transfer will move it at the replication stage after the parallel copy operation is completed. However, when re-activated, the transfer will not be able to partition a collection because the _id field type requirement will not be met for some of the documents in the collection.

6. Activate the transfer.

7. Wait for the transfer status to change to Replicating.

8. Switch the source cluster to "read-only" mode and transfer the load to the target cluster.

9. On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data copying was completed are transferred to the target cluster.

10. Deactivate the transfer and wait for its status to change to Stopped.

    For more information about transfer statuses, see Transfer lifecycle.

11. Delete the stopped transfer.

12. Delete the source and target endpoints.

Example of migrating a collectionExample of migrating a collection

1. Set up the source cluster.
2. Create and configure a target cluster.
3. Set up endpoints and a transfer.
4. Transfer data.
5. Delete the resources you created.

Set up the source clusterSet up the source cluster

1. Connect to the mongos host of the MongoDB source cluster using mongosh.

2. Create a db1.

3. Create a user with the db1 owner permissions and log in with that user's credentials:

   use admin
   db.createUser({
     user: "user1",
     pwd: passwordPrompt(),
     roles: [
         {
             role: "dbOwner",
             db: "db1"
         }
     ]
   })
   
   db.auth("user1")
   

4. Fill in collection1 in db1 with random values. 200,000 entries will be added to the collection:

   use db1
   function randomName() {
     // Base 36 uses letters and digits to represent a number:
     return (Math.random()+1).toString(36).substring(2);
   }
   
   var day = 1000 * 60 * 60 * 24;
   function randomDate () {
     return new Date(Date.now() - (Math.floor(Math.random() * day)));
   }
   
   for (var i = 1; i <= 200000; ++i) {
     db.collection1.insertOne({
      name: randomName(),
      creationDate: randomDate(),
      uid: i
     });
   }
   

Create and configure a target clusterCreate and configure a target cluster

Set up endpoints and a transferSet up endpoints and a transfer

Transfer the dataTransfer the data

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Deactivate the transfer and wait for its status to change to Stopped.

   To learn more about the transfer lifecycle, see this Yandex Data Transfer guide.

2. Delete the stopped transfer.

3. Delete endpoints for both the source and target.

4. Delete the created Yandex StoreDoc cluster.