Migrating collections from MongoDB to Managed Service for MongoDB
To migrate collections stored in a third-party MongoDB cluster to a Managed Service for MongoDB cluster, you need to transfer the data, write-lock the old database, and transfer the load to a Yandex Cloud cluster.
Getting started
Make sure that you can connect to the source cluster hosts from the internet.
Migrating data using Yandex Data Transfer
-
Create a source endpoint with the following parameters:
-
Database type:
MongoDB
-
Endpoint parameters → Connection settings:
Custom installation
Specify the parameters for connecting to the source cluster.
Note
The service does not support transferring Time Series collections
, so you should exclude these collections in the endpoint settings. -
-
Create a target endpoint with the following parameters:
-
Database type:
MongoDB
-
Endpoint parameters → Connection settings:
Managed Service for MongoDB cluster
Specify the ID of the target cluster.
-
-
Create a transfer of the Snapshot and increment type that will use the created endpoints.
To make large collections (over 1 GB) copy more quickly, enable parallel copy in the transfer settings. Specify two or more workers. The collection will split into the specified number of parts that will be copied concurrently.
For parallel copy to work, the data type
in the_id
field should be the same for all documents in a collection. If a transfer discovers a type mismatch, the collection will not be partitioned but transferred in a single thread instead. If needed, remove documents with mismatched data types from the collection before starting a transfer.Note
If a document with a different data type is added to a collection after a transfer starts, the transfer will move it at the replication stage after the parallel copy operation is completed. However, when re-enabled, the transfer will not be able to partition a collection because the
_id
field's type requirement will not be met for some of the documents in the collection. -
Wait for the transfer status to change to Replicating.
-
Switch the source cluster to read-only and transfer the load to the target cluster.
-
On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data copying was completed are transferred to the target cluster.
-
Deactivate the transfer and wait for its status to change to Stopped.
For more information about transfer statuses, see Transfer lifecycle.
-
Delete endpoints for both the source and target.
Example of migrating a collection
- Set up the source cluster.
- Create and configure a target cluster.
- Set up endpoints and a transfer.
- Transfer data.
- Delete the resources you created.
Set up the source cluster
-
Connect to the
mongos
host of the MongoDB source cluster usingmongosh
. -
Create a database named
db1
. -
Create a user with the
db1
owner permissions and log in with that user's credentials:use admin db.createUser({ user: "user1", pwd: passwordPrompt(), roles: [ { role: "dbOwner", db: "db1" } ] }) db.auth("user1")
-
Fill in
collection1
indb1
with random values. 200,000 entries will be added to the collection:use db1 function randomName() { // Base 36 uses letters and digits to represent a number: return (Math.random()+1).toString(36).substring(2); } var day = 1000 * 60 * 60 * 24; function randomDate () { return new Date(Date.now() - (Math.floor(Math.random() * day))); } for (var i = 1; i <= 200000; ++i) { db.collection1.insertOne({ name: randomName(), creationDate: randomDate(), uid: i }); }
Create and configure a target cluster
- Create a Managed Service for MongoDB cluster with any suitable configuration.
- Create a database named
db1
. - Create a user named
user_transfer
with thereadWrite
role for the created database.
- Create a Managed Service for MongoDB cluster with any suitable configuration. There must be at least two hosts in the cluster.
- Enable sharding.
- Create a database named
db1
. - Create a user named
user_transfer
with thereadWrite
role for the created database and themdbShardingManager
role for theadmin
service base. - Following these steps, create an empty sharded collection named
collection1
indb1
and configure it.
Set up endpoints and a transfer
- Create an endpoint for the source cluster:
- Database type:
MongoDB
- Connection settings:
Custom installation
- CA certificate: Upload the certificate file if a connection to the source cluster requires encryption.
- Host list: Specify the FQDN of the source cluster hosts.
- Port: Specify the port for connecting to the hosts.
- Authentication source:
db1
- User:
user1
- Password: Enter the
user1
password.
- Database type:
- Create an endpoint for the target cluster:
- Database type:
MongoDB
- Connection settings:
MDB cluster
- Specify the ID of the target cluster.
- User:
user_transfer
- Password: Enter the
user_transfer
password. - Cleanup policy:
Drop
- Database type:
- Create a transfer:
- Transfer type: Snapshot and increment
- Source: Select the created endpoint for the source cluster.
- Target: Select the created endpoint for the target cluster.
- Create an endpoint for the source cluster:
- Database type:
MongoDB
- Connection settings:
Custom installation
- CA certificate: Upload the certificate file if a connection to the source cluster requires encryption.
- Host list: Specify the FQDN of the source cluster hosts.
- Port: Specify the port for connecting to the hosts.
- Authentication source:
db1
- User:
user1
- Password: Enter the
user1
password.
- Database type:
- Create an endpoint for the target cluster:
- Database type:
MongoDB
- Connection settings:
MDB cluster
- Specify the ID of the target cluster.
- User:
user_transfer
- Password: Enter the
user_transfer
password. - Cleanup policy:
Don't cleanup
orTruncate
.
- Database type:
- Create a transfer:
- Transfer type: Snapshot and increment
- Source: Select the created endpoint for the source cluster.
- Target: Select the created endpoint for the target cluster.
Transfer the data
-
Activate the created transfer.
-
Wait for the transfer status to change to Replicating.
-
Switch the cluster source to read-only and transfer the load to the target cluster.
-
On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data copying was completed are transferred to the target cluster.
-
Connect to the target cluster.
-
Make sure
collection1
has been transferred and contains 200,000 documents as in the source cluster:use db1 db.collection1.countDocuments()
200000
-
Activate the created transfer.
-
Wait for the transfer status to change to Replicating.
-
Switch the source cluster to read-only and transfer the load to the target cluster.
-
On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data copying was completed are transferred to the target cluster.
-
Connect to the target cluster.
-
Make sure
collection1
has been transferred, contains 200,000 documents as in the source cluster, and the documents are distributed across shards:use db1 db.collection1.countDocuments()
200000
db.collection1.getShardDistribution()
Shard rs01 at rs01/<host_1> { data: '7.05MiB', docs: 99752, chunks: 2, 'estimated data per chunk': '3.52MiB', 'estimated docs per chunk': 49876 } --- Shard rs02 at rs02/<host_2> { data: '7.08MiB', docs: 100248, chunks: 2, 'estimated data per chunk': '3.54MiB', 'estimated docs per chunk': 50124 } --- ...
Delete the resources you created
Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:
-
Deactivate the transfer and wait for its status to change to Stopped.
To learn more about the transfer lifecycle, see the Yandex Data Transfer documentation.
-
Delete the stopped transfer.