Transferring data from a MongoDB source endpoint

1. Estimate the total number of databases for transfer and the total MongoDB workload. If the workload on the database exceeds 10,000 writes per second, create multiple endpoints and transfers. For more information, see Transferring data from a MongoDB source endpoint.

2. Make sure the settings for the network hosting the cluster allow public connections from IP addresses used by Data Transfer.

3. Make sure that the MongoDB version on the target is 4.0 or higher.

4. Make sure the MongoDB cluster is configured so that it returns correctly resolving IP addresses or FQDNs (fully qualified domain names) in response to requests.

5. Configure access to the source cluster from Yandex Cloud. To configure a source cluster for connections from the internet:

   1. In the configuration file, change the net.bindIp setting from 127.0.0.1 to 0.0.0.0:

      # network interfaces
      net:
        port: 27017
        bindIp: 0.0.0.0
      

   2. Restart the mongod service:

      sudo systemctl restart mongod.service
      

6. If the source cluster does not use replication, enable it:

   1. Add replication settings to the /etc/mongod.conf configuration file:

      replication:
        replSetName: <replica_set_name>
      

   2. Restart the mongod service:

      sudo systemctl restart mongod.service
      

   3. Connect to MongoDB and initialize the replica set with this command:

      rs.initiate({
          _id: "<replica_set_name>",
          members: [{
              _id: 0,
              host: "<IP_address_listening_to_MongoDB>:<port>"
          }]
      });
      

7. Create a user with the readWrite role for all source databases to be replicated:

   use admin
   db.createUser({
       user: "<username>",
       pwd: "<password>",
       mechanisms: ["SCRAM-SHA-1"],
       roles: [
           {
               db: "<source_database_1_name>",
               role: "readWrite"
           },
           {
               db: "<source_database_2_name>",
               role: "readWrite"
           },
           ...
       ]
   });
   

   Once started, the transfer will connect to the source on behalf of this user. The readWrite role is required so that a transfer can write data to the __data_transfer.__dt_cluster_time service collection.

   Note

   For MongoDB 3.6 or higher, it is enough to assign the created user the read role for databases to replicate.

8. When using MongoDB 3.6 or higher, to run the transfer, the user must have the read permission to the local.oplog.rs collection and the read and write permissions to the __data_transfer.__dt_cluster_time collection. To assign a user the clusterAdmin role granting these privileges, connect to MongoDB and run the following commands:

   use admin;
   db.grantRolesToUser("<username>", ["clusterAdmin"]);
   

   To issue more granular privileges, you can assign the clusterMonitor role required for read access to the local.oplog.rs and grant read and write access to the __data_transfer.__dt_cluster_time system collection.

• Endpoint type: mongo-source.

• --cluster-id: ID of the cluster you need to connect to.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: mongo_source.

• connection.connection_options.mdb_cluster_id: ID of cluster to connect to.

• subnet_id: ID of the subnet hosting the cluster. If not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the cluster. For more information, see Networking in Yandex Data Transfer.

  Security groups and the subnet_id subnet, if the latter is specified, must belong to the same network as the cluster.

  Note

  In Terraform, it is not required to specify a network for security groups.

• auth_source: Name of the cluster database.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Here is an example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    mongo_source {
      security_groups = ["<list_of_security_group_IDs>"]
      subnet_id       = "<subnet_ID>"
      connection {
        connection_options {
          mdb_cluster_id = "<cluster_ID>"
          auth_source    = "<DB_name>"
          user           = "<username>"
          password {
            raw = "<user_password>"
          }
        }
      }
      <additional_endpoint_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• mdbClusterId: ID of the cluster you need to connect to.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• Endpoint type: mongo-source.

• --host: IP address or FQDN of the master host you want to connect to.

• --port: Number of the port that Data Transfer will use for the connection.

• --ca-certificate — If the transmitted data needs to be encrypted, for example, to meet the requirements of PCI DSS.

• --subnet-id: ID of the subnet the host resides in.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: mongo_source.

• on_premise.port: Port number that Data Transfer will use for connections.

• connection.connection_options.on_premise.tls_mode.enabled.ca_certificate: CA certificate used if the data being transferred must be encrypted to comply with the PCI DSS requirements.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the VM with the database. For more information, see Networking in Yandex Data Transfer.

  Security groups must belong to the same network as the subnet_id subnet, if the latter is specified.

  Note

  In Terraform, it is not required to specify a network for security groups.

• subnet_id: ID of the subnet hosting the cluster. If not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• connection.connection_options.on_premise.replica_set: Specify the name of the replica set.

• connection.connection_options.on_premise.hosts: Specify the IP addresses or FQDN of the hosts to connect to.

• auth_source: Name of the cluster database.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Here is an example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    mongo_source {
      security_groups = ["<list_of_security_group_IDs>"]
      subnet_id       = "<subnet_ID>"
      connection {
        connection_options {
          on_premise {
            hosts       = [ "list of replica set hosts" ]
            port        = "<port_for_connection>"
            replica_set = "<replica_set_name>"
            tls_mode {
              enabled {
                ca_certificate = "<certificate_in_PEM_format>"
              }
            }
          }
          auth_source = "<DB_name>"
          user        = "<username>"
          password {
            raw = "<user_password>"
          }
        }
      }
      <additional_endpoint_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• onPremise: Database connection parameters:

  • hosts — IP address or FQDN of the master host to connect to.

  • port: The number of the port that Data Transfer will use for the connection.

  • tlsMode: Parameters of encryption of transmitted data if it is required, for example, to meet the requirements of PCI DSS.

  • subnetId: ID of the subnet the host resides in.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• --include-collection: Transfer data only from the listed collections. The values are specified in <database_name>.<collection_name> format. All collections are transferred by default.

  When you add new collections while editing an endpoint used in Snapshot and increment or Replication transfers with the Replicating status, the data history for these collections will not get uploaded. To add a collection with its historical data, use the List of objects for transfer field in the transfer settings.

• --exclude-collection: Transfer data from all collections except the specified ones. The values are specified in <database_name>.<collection_name> format.

• --prefer-secondary: Set to true to use replicas (if there are any in the cluster) instead of the master host to read data.

• collections: Transfer data only from the listed collections. All collections are transferred by default.

  When you add new collections while editing an endpoint used in Snapshot and increment or Replication transfers with the Replicating status, the data history for these collections will not get uploaded. To add a collection with its historical data, use the List of objects for transfer field in the transfer settings.

• excluded_collections: Data is transferred from all collections except the specified ones.

• secondary_preferred_mode: Set to true to use replicas (if there are any in the cluster) instead of the master host to read data.

• collections: Transfer data only from the listed collections. You need to specify the following for each collection:

  • databaseName: Database name
  • collectionName: Collection name

  All collections are transferred by default.

  When you add new collections while editing an endpoint used in Snapshot and increment or Replication transfers with the Replicating status, the data history for these collections will not get uploaded. To add a collection with its historical data, use the List of objects for transfer field in the transfer settings.

• excludedCollections: Transfer data from all collections except the specified ones. You need to specify the following for each collection:

  • databaseName: Database name
  • collectionName: Collection name

• secondaryPreferredMode: Set to true to use replicas (if there are any in the cluster) instead of the master host to read data.