Transferring data from a MongoDB source endpoint

1. Estimate the total number of databases for transfer and the total MongoDB workload. If the workload on the database exceeds 10,000 writes per second, create multiple endpoints and transfers. For more information, see Transferring data from a MongoDB source endpoint.

2. If not planning to use Cloud Interconnect or VPN for connections to an external cluster, make such cluster accessible from the Internet from IP addresses used by Data Transfer.

   For details on linking your network up with external resources, see this concept.

3. Make sure the MongoDB version on the target is 4.0 or higher.

4. Make sure the MongoDB cluster is configured so that it returns correctly resolving IP addresses or FQDNs (fully qualified domain names) in response to requests.

5. Configure access to the source cluster from Yandex Cloud. To configure a source cluster for connections from the internet:

   1. In the configuration file, change net.bindIp from 127.0.0.1 to 0.0.0.0:

      # network interfaces
      net:
        port: 27017
        bindIp: 0.0.0.0
      

   2. Restart mongod:

      sudo systemctl restart mongod.service
      

6. If the source cluster does not use replication, enable it:

   1. Add the replication settings to the /etc/mongod.conf configuration file:

      replication:
        replSetName: <replica_set_name>
      

   2. Restart mongod:

      sudo systemctl restart mongod.service
      

   3. Connect to MongoDB and initialize the replica set with this command:

      rs.initiate({
          _id: "<replica_set_name>",
          members: [{
              _id: 0,
              host: "<IP_address_listened_by_MongoDB>:<port>"
          }]
      });
      

7. Create a user with the readWrite role for all the source databases to replicate:

   use admin
   db.createUser({
       user: "<username>",
       pwd: "<password>",
       mechanisms: ["SCRAM-SHA-1"],
       roles: [
           {
               db: "<source_database_1_name>",
               role: "readWrite"
           },
           {
               db: "<source_database_2_name>",
               role: "readWrite"
           },
           ...
       ]
   });
   

   Once started, the transfer will connect to the source on behalf of this user. The readWrite role is required to enable the transfer to write data to the __data_transfer.__dt_cluster_time service collection.

   Note

   For MongoDB 3.6 or higher, you only need to assign the created user the read role for the databases to replicate.

8. When using MongoDB 3.6 or higher, to run the transfer, the user must have the read permission for the local.oplog.rs collection and the read and write permissions for the __data_transfer.__dt_cluster_time collection. To assign a user the clusterAdmin role granting these permissions, connect to MongoDB and run the following commands:

   use admin;
   db.grantRolesToUser("<username>", ["clusterAdmin"]);
   

   To grant more granular permissions, you can assign the clusterMonitor role required for reading the local.oplog.rs collection and grant read and write access to the __data_transfer.__dt_cluster_time system collection.

• Endpoint type: mongo-source.

• --cluster-id: ID of the cluster you need to connect to.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: mongo_source.

• connection.connection_options.mdb_cluster_id: ID of cluster to connect to.

• subnet_id: ID of the subnet the cluster is in. The transfer will use this subnet to access the cluster. If the ID is not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the cluster. For more information, see Networking in Yandex Data Transfer.

  Security groups and the subnet_id subnet, if the latter is specified, must belong to the same network as the cluster.

  Note

  In Terraform, it is not required to specify a network for security groups.

• auth_source: Name of the cluster database.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Here is the configuration file example:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    mongo_source {
      security_groups = ["<list_of_security_group_IDs>"]
      subnet_id       = "<subnet_ID>"
      connection {
        connection_options {
          mdb_cluster_id = "<cluster_ID>"
          auth_source    = "<DB_name>"
          user           = "<username>"
          password {
            raw = "<user_password>"
          }
        }
      }
      <additional_endpoint_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• mdbClusterId: ID of the cluster you need to connect to.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• Endpoint type: mongo-source.

• --host: IP address or FQDN of the master host you want to connect to.

• --port: Number of the port that Data Transfer will use for the connection.

• --ca-certificate: CA certificate if the data to transfer must be encrypted to comply with PCI DSS requirements.

  Warning

  If no certificate is added, the transfer may fail with an error.

• --subnet-id: ID of the subnet the host is in. The transfer will use that subnet to access the host.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: mongo_source.

• on_premise.port: Port number that Data Transfer will use for connections.

• connection.connection_options.on_premise.tls_mode.enabled.ca_certificate: CA certificate if the data to transfer must be encrypted, e.g., to comply with the PCI DSS requirements.

  Warning

  If no certificate is added, the transfer may fail with an error.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the VM with the database. For more information, see Networking in Yandex Data Transfer.

  Security groups must belong to the same network as the subnet_id subnet, if the latter is specified.

  Note

  In Terraform, it is not required to specify a network for security groups.

• subnet_id: ID of the subnet the cluster is in. The transfer will use this subnet to access the cluster. If the ID is not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• connection.connection_options.on_premise.replica_set: Specify the name of the replica set.

• connection.connection_options.on_premise.hosts: Specify the IP addresses or FQDN of the hosts to connect to.

• auth_source: Name of the cluster database.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Here is the configuration file example:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    mongo_source {
      security_groups = ["<list_of_security_group_IDs>"]
      subnet_id       = "<subnet_ID>"
      connection {
        connection_options {
          on_premise {
            hosts       = [ "list of replica set hosts" ]
            port        = "<port_for_connection>"
            replica_set = "<replica_set_name>"
            tls_mode {
              enabled {
                ca_certificate = "<certificate_in_PEM_format>"
              }
            }
          }
          auth_source = "<DB_name>"
          user        = "<username>"
          password {
            raw = "<user_password>"
          }
        }
      }
      <additional_endpoint_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• onPremise: Database connection parameters:

  • hosts — IP address or FQDN of the master host to connect to.

  • port: The number of the port that Data Transfer will use for the connection.

  • tlsMode: Parameters for encrypting the data to transfer, if required, e.g., for compliance with the PCI DSS requirements.

    • disabled: Disabled.
    • enabled: Enabled.

      • caCertificate: CA certificate.

        Warning

        If no certificate is added, the transfer may fail with an error.

  • subnetId: ID of the subnet the host is in. The transfer will use that subnet to access the host.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• --include-collection: Transfer data only from the listed collections. The values are specified in <database_name>.<collection_name> format. All collections are transferred by default.

  When you add new collections while editing an endpoint used in Snapshot and increment or Replication transfers with the Replicating status, the data history for these collections will not get uploaded. To add a collection with its historical data, use the List of objects for transfer field in the transfer settings.

• --exclude-collection: Transfer data from all collections except the specified ones. The values are specified in <database_name>.<collection_name> format.

• --prefer-secondary: Set to true to use replicas (if there are any in the cluster) instead of the master host to read data.

• collections: Transfer data only from the listed collections. All collections are transferred by default.

  When you add new collections while editing an endpoint used in Snapshot and increment or Replication transfers with the Replicating status, the data history for these collections will not get uploaded. To add a collection with its historical data, use the List of objects for transfer field in the transfer settings.

• excluded_collections: Data is transferred from all collections except the specified ones.

• secondary_preferred_mode: Set to true to use replicas (if there are any in the cluster) instead of the master host to read data.

• collections: Transfer data only from the listed collections. You need to specify the following for each collection:

  • databaseName: Database name
  • collectionName: Collection name

  All collections are transferred by default.

  When you add new collections while editing an endpoint used in Snapshot and increment or Replication transfers with the Replicating status, the data history for these collections will not get uploaded. To add a collection with its historical data, use the List of objects for transfer field in the transfer settings.

• excludedCollections: Transfer data from all collections except the specified ones. You need to specify the following for each collection:

  • databaseName: Database name
  • collectionName: Collection name

• secondaryPreferredMode: Set to true to use replicas (if there are any in the cluster) instead of the master host to read data.