Transferring data to a MongoDB target endpoint

1. If not planning to use Cloud Interconnect or VPN for connections to an external cluster, make such cluster accessible from the Internet from IP addresses used by Data Transfer.

   For details on linking your network up with external resources, see this concept.

2. Make sure the MongoDB version on the target is not lower than that on the source.

3. Make sure the MongoDB cluster is configured so that it returns correctly resolving IP addresses or FQDNs (fully qualified domain names) in response to requests.

4. Configure the target cluster to allow connections from the internet:

   1. In the configuration file, change net.bindIp from 127.0.0.1 to 0.0.0.0:

      # network interfaces
      net:
        port: 27017
        bindIp: 0.0.0.0
      

   2. Restart mongod:

      sudo systemctl restart mongod.service
      

5. If the target cluster does not use replication, enable it:

   1. Add the replication settings to the /etc/mongod.conf configuration file:

      replication:
        replSetName: <replica_set_name>
      

   2. Restart mongod:

      sudo systemctl restart mongod.service
      

   3. Connect to MongoDB and initialize the replica set with this command:

      rs.initiate({
          _id: "<replica_set_name>",
          members: [{
              _id: 0,
              host: "<IP_address_listened_by_MongoDB>:<port>"
          }]
      });
      

6. Connect to the cluster and create a target database:

   use <database_name>
   

7. Create a user with the readWrite role for the target database:

   use admin;
   db.createUser({
       user: "<username>",
       pwd: "<password>",
       mechanisms: ["SCRAM-SHA-1"],
       roles: [
           {
               db: "<target_database_name>",
               role: "readWrite"
           }
       ]
   });
   

   Once started, the transfer will connect to the target on behalf of this user.

8. To shard the migrated collections in the target cluster:

   1. Set up a database and populate it with empty collections with the same names as those in the source.

      Data Transfer does not automatically shard the migrated collections. Sharding large collections may take a long time and slow down the transfer.

   2. Enable target database sharding:

      sh.enableSharding("<target_database_name>")
      

   3. Shard every collection based on its namespace:

      sh.shardCollection("<target_database_name>.<collection_name>", { <field_name>: <1|"hashed">, ... });
      

      For the shardCollection() function description, see the MongoDB documentation.

   4. To make sure that sharding is set up and enabled, get a list of available shards:

      sh.status()
      

   5. If sharding uses any key other than _id (default), assign the clusterManager system role to the user Data Transfer will use for connection to the target cluster:

      use admin;
      db.grantRolesToUser("<username>", ["clusterManager"]);
      

   6. When creating a target endpoint, select DISABLED or TRUNCATE as your cleanup policy.

      Selecting the DROP policy will result in the service deleting all the data from the target database, including sharded collections, and replacing them with new unsharded ones when a transfer is activated.

   Learn more about sharding in the MongoDB documentation.

• Endpoint type: mongo-target.

• --cluster-id: ID of the cluster you need to connect to.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: mongo_target.

• connection.connection_options.mdb_cluster_id: ID of cluster to connect to.

• subnet_id: ID of the subnet the cluster is in. The transfer will use this subnet to access the cluster. If the ID is not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the cluster. For more information, see Networking in Yandex Data Transfer.

  Security groups and the subnet_id subnet, if the latter is specified, must belong to the same network as the cluster.

  Note

  In Terraform, it is not required to specify a network for security groups.

• auth_source: Name of the cluster database.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Here is the configuration file example:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    mongo_target {
      security_groups = ["<list_of_security_group_IDs>"]
      subnet_id       = "<subnet_ID>"
      connection {
        connection_options {
          mdb_cluster_id = "<cluster_ID>"
          auth_source    = "<DB_name>"
          user           = "<username>"
          password {
            raw = "<user_password>"
          }
        }
      }
      <additional_endpoint_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• mdbClusterId: ID of the cluster you need to connect to.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• Endpoint type: mongo-target.

• --host: IP address or FQDN of the master host you want to connect to.

• --port: Number of the port that Data Transfer will use for the connection.

• --ca-certificate: CA certificate if the data to transfer must be encrypted to comply with PCI DSS requirements.

  Warning

  If no certificate is added, the transfer may fail with an error.

• --subnet-id: ID of the subnet the host is in. The transfer will use that subnet to access the host.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: mongo_target.

• on_premise.port: Port number that Data Transfer will use for connections.

• connection.connection_options.on_premise.tls_mode.enabled.ca_certificate: CA certificate if the data to transfer must be encrypted, e.g., to comply with the PCI DSS requirements.

  Warning

  If no certificate is added, the transfer may fail with an error.

• security_groups: Security groups for network traffic.

  Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the VM with the database. For more information, see Networking in Yandex Data Transfer.

  Security groups must belong to the same network as the subnet_id subnet, if the latter is specified.

  Note

  In Terraform, it is not required to specify a network for security groups.

• subnet_id: ID of the subnet the cluster is in. The transfer will use this subnet to access the cluster. If the ID is not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• connection.connection_options.on_premise.replica_set: Specify the name of the replica set.

• connection.connection_options.on_premise.hosts: Specify the IP addresses or FQDN of the hosts to connect to.

• auth_source: Name of the cluster database.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Here is the configuration file example:

resource "yandex_datatransfer_endpoint" "<endpoint_name_in_Terraform>" {
  name = "<endpoint_name>"
  settings {
    mongo_target {
      security_groups = ["<list_of_security_group_IDs>"]
      subnet_id       = "<subnet_ID>"
      connection {
        connection_options {
          on_premise {
            hosts       = [ "list of replica set hosts" ]
            port        = "<port_for_connection>"
            replica_set = "<replica_set_name>"
            tls_mode {
              enabled {
                ca_certificate = "<certificate_in_PEM_format>"
              }
            }
          }
          auth_source = "<DB_name>"
          user        = "<username>"
          password {
            raw = "<user_password>"
          }
        }
      }
      <additional_endpoint_settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• onPremise: Database connection parameters:

  • hosts — IP address or FQDN of the master host to connect to.

  • port: The number of the port that Data Transfer will use for the connection.

  • tlsMode: Parameters for encrypting the data to transfer, if required, e.g., for compliance with the PCI DSS requirements.

    • disabled: Disabled.
    • enabled: Enabled.

      • caCertificate: CA certificate.

        Warning

        If no certificate is added, the transfer may fail with an error.

  • subnetId: ID of the subnet the host is in. The transfer will use that subnet to access the host.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Networking in Yandex Data Transfer.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

--target-database: Specify the database name if you want to create collections in a database that is different from the source database.

• database: Specify the database name if you want to create collections in a database that is different from the source database.

• cleanup_policy: Select a way to clean up data in the target database before the transfer:

  • DISABLED: Do not clean up (default).

    Select this option if only replication without copying data is performed.

  • DROP: Completely delete the collections included in the transfer.

    Use this option so that the latest collection version is always transferred to the target database from the source whenever the transfer is activated.

  • TRUNCATE: Delete only the data from the collections included in the transfer but keep the collections.

    Use this option if the structure of collections in the target database differs from the one that would have been transferred from the source during the transfer.

• database: Specify the database name if you want to create collections in a database that is different from the source database.

• cleanupPolicy: Select a way to clean up data in the target database before the transfer:

  • DISABLED: Do not clean up (default).

    Select this option if only replication without copying data is performed.

  • DROP: Completely delete the collections included in the transfer.

    Use this option so that the latest collection version is always transferred to the target database from the source whenever the transfer is activated.

  • TRUNCATE: Delete only the data from the collections included in the transfer but keep the collections.

    Use this option if the structure of collections in the target database differs from the one that would have been transferred from the source during the transfer.