Migrating databases from Yandex Managed Service for MySQL® to a third-party MySQL® cluster

To migrate a database deployed in a Managed Service for MySQL® cluster to a third-party MySQL® cluster:

1. Transfer the data.
2. Disable data writes to the source database.
3. Switch all traffic to the third-party cluster.

You can migrate your databases from one MySQL® version to another, e.g., from MySQL® 5.7 to MySQL® 8. Note that the third-party cluster must run a major MySQL® version not lower than that on your Managed Service for MySQL® cluster.

There are two ways to migrate data from a Managed Service for MySQL® source cluster to a third-party MySQL® target cluster:

• Transfer data using Yandex Data Transfer.

  This method enables you to migrate the entire database without downtime for users.

  Learn more in Problems addressed by Yandex Data Transfer.

• Transfer data using external replication.

  External replication enables you to migrate databases between MySQL® clusters using native DBMS tools.

  Use this approach only if, for some reason, transferring data with Yandex Data Transfer is not an option.

Getting startedGetting started

Set up the target cluster:

• Create a MySQL® database with any suitable configuration.
• Make sure you can connect to the target cluster hosts from the internet.

Additionally, to migrate data using external MySQL® replication:

• Check that all source cluster hosts are accessible via public IP addresses to make sure the target cluster can connect to the source cluster. To do this:
  • Add hosts with public IP addresses.
  • Delete hosts without public IP addresses.
• Install the Managed Service for MySQL® server SSL certificates on the target cluster hosts. These certificates are required to connect to a publicly accessible source cluster.
• If required, set up a firewall and configure security groups to enable connections from the target cluster to the source cluster and to each cluster individually, e.g., using mysql.
• Make sure the target cluster hosts can connect to the source cluster hosts.
• Make sure you can connect to both the source cluster and the target cluster using SSL.

Transferring data using Yandex Data TransferTransferring data using Yandex Data Transfer

Required paid resourcesRequired paid resources

• Managed Service for MySQL® cluster: computing resources allocated to hosts, size of storage and backups (see Managed Service for MySQL® pricing).
• Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• Each transfer: use of computing resources and number of transferred data rows (see Data Transfer pricing).

Migrate the databaseMigrate the database

1. Set up the source cluster database.

2. Set up the target cluster database.

3. Create endpoints and a transfer:

   Manually

   Terraform

   1. Create a source endpoint:

      • Database type: MySQL.

      • Connection settings: Managed Service for MySQL cluster.

        Specify the source cluster ID.

   2. Create a target endpoint:

      • Database type: MySQL.

      • Connection settings: Custom installation.

        Configure the target cluster connection settings.

   3. Create a transfer of the Snapshot and increment-type that will use the endpoints you created.

   4. Activate the transfer.

   1. If you do not have Terraform yet, install it.

   2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

   3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

   4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

   5. Download the data-transfer-mmy-mysql.tf file with the transfer and endpoint configuration file to the same working directory.

   6. Specify the following in the configuration file:

      • Source endpoint parameters.
      • Target endpoint parameters.

   7. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

   8. Create the required infrastructure:

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

      The transfer will activate automatically upon creation.

   Warning

   Avoid changing the data schema in either the source or target cluster while the transfer is in progress. For more information, see Working with databases during transfer.

4. Wait for the transfer status to change to Replicating.

5. Switch the source cluster to "read-only" mode and move the workload to the target cluster.

6. On the transfer monitoring page, wait until the Maximum data transfer delay value drops to zero. This indicates that the target cluster now contains all changes made in the source cluster after the data copy completed.

7. Deactivate the transfer and wait for its status to change to Stopped.

   For more information about transfer statuses, see Transfer lifecycle.

8. Delete the endpoints and transfer you created:

   Manually

   Terraform

   1. Ensure the transfer is stopped, then delete it.
   2. Delete the source and target endpoints.

   1. In the terminal window, go to the directory containing the infrastructure plan.

      Warning

      Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

   2. Delete resources:

      1. Run this command:

         terraform destroy
         

      2. Confirm deleting the resources and wait for the operation to complete.

      All the resources described in the Terraform manifests will be deleted.

Transferring data using external replicationTransferring data using external replication

1. Transfer the logical database dump.
2. Configure the user in the source cluster to manage replication.
3. Start replication in the target cluster.
4. Monitor the migration process until it is complete.
5. Complete your migration.

Required paid resourcesRequired paid resources

• Managed Service for MySQL® cluster: computing resources allocated to hosts, size of storage and backups (see Managed Service for MySQL® pricing).
• Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Transfer the logical database dumpTransfer the logical database dump

A logical dump is a file with a sequence of commands that can restore the database state. You can create it with mysqldump. Before creating a logical dump, pause database writes to ensure its completeness.

1. Get the current binary log position to ensure consistency when restoring the logical dump:

   SHOW MASTER STATUS;
   

   +-------------------------+----------+--------------+------------------+-----------------------------+
   | File                    | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set           |
   +-------------------------+----------+--------------+------------------+-----------------------------+
   | mysql-bin-log-...000224 |  2058567 |              |                  | d827098b-...00b86:1-1575866 |
   +-------------------------+----------+--------------+------------------+-----------------------------+
   1 row in set (0.00 sec)
   

   Save the File and Position values, as you will need them to start replication.

2. Create a dump of the source cluster database:

   mysqldump \
       --databases=<DB_name> \
       --routines \
       --host=<master_host_FQDN> \
       --ssl-ca=<path_to_SSL_certificate> \
       --user=<DB_owner_username> > <dump_file>
   

   Tip

   Use a special FQDN that always points to the current master host of Managed Service for MySQL® clusters.

3. Restore the database from the dump in the target cluster:

   Connecting with SSL

   Connecting without SSL

   mysql --host=<master_host_FQDN> \
         --user=<username> \
         --password \
         --port=3306 \
         --ssl-ca=<path_to_SSL_certificate> \
         --ssl-mode=VERIFY_IDENTITY \
         --line-numbers \
         <DB_name> < <dump_file>
   

   mysql --host=<master_host_FQDN> \
         --user=<username> \
         --password \
         --port=3306 \
         --line-numbers \
         <DB_name> < <dump_file>
   

4. Create a user in the target cluster with full permissions for the database being migrated:

   CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
   GRANT ALL PRIVILEGES ON <DB_name>.* TO '<username>'@'%';
   

Configure the user in the source cluster to manage replicationConfigure the user in the source cluster to manage replication

MySQL® uses the master-replica model for replication: the target cluster copies the source cluster's binary log changes to its own relay log. The replica host applies these changes to its data.

To capture binary log changes and manage replication in the source cluster:

1. Create a user.
2. Assign it the ALL_PRIVILEGES role for the source cluster database.
3. Grant the user the REPLICATION CLIENT and REPLICATION SLAVE global privileges as well.

The target cluster will connect to the source cluster as this user.

Start replication in the target clusterStart replication in the target cluster

1. Edit the /etc/mysql/my.cnf file containing the target cluster configuration to start replication:

   [mysqld]
   ...
   log_bin = mysql-bin
   server_id = 2
   relay-log = /var/lib/mysql/mysql-relay-bin
   relay-log-index = /var/lib/mysql/mysql-relay-bin.index
   
   gtid-mode = on
   enforce-gtid-consistency = on
   

   Where:

   • log_bin: Binary log name in the target cluster.
   • server_id: Target cluster ID. The default value is 1. However, to run replication, make sure the values of the source and target cluster IDs are different.
   • relay-log: Path to the relay log.
   • relay-log-index: Path to the relay log index.

   Also, enable gtid-mode and enforce-gtid-consistency for replication. In Managed Service for MySQL® clusters, they are always enabled.

2. Restart mysql:

   sudo systemctl restart mysql
   

3. Connect to the target cluster as the user with full access to the database you want to migrate.

4. Enable replication for this database and disable it for system databases, since they replicate by default:

   CHANGE REPLICATION FILTER
       REPLICATE_DO_DB=(
           <target_cluster_DB_name>
       ),
       REPLICATE_IGNORE_DB=(
           sys,
           mysql,
           performance_schema,
           information_schema
       );
   

5. To assign a master for the target cluster, specify the parameters of the master host in the source cluster:

   Tip

   Use a special FQDN that always points to the current master host of Managed Service for MySQL® clusters.

   CHANGE MASTER TO
         MASTER_HOST = '<master_host_FQDN>',
         MASTER_USER = '<user_for_replication>',
         MASTER_PASSWORD = '<user_password>',
         MASTER_LOG_FILE = '<File_value_from_binary_log_position_query>',
         MASTER_LOG_POS = <Position_value_from_binary_log_position_query>,
         MASTER_SSL_CA = '<path_to_SSL_certificate>',
         MASTER_SSL_VERIFY_SERVER_CERT = 0,
         MASTER_SSL = 1;
   

6. Start applying the relay log:

   START SLAVE;
   

   This initiates data migration from the source cluster database to the target cluster database.

7. After successfully starting the replication, run this command once:

   STOP SLAVE;
   CHANGE MASTER TO MASTER_AUTO_POSITION = 1;
   START SLAVE;
   

   This is to ensure the replication will be reconfigured to use the new master host if the master host in the source cluster changes. For more information about configurations, see this MySQL® article.

Monitor your data migrationMonitor your data migration

Run the command that returns the replication status:

SHOW SLAVE STATUS\G

*************************** 1. row ***************************
           Slave_IO_State: Waiting for master to send event
              Master_Host: rc1a-hxk9audl********.mdb.yandexcloud.net
              Master_User: replica-my
              Master_Port: 3306
            Connect_Retry: 60
          Master_Log_File: mysql-bin-log-rc1a-hxk9audl********-mdb-yandexcloud-net.000225
      Read_Master_Log_Pos: 1702815
           Relay_Log_File: 6b6d647a39b6-relay-bin.000084
            Relay_Log_Pos: 409
            ...

The following fields contain info on the replication status:

• Slave_IO_State and Slave_SQL_Running_State: I/O state of the binary log and relay log streams. If replication is successful, both streams are active.
• Read_Master_Log_Pos: Last position read from the master host log.
• Seconds_Behind_Master: Replica's lag behind the master, in seconds.
• Last_IO_Error and Last_SQL_Error: Replication errors.

For more information about the replication status, see this MySQL® article.

Complete your migrationComplete your migration

1. Stop all writes to the source cluster and make sure the application does not write data to its database. You can do this by updating the MAX_UPDATES_PER_HOUR user-defined setting of the source cluster to 1.

2. Wait for the Seconds_Behind_Master value to decrease to zero. This means that all changes that occurred in the source cluster after creating the logical dump have been transferred to the target cluster.

3. Stop replication in the target cluster:

   STOP SLAVE;
   

4. Switch the load to the target cluster.

5. Delete the user managing replication in the source cluster.

6. Delete the user with full access to the migrated database in the target cluster if this user is no longer needed.