Migrating a database from Managed Service for MySQL® to a third-party MySQL® cluster
Note
To learn about migrating data from a third-party MySQL® cluster, see this tutorial.
To migrate a database deployed in a Managed Service for MySQL® cluster to a third-party MySQL® cluster:
- Transfer the data.
- Disable data writes to the source database.
- Switch all traffic to the third-party cluster.
You can migrate your databases from one MySQL® version to another, e.g., from MySQL® 5.7 to MySQL® 8. Note that the third-party cluster must run a major MySQL® version not lower than that on your Managed Service for MySQL® cluster.
There are two ways to migrate data from a Managed Service for MySQL® source cluster to a third-party MySQL® target cluster:
-
Transfer data using Yandex Data Transfer.
This method enables you to migrate the entire database without downtime for users.
Learn more in Problems addressed by Yandex Data Transfer.
-
Transfer data using external replication.
External replication
enables you to migrate databases between MySQL® clusters using native DBMS tools.Use this approach only if, for some reason, transferring data with Yandex Data Transfer is not an option.
Getting started
Set up the target cluster:
- Create a MySQL® database
with any suitable configuration. - Make sure you can connect to the target cluster hosts from the internet.
Additionally, to migrate data using external MySQL® replication:
- Check that all source cluster hosts are accessible via public IP addresses to make sure the target cluster can connect to the source cluster. To do this:
- Add hosts with public IP addresses.
- Delete hosts without public IP addresses.
- Install the Managed Service for MySQL® server SSL certificates on the target cluster hosts. These certificates are required to connect to a publicly accessible source cluster.
- If required, set up a firewall and configure security groups to enable connections from the target cluster to the source cluster and to each cluster individually, e.g., using mysql
. - Make sure the target cluster hosts can connect to the source cluster hosts.
- Make sure you can connect to both the source cluster and the target cluster using SSL.
Transferring data using Yandex Data Transfer
Required paid resources
- Managed Service for MySQL® cluster: computing resources allocated to hosts, size of storage and backups (see Managed Service for MySQL® pricing).
- Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
- Each transfer: use of computing resources and number of transferred data rows (see Data Transfer pricing).
Migrate the database
-
Create endpoints and a transfer:
ManuallyTerraform-
-
Database type:
MySQL. -
Connection settings:
Managed Service for MySQL cluster.Specify the source cluster ID.
-
-
-
Database type:
MySQL. -
Connection settings:
Custom installation.Configure the target cluster connection settings.
-
-
Create a transfer of the Snapshot and increment-type that will use the endpoints you created.
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
Download the data-transfer-mmy-mysql.tf
file with the transfer and endpoint configuration file to the same working directory. -
Specify the following in the configuration file:
-
Make sure the Terraform configuration files are correct using this command:
terraform validateTerraform will show any errors found in your configuration files.
-
Create the required infrastructure:
-
Run this command to view the planned changes:
terraform planIf you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
-
If everything looks correct, apply the changes:
-
Run this command:
terraform apply -
Confirm updating the resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
.The transfer will activate automatically upon creation.
-
Warning
Avoid changing the data schema in either the source or target cluster while the transfer is in progress. For more information, see Working with databases during transfer.
-
-
Wait for the transfer status to change to Replicating.
-
Switch the source cluster to "read-only" mode and move the workload to the target cluster.
-
On the transfer monitoring page, wait until the Maximum data transfer delay value drops to zero. This indicates that the target cluster now contains all changes made in the source cluster after the data copy completed.
-
Deactivate the transfer and wait for its status to change to Stopped.
For more information about transfer statuses, see Transfer lifecycle.
-
Delete the endpoints and transfer you created:
ManuallyTerraform-
In the terminal window, go to the directory containing the infrastructure plan.
Warning
Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
-
Delete resources:
-
Run this command:
terraform destroy -
Confirm deleting the resources and wait for the operation to complete.
All the resources described in the Terraform manifests will be deleted.
-
-
Transferring data using external replication
- Transfer the logical database dump.
- Configure the user in the source cluster to manage replication.
- Start replication in the target cluster.
- Monitor the migration process until it is complete.
- Complete your migration.
Required paid resources
- Managed Service for MySQL® cluster: computing resources allocated to hosts, size of storage and backups (see Managed Service for MySQL® pricing).
- Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
Transfer the logical database dump
A logical dump is a file with a sequence of commands that can restore the database state. You can create it with mysqldump
Warning
If your database contains user-defined stored procedures, grant the database owner the SHOW ROUTINE administrative privilege to perform a logical dump.
-
Get the current binary log position to ensure consistency when restoring the logical dump:
SHOW MASTER STATUS;+-------------------------+----------+--------------+------------------+-----------------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +-------------------------+----------+--------------+------------------+-----------------------------+ | mysql-bin-log-...000224 | 2058567 | | | d827098b-...00b86:1-1575866 | +-------------------------+----------+--------------+------------------+-----------------------------+ 1 row in set (0.00 sec)Save the
FileandPositionvalues, as you will need them to start replication. -
Create a dump of the source cluster database:
mysqldump \ --databases=<DB_name> \ --routines \ --host=<master_host_FQDN> \ --ssl-ca=<path_to_SSL_certificate> \ --user=<DB_owner_username> > <dump_file>Tip
Use a special FQDN that always points to the current master host of Managed Service for MySQL® clusters.
-
Restore the database from the dump in the target cluster:
Connecting with SSLConnecting without SSLmysql --host=<master_host_FQDN> \ --user=<username> \ --password \ --port=3306 \ --ssl-ca=<path_to_SSL_certificate> \ --ssl-mode=VERIFY_IDENTITY \ --line-numbers \ <DB_name> < <dump_file>mysql --host=<master_host_FQDN> \ --user=<username> \ --password \ --port=3306 \ --line-numbers \ <DB_name> < <dump_file> -
Create a user in the target cluster with full permissions for the database being migrated:
CREATE USER '<username>'@'%' IDENTIFIED BY '<password>'; GRANT ALL PRIVILEGES ON <DB_name>.* TO '<username>'@'%';
Configure the user in the source cluster to manage replication
MySQL® uses the master-replica
model for replication: the target cluster copies the source cluster's binary log changes to its own relay log. The replica host applies these changes to its data.
To capture binary log changes and manage replication in the source cluster:
- Create a user.
- Assign it the
ALL_PRIVILEGESrole for the source cluster database. - Grant the user the
REPLICATION CLIENTandREPLICATION SLAVEglobal privileges as well.
The target cluster will connect to the source cluster as this user.
Start replication in the target cluster
-
Edit the
/etc/mysql/my.cnffile containing the target cluster configuration to start replication:[mysqld] ... log_bin = mysql-bin server_id = 2 relay-log = /var/lib/mysql/mysql-relay-bin relay-log-index = /var/lib/mysql/mysql-relay-bin.index gtid-mode = on enforce-gtid-consistency = onWhere:
log_bin: Binary log name in the target cluster.server_id: Target cluster ID. The default value is1. However, to run replication, make sure the values of the source and target cluster IDs are different.relay-log: Path to the relay log.relay-log-index: Path to the relay log index.
Also, enable
gtid-modeandenforce-gtid-consistencyfor replication. In Managed Service for MySQL® clusters, they are always enabled. -
Restart
mysql:sudo systemctl restart mysql -
Connect to the target cluster as the user with full access to the database you want to migrate.
-
Enable replication for this database and disable it for system databases, since they replicate by default:
CHANGE REPLICATION FILTER REPLICATE_DO_DB=( <target_cluster_DB_name> ), REPLICATE_IGNORE_DB=( sys, mysql, performance_schema, information_schema ); -
To assign a master for the target cluster, specify the parameters of the master host in the source cluster:
Tip
Use a special FQDN that always points to the current master host of Managed Service for MySQL® clusters.
CHANGE MASTER TO MASTER_HOST = '<master_host_FQDN>', MASTER_USER = '<user_for_replication>', MASTER_PASSWORD = '<user_password>', MASTER_LOG_FILE = '<File_value_from_binary_log_position_query>', MASTER_LOG_POS = <Position_value_from_binary_log_position_query>, MASTER_SSL_CA = '<path_to_SSL_certificate>', MASTER_SSL_VERIFY_SERVER_CERT = 0, MASTER_SSL = 1; -
Start applying the relay log:
START SLAVE;This initiates data migration from the source cluster database to the target cluster database.
-
After successfully starting the replication, run this command once:
STOP SLAVE; CHANGE MASTER TO MASTER_AUTO_POSITION = 1; START SLAVE;This is to ensure the replication will be reconfigured to use the new master host if the master host in the source cluster changes. For more information about configurations, see this MySQL® article
.
Monitor your data migration
Run the command that returns the replication status:
SHOW SLAVE STATUS\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: rc1a-hxk9audl********.mdb.yandexcloud.net
Master_User: replica-my
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin-log-rc1a-hxk9audl********-mdb-yandexcloud-net.000225
Read_Master_Log_Pos: 1702815
Relay_Log_File: 6b6d647a39b6-relay-bin.000084
Relay_Log_Pos: 409
...
The following fields contain info on the replication status:
Slave_IO_StateandSlave_SQL_Running_State: I/O state of the binary log and relay log streams. If replication is successful, both streams are active.Read_Master_Log_Pos: Last position read from the master host log.Seconds_Behind_Master: Replica's lag behind the master, in seconds.Last_IO_ErrorandLast_SQL_Error: Replication errors.
For more information about the replication status, see this MySQL® article
Complete your migration
-
Stop all writes to the source cluster and make sure the application does not write data to its database. You can do this by updating the
MAX_UPDATES_PER_HOURuser-defined setting of the source cluster to1. -
Wait for the
Seconds_Behind_Mastervalue to decrease to zero. This means that all changes that occurred in the source cluster after creating the logical dump have been transferred to the target cluster. -
Stop replication in the target cluster:
STOP SLAVE; -
Switch the load to the target cluster.
-
Delete the user managing replication in the source cluster.
-
Delete the user with full access to the migrated database in the target cluster if this user is no longer needed.