Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Migrating a database from a third-party MySQL® cluster to a Yandex Managed Service for MySQL® cluster

Written by
Updated at July 30, 2024

There are two ways to migrate data from a third-party source cluster to a Managed Service for MySQL® target cluster:

  • Transferring data using Yandex Data Transfer.

    This method is easy to configure, does not require the creation of an intermediate VM, and allows you to transfer the entire database without interrupting user service. To use it, allow connections to the source cluster from the internet.

    For more information, see What tasks Yandex Data Transfer is used for.

  • Transferring data by creating and restoring a logical dump.

    A logical dump is a file with a set of commands running which one by one you can restore the state of a database. To ensure that a logical dump is complete, before creating it, switch the source cluster to read-only mode.

    Use this method only if, for some reason, it is not possible to migrate data using Data Transfer.

Transferring data using Data Transfer

To transfer a database from MySQL® to Managed Service for MySQL®:

  1. Start data transfer.
  2. Finish data transfer.

If you no longer need the resources you created, delete them.

Start data transfer

  1. Prepare the source cluster.

  2. Prepare the infrastructure and start the data transfer:

    1. Create a Managed Service for MySQL® target cluster in any suitable configuration. Note that:

      • The MySQL® version must be the same or higher than the version in the source cluster.

        Transferring data with MySQL® major version upgrade is possible but not guaranteed. For more information, see the MySQL® documentation.

        You cannot perform migration while downgrading MySQL® version.

      • SQL mode must be the same as in the source cluster.

    2. Prepare the target cluster.

    3. Create a source endpoint:

      • Database type: MySQL.

      • Endpoint parameters → Connection settings: Custom installation.

        Specify the parameters for connecting to the source cluster.

    4. Create a target endpoint:

      • Database type: MySQL.

      • Endpoint parameters → Connection settings: Managed Service for MySQL cluster.

        Select a target cluster from the list and specify its connection settings.

    5. Create a transfer of the Snapshot and increment type that will use the created endpoints.

    6. Activate your transfer.

      Warning

      Abstain from making any changes to the data schema in the source and target clusters when the data transfer is running. For more information, see Working with databases during transfer.

    1. Prepare the source cluster.

    2. If you do not have Terraform yet, install it.

    3. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

    4. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

    5. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

    6. Download the data-transfer-mysql-mmy.tf configuration file to the same working directory.

      This file describes:

      • Network.
      • Subnet.
      • Security group and the rule required to connect to a cluster.
      • Managed Service for MySQL® target cluster.
      • Source endpoint.
      • Target endpoint.
      • Transfer.

    7. Specify in the data-transfer-mysql-mmy.tf file:

      • Source endpoint parameters.

      • Target cluster parameters also used as target endpoint parameters:

        • target_mysql_version: MySQL® version must be the same or higher than the version in the source cluster.
        • target_sql_mode: SQL mode must be the same as in the source cluster.
        • target_db_name: Database name.
        • target_user and target_password: Database owner username and password.

    8. Make sure the Terraform configuration files are correct using this command:

      terraform validate

      If there are any errors in the configuration files, Terraform will point them out.

    9. Create the required infrastructure:

      1. Run the command to view planned changes:

        terraform plan

        If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

        1. Run the command:

          terraform apply

        2. Confirm the update of resources.

        3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

      Once created, your transfer will be activated automatically.

Finish data transfer

  1. Wait for the transfer status to change to Replicating.

  2. Switch the source cluster to read-only and transfer the load to the target cluster.

  3. On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data was copied are transferred to the target cluster.

  4. Deactivate the transfer and wait for its status to change to Stopped.

    For more information about transfer statuses, see Transfer lifecycle.

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the data-transfer-mysql-mmy.tf configuration file.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the resources described in the configuration file data-transfer-mysql-mmy.tf will be deleted.

For a real example of MySQL® database migration using Data Transfer, see Syncing MySQL data using Yandex Data Transfer.

Transferring data by creating and restoring a logical dump

To move data to a Managed Service for MySQL® cluster, create a logical dump of the desired database and restore it to the target cluster. You can do this using two methods:

  • Using the mydumper and the myloader utilities. A database dump is created as a collection of files in a separate folder.
  • Using the mysqldump and the mysql utilities. A database dump is created as a single file.

Migration stages:

  1. Create a dump of the database you want to migrate.

  2. (Optional) Upload a dump to an intermediate virtual machine in Yandex Cloud.

    Transfer your data to an intermediate VM in Yandex Compute Cloud if:

    • Your Managed Service for MySQL® cluster is not accessible from the internet.
    • Your hardware or connection to the cluster in Yandex Cloud is not very reliable.

    The larger the amount of data to be migrated and the required migration speed, the higher the virtual machine requirements: number of processor cores, RAM, and disk space.

  3. Restore data from the dump.

If you no longer need the resources you created, delete them.

Getting started

Create the required resources:

  1. Create a Managed Service for MySQL® target cluster with any suitable configuration. Note that:

    • The MySQL® version must be the same or higher than the version in the source cluster.

      Transferring data with MySQL® major version upgrade is possible but not guaranteed. For more information, see the MySQL® documentation.

      You cannot perform migration while downgrading MySQL® version.

    • SQL mode must be the same as in the source cluster.

  2. (Optional step) Create a VM based on Ubuntu 20.04 LTS with the following parameters:

    • Disks and file storages → Size: Sufficient to store both archived and unarchived dumps.

      The recommended size is two or three times the total dump and dump archive size.

    • Network settings:

      • Subnet: Select a subnet on the cloud network hosting the target cluster.
      • Public IP: Select Auto or one address from a list of reserved IPs.

  3. If you use security groups for the intermediate VM and the Managed Service for MySQL® cluster, configure them.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the data-migration-mysql-mmy.tf configuration file to the same working directory.

    This file describes:

    • Network.
    • Subnet.
    • Security group and the rule required to connect to a cluster.
    • Managed Service for MySQL® cluster with public internet access.
    • (Optional) Virtual machine with public internet access.

  6. Specify in the file data-migration-mysql-mmy.tf:

    • Target cluster parameters:

      • target_mysql_version: MySQL® version must be the same or higher than the version in the source cluster.
      • target_sql_mode: SQL mode must be the same as in the source cluster.
      • target_db_name: Database name.
      • target_user and target_password: Database owner username and password.

    • (Optional) Virtual machine parameters:

      • vm_image_id: ID of the public image with Ubuntu and no GPU, e.g., for Ubuntu 20.04 LTS.
      • vm_username and vm_public_key: Username and absolute path to a public key that will be used to access the virtual machine. By default, the specified username is ignored in the Ubuntu 20.04 LTS image. A user with the ubuntu username is created instead. Use it to connect to the instance.

  7. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Creating a dump

  1. Switch the database to read-only mode to avoid losing data that can appear while creating the dump.

  2. Install the mysqldump utility in the source cluster as follows (Ubuntu):

    sudo apt update && sudo apt install mysql-client --yes

  3. Create a database dump:

    mysqldump \
    --host=<FQDN_or_IP> \
    --user=<username> \
    --password \
    --port=<port> \
    --set-gtid-purged=OFF \
    --quick \
    --single-transaction \
    <DB_name> > ~/db_dump.sql

    Where --host is the FQDN or IP of the source cluster's master host.

    If required, provide additional parameters in the create dump command:

    • --events, if there are recurring events in your database.
    • --routines, if your database stores procedures and functions.

    For InnoDB tables, use the --single-transaction option to guarantee data integrity.

  4. In the dump file, change the table engine names to InnoDB:

    sed -i -e 's/MyISAM/InnoDB/g' -e 's/MEMORY/InnoDB/g' db_dump.sql

  5. Archive the dump:

    tar -cvzf db_dump.tar.gz ~/db_dump.sql

  1. Switch the database to read-only mode to avoid losing data that can appear while creating the dump.

  2. Create a directory for the dump files:

    mkdir db_dump

  3. Install the mydumper utility in the source cluster as follows (Ubuntu):

    sudo apt update && sudo apt install mydumper --yes

  4. Create a database dump:

    mydumper \
    --triggers \
    --events \
    --routines \
    --outputdir=db_dump \
    --rows=10000000 \
    --threads=8 \
    --compress \
    --database=<DB_name> \
    --user=<username> \
    --ask-password \
    --host=<FQDN_or_IP>

    Where:

    • --triggers: Trigger dump.
    • --events: Event dump.
    • --routines: Stored procedure and function dump.
    • --outputdir: Dump file directory.
    • --rows: Number of rows in table fragments. The smaller the value, the more files in a dump.
    • --threads: Number of threads used. The recommended value is equal to half the server's free cores.
    • --compress: Output file compression.
    • --host: FQDN or IP of the source cluster's master host.

  5. In the dump file, change the table engine names to InnoDB:

    sed -i -e 's/MyISAM/InnoDB/g' -e 's/MEMORY/InnoDB/g' `find /db_dump -name '*-schema.sql'`

  6. Archive the dump:

    tar -cvzf db_dump.tar.gz ~/db_dump

(Optional) Uploading a dump to a virtual machine in Yandex Cloud

  1. Connect to an intermediate virtual machine over SSH.

  2. Copy the archive containing the database dump to the intermediate virtual machine using the scp utility, for instance:

    scp ~/db_dump.tar.gz <VM_username>@<VM_public_IP>:~/db_dump.tar.gz

  3. Extract the dump from the archive:

    tar -xzf ~/db_dump.tar.gz

Restoring data

Alert

For Managed Service for MySQL® clusters, AUTOCOMMIT is enabled by default. Do not disable AUTOCOMMIT during the client session when restoring the database from the dump, otherwise the host storage may overflow and the cluster may not function properly.

This method is suitable if you created your dump with the mysqldump utility.

  1. Install the mysql utility on the host the dump is being restored from as follows (Ubuntu):

    sudo apt update && sudo apt install mysql-client --yes

  2. Start the database restore from the dump:

    • If you restore a dump from the VM in Yandex Cloud:

      mysql \
    --host=c-<target_cluster_ID>.rw.mdb.yandexcloud.net \
    --user=<username> \
    --port=3306 \
    <DB_name> < ~/db_dump.sql

    • If you are restoring your dump from a host connecting to Yandex Cloud from the internet, obtain an SSL certificate and provide the --ssl-ca and the --ssl-mode parameters in the restore command:

      mysql \
    --host=c-<target_cluster_ID>.rw.mdb.yandexcloud.net \
    --user=<username> \
    --port=3306 \
    --ssl-ca=~/.mysql/root.crt \
    --ssl-mode=VERIFY_IDENTITY \
    <DB_name> < ~/db_dump.sql

This method is suitable if you created your dump using the mydumper utility and are using an intermediate virtual machine for the restore.

  1. Install the myloader utility to the host that you are using to restore the dump as follows (Ubuntu):

    sudo apt update && sudo apt install mydumper --yes

  2. Start the database restore from the dump:

    myloader \
    --host=c-<target_cluster_ID>.rw.mdb.yandexcloud.net \
    --directory=db_dump/ \
    --overwrite-tables \
    --threads=8 \
    --compress-protocol \
    --user=<username> \
    --ask-password

You can get the cluster ID with a list of clusters in the folder.

Deleting created resources

Delete the resources you no longer need to avoid paying for them:

To delete the infrastructure created with Terraform:

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the data-migration-mysql-mmy.tf configuration file.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the resources described in the configuration file data-migration-mysql-mmy.tf will be deleted.

Previous
MongoDB performance analysis and tuning
Next
Managed Service for MySQL® performance analysis and tuning