Contact UsGet started

Migrating data from a third-party MySQL® cluster

Written by
Updated at December 11, 2024

Note

Data migration to a third-party MySQL® cluster is described in Migrating a database from Managed Service for MySQL® to a third-party MySQL® cluster.

There are two ways to migrate data from a third-party source cluster to a Managed Service for MySQL® target cluster:

Transferring data using Data Transfer

To transfer a database from MySQL® to Managed Service for MySQL®:

  1. Start data transfer.
  2. Finish data transfer.

If you no longer need the resources you created, delete them.

Start data transfer

  1. Prepare the source cluster.

  2. Prepare the infrastructure and start the data transfer:

    1. Create a Managed Service for MySQL® target cluster in any suitable configuration. Note that:

      • The MySQL® version must be the same or higher than the version in the source cluster.

        Transferring data with MySQL® major version upgrade is possible but not guaranteed. For more information, see the MySQL® documentation.

        You cannot perform migration while downgrading MySQL® version.

      • SQL mode must be the same as in the source cluster.

    2. Prepare the target cluster.

    3. Create a source endpoint:

      • Database type: MySQL.

      • Endpoint parameters → Connection settings: Custom installation.

        Specify the parameters for connecting to the source cluster.

    4. Create a target endpoint:

      • Database type: MySQL.

      • Endpoint parameters → Connection settings: Managed Service for MySQL cluster.

        Select a target cluster from the list and specify its connection settings.

    5. Create a transfer of the Snapshot and increment type that will use the created endpoints.

    6. Activate your transfer.

      Warning

      Abstain from making any changes to the data schema in the source and target clusters when the data transfer is running. For more information, see Working with databases during transfer.

    1. Prepare the source cluster.

    2. If you do not have Terraform yet, install it.

    3. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

    4. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

    5. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

    6. Download the data-transfer-mysql-mmy.tf configuration file to the same working directory.

      This file describes:

      • Network.
      • Subnet.
      • Security group and the rule required to connect to a cluster.
      • Managed Service for MySQL® target cluster.
      • Source endpoint.
      • Target endpoint.
      • Transfer.

    7. Specify in the data-transfer-mysql-mmy.tf file:

      • Source endpoint parameters.

      • Target cluster parameters also used as target endpoint parameters:

        • target_mysql_version: MySQL® version must be the same or higher than the version in the source cluster.
        • target_sql_mode: SQL mode must be the same as in the source cluster.
        • target_db_name: Database name.
        • target_user and target_password: Database owner username and password.

    8. Make sure the Terraform configuration files are correct using this command:

      terraform validate

      If there are any errors in the configuration files, Terraform will point them out.

    9. Create the required infrastructure:

      1. Run the command to view planned changes:

        terraform plan

        If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

        1. Run the command:

          terraform apply

        2. Confirm the update of resources.

        3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

      Once created, your transfer will be activated automatically.

Finish data transfer

  1. Wait for the transfer status to change to Replicating.

  2. Switch the source cluster to read-only and transfer the load to the target cluster.

  3. On the transfer monitoring page, wait for the Maximum data transfer delay metric to decrease to zero. This means that all changes that occurred in the source cluster after data was copied are transferred to the target cluster.

  4. Deactivate the transfer and wait for its status to change to Stopped.

    For more information about transfer statuses, see Transfer lifecycle.

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the data-transfer-mysql-mmy.tf configuration file.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the resources described in the configuration file data-transfer-mysql-mmy.tf will be deleted.

For a real example of MySQL® database migration using Data Transfer, see Syncing MySQL data using Yandex Data Transfer.

Transferring data by creating and restoring a logical dump

To move data to a Managed Service for MySQL® cluster, create a logical dump of the desired database and restore it to the target cluster. There are two ways to do this:

  • Use the mydumper and myloader utilities. A database dump is created as a collection of files in a separate folder.
  • Use mysqldump and mysql. A database dump is created as a single file.

Migration stages:

  1. Create a dump of the database you want to migrate.

  2. (Optional) Upload a dump to an intermediate virtual machine in Yandex Cloud.

    Transfer your data to an intermediate VM in Yandex Compute Cloud if:

    • Your Managed Service for MySQL® cluster is not accessible from the internet.
    • Your hardware or connection to the cluster in Yandex Cloud is not very reliable.

    The larger the amount of data to be migrated and the required migration speed, the higher the virtual machine requirements: number of processor cores, RAM, and disk space.

  3. Restore data from the dump.

If you no longer need the resources you created, delete them.

Getting started

Create the required resources:

  1. Create a Managed Service for MySQL® target cluster in any suitable configuration. In which case:

    • The MySQL® version must be the same or higher than the version in the source cluster.

      Transferring data with MySQL® major version upgrade is possible but not guaranteed. For more information, see the MySQL® documentation.

      You cannot perform migration while downgrading MySQL® version.

    • SQL mode must be the same as in the source cluster.

  2. (Optional step) Create a VM based on Ubuntu 20.04 LTS with the following parameters:

    • Disks and file storages → Size: Sufficient to store both archived and unarchived dumps.

      The recommended size is two or three times the total dump and dump archive size.

    • Network settings:

      • Subnet: Select a subnet on the cloud network hosting the target cluster.
      • Public IP: Select Auto or one address from a list of reserved IPs.

  3. If you use security groups for the intermediate VM and the Managed Service for MySQL® cluster, configure them.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the data-migration-mysql-mmy.tf configuration file to the same working directory.

    This file describes:

    • Network.
    • Subnet.
    • Security group and the rule required to connect to a cluster.
    • Managed Service for MySQL® cluster with public internet access.
    • (Optional) Virtual machine with public internet access.

  6. Specify the following in the data-migration-mysql-mmy.tf file:

    • Target cluster parameters:

      • target_mysql_version: MySQL® version. Must be the same or higher than in the source cluster.
      • target_sql_mode: SQL mode. It must be the same as in the source cluster.
      • target_db_name: Database name.
      • target_user and target_password: Name and user password of the database owner.

    • (Optional) Virtual machine parameters:

      • vm_image_id: ID of the public image with Ubuntu without GPU, e.g., for Ubuntu 20.04 LTS.
      • vm_username and vm_public_key: Username and absolute path to the public key, for access to the VM. By default, the specified username is ignored in the Ubuntu 20.04 LTS image. A user with the ubuntu username is created instead. Use it to connect to the instance.

  7. Check that the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Creating a dump

  1. Switch the database to read-only mode to avoid losing data that can appear while creating the dump.

  2. Install mysqldump in the source cluster, e.g., for Ubuntu:

    sudo apt update && sudo apt install mysql-client --yes

  3. Create a database dump:

    mysqldump \
    --host=<FQDN_or_IP_address> \
    --user=<username> \
    --password \
    --port=<port> \
    --set-gtid-purged=OFF \
    --quick \
    --single-transaction \
    <DB_name> > ~/db_dump.sql

    Where --host is the FQDN or IP address of the master host in the source cluster.

    If required, provide additional parameters in the create dump command:

    • --events, if there are recurring events in your database.
    • --routines, if your database stores procedures and functions.

    For InnoDB tables, use the --single-transaction option for data integrity.

  4. In the dump file, change the table engine names to InnoDB:

    sed -i -e 's/MyISAM/InnoDB/g' -e 's/MEMORY/InnoDB/g' db_dump.sql

  5. Archive the dump:

    tar -cvzf db_dump.tar.gz ~/db_dump.sql

  1. Switch the database to read-only mode to avoid losing data that can appear while creating the dump.

  2. Create a directory for the dump files:

    mkdir db_dump

  3. Install mydumper in the source cluster, e.g., for Ubuntu:

    sudo apt update && sudo apt install mydumper --yes

  4. Create a database dump:

    mydumper \
    --triggers \
    --events \
    --routines \
    --outputdir=db_dump \
    --rows=10000000 \
    --threads=8 \
    --compress \
    --database=<DB_name> \
    --user=<username> \
    --ask-password \
    --host=<FQDN_or_IP_address>

    Where:

    • --triggers: Trigger dump.
    • --events: Event dump.
    • --routines: Stored procedure and function dump.
    • --outputdir: Dump file directory.
    • --rows: Number of rows in table fragments. The smaller the value, the more files in a dump.
    • --threads: Number of threads in use. The recommended value is equal to half the server's free cores.
    • --compress: Output file compression.
    • Where --host is the FQDN or IP address of the master host in the source cluster.

  5. In the dump file, change the table engine names to InnoDB:

    sed -i -e 's/MyISAM/InnoDB/g' -e 's/MEMORY/InnoDB/g' `find /db_dump -name '*-schema.sql'`

  6. Archive the dump:

    tar -cvzf db_dump.tar.gz ~/db_dump

(Optional) Uploading a dump to a virtual machine in Yandex Cloud

  1. Connect to an intermediate virtual machine over SSH.

  2. Copy the archive containing the database dump to the intermediate virtual machine, e.g., using scp:

    scp ~/db_dump.tar.gz <VM_user_name>@<VM_public_IP_address>:~/db_dump.tar.gz

  3. Extract the dump from the archive:

    tar -xzf ~/db_dump.tar.gz

Restoring data

Alert

For Managed Service for MySQL® clusters, AUTOCOMMIT is enabled by default. Do not disable AUTOCOMMIT during the client session when restoring the database from the dump, otherwise the host storage may overflow and the cluster may not function properly.

This method is suitable if you used mysqldump to create the dump.

  1. Install the mysql utility to the host you are using to restore the dump, e.g., for Ubuntu:

    sudo apt update && sudo apt install mysql-client --yes

  2. Start the database restore from the dump:

    • If you restore a dump from the VM in Yandex Cloud:

      mysql \
    --host=c-<target_cluster_ID>.rw.mdb.yandexcloud.net \
    --user=<username> \
    --port=3306 \
    <DB_name> < ~/db_dump.sql

    • If you are restoring the dump from a host connecting to Yandex Cloud from the internet, get an SSL certificate and provide the --ssl-ca and the --ssl-mode parameters in the restore command:

      mysql \
    --host=c-<target_cluster_ID>.rw.mdb.yandexcloud.net \
    --user=<username> \
    --port=3306 \
    --ssl-ca=~/.mysql/root.crt \
    --ssl-mode=VERIFY_IDENTITY \
    <DB_name> < ~/db_dump.sql

This method is suitable if you created the dump with mydumper and are using an intermediate virtual machine to restore it.

  1. Install the myloader utility to the host you are using to restore the dump, e.g., for Ubuntu:

    sudo apt update && sudo apt install mydumper --yes

  2. Start the database restore from the dump:

    myloader \
    --host=c-<target_cluster_ID>.rw.mdb.yandexcloud.net \
    --directory=db_dump/ \
    --overwrite-tables \
    --threads=8 \
    --compress-protocol \
    --user=<username> \
    --ask-password

You can get the cluster ID with a list of clusters in the folder.

Deleting the created resources

Delete the resources you no longer need to avoid paying for them:

To delete the infrastructure created with Terraform:

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the data-migration-mysql-mmy.tf configuration file.

  3. Check that the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the resources described in the data-migration-mysql-mmy.tf configuration file will be deleted.

Previous
Exporting a database to Yandex Data Processing
Next
Migrating a database from Managed Service for MySQL® to MySQL®