Migrating databases from a third-party Redis cluster

For data migration, Redis uses a logical dump: this is a file with a sequence of commands to restore the state of databases in the cluster. There are several ways to create a dump. The following example will use redis-dump-go.

To migrate Redis databases from the source cluster to the target cluster:

1. Connect to the source cluster and create a logical dump.

2. (Optional) Upload the dump to the intermediate virtual machine.

   You must transfer data to Yandex Compute Cloud using a virtual machine if:

   • Your Managed Service for Redis cluster is not accessible from the internet.
   • Your hardware or connection to the cluster in Yandex Cloud is not very reliable.

3. Restore the dump on the target cluster.

4. Make sure the dump is fully restored.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Prepare the infrastructurePrepare the infrastructure

Install additional softwareInstall additional software

1. (Optional) Install utilities on the local machine for downloading and uploading files over SSH, such as:

   • WinSCP
   • Putty SCP

2. Make sure that GNU Screen is installed on the source cluster.

   It might take a long time to create and restore a dump. To keep these processes alive when your SSH session times out, start them using this utility. If your SSH connection breaks while creating or restoring the dump, reconnect and restore the session state using the command:

   screen -R
   

Connect to the source cluster and create a logical dumpConnect to the source cluster and create a logical dump

1. Connect to the source cluster's master host via SSH.

2. Download the archive with the redis-dump-go utility from the project page. The examples below use version 0.5.1.

   wget https://github.com/yannh/redis-dump-go/releases/download/v0.5.1/redis-dump-go_0.5.1_linux_amd64.tar.gz
   

3. Unpack the archive to the current directory:

   tar xf redis-dump-go_0.5.1_linux_amd64.tar.gz
   

4. Get familiar with the utility launch parameters:

   ./redis-dump-go -h
   

5. If connecting to the Redis cluster requires a password, enter it in the REDISDUMPGO_AUTH environment variable:

   export REDISDUMPGO_AUTH="<Redis_password>"
   

6. Start an interactive screen session:

   screen
   

7. Launch the creation of a logical dump:

   ./redis-dump-go \
       -host <IP_address_or_FQDN_of_Redis_cluster_master_host> \
       -port <Redis_port> > <dump_file>
   

   Tip

   As the dump is created, the number of processed keys is shown on the screen. Remember or write down the last output value: you will need it to check whether the dump has been restored completely in the target cluster.

8. When the dump has been created, download it to your computer.

9. End the interactive screen session:

   exit
   

(Optional) Upload the dump to the intermediate virtual machine(Optional) Upload the dump to the intermediate virtual machine

1. Connect to the intermediate virtual machine via SSH.

2. Upload the dump from your computer to an intermediate virtual machine however is convenient.

Restore the dump on the target clusterRestore the dump on the target cluster

1. Connect to the cluster and run an interactive screen session:

   screen
   

2. Start the dump recovery process:

   Connecting without TLS

   Connecting via TLS

   Before connecting, install the dependencies:

   sudo apt update && sudo apt install -y redis-tools
   

   Connecting via Sentinel

   host=$(redis-cli \
     -h <FQDN_of_any_Redis_host> \
     -p 26379 \
     sentinel \
     get-master-addr-by-name \
     no-shards-no-tls | head -n 1)
   redis-cli \
     -h ${host} \
     -p 6379 \
     -a <target cluster password> \
     --pipe < <dump_file>
   

   Connecting directly to the master host

   redis-cli \
     -h <FQDN_of_any_host> \
     -p 6379 \
     -a <target_cluster_password> \
     --pipe < <dump file>
   

   When connecting to a non-sharded cluster, instead of the master host's FQDN, you can use special FQDNs.

   Connecting to a sharded cluster

   1. Create a script containing the dump-loading commands:

      load-dump.sh

      shards=('<FQDN_of_master_host_in_shard_1>' \
              ...
              '<FQDN_of_master_host_in_shard_N>')
      
      for shard in "${shards[@]}" ; do
        redis-cli -h "${shard}" \
                  -p 6379 \
                  -a "<target_cluster_password>" \
                  --pipe < <dump_file>
      done
      

   2. Run the script:

      bash ./load-dump.sh
      

      As you run the script, you will see messages about data insertion errors. This is normal behavior for the redis-cli command, because in a sharded cluster, each shard stores only a certain part of the data. For more information, see Sharding in Managed Service for Redis.

   Before connecting, install the dependencies:

   Build the redis-tools utility with TLS support in one of two ways:

   • From a repository

     1. Connect a repository:

        sudo apt-add-repository ppa:redislabs/redis
        

        Packages in this repository have already been built with the BUILD_TLS=yes flag.

     2. Install the utility:

        sudo apt update && sudo apt install -y redis-tools
        

   • Manually

     Go to the directory you want to download the distribution to. Download the stable version of the utility, then build and install it:

     wget https://download.redis.io/redis-stable.tar.gz && \
     tar -xzvf redis-stable.tar.gz && \
     cd redis-stable && \
     make BUILD_TLS=yes && \
     sudo make install && \
     sudo cp ./src/redis-cli /usr/bin/
     

   Connecting via Sentinel

   host=$(redis-cli \
          -h <FQDN_of_any_Redis_host> \
          -p 26379 \
          sentinel \
          get-master-addr-by-name \
          no-shards-tls | head -n 1)
   redis-cli \
       -h ${host} \
       -p 6380 \
       -a <target_cluster_password> \
       --tls \
       --cacert ~/.redis/YandexInternalRootCA.crt \
       --pipe < <dump_file>
   

   Connecting directly to the master host

   redis-cli \
       -h c-<cluster_ID>.rw.mdb.yandexcloud.net \
       -p 6380 \
       -a <target_cluster_password> \
       --tls \
       --cacert ~/.redis/YandexInternalRootCA.crt \
       --pipe < <dump_file>
   

   When connecting to a non-sharded cluster, instead of the master host's FQDN, you can use special FQDNs.

   Connecting to a sharded cluster

   1. Create a script containing the dump-loading commands:

      load-dump.sh

      shards=('<FQDN_of_master_host_in_shard_1>' \
              ...
              '<FQDN_of_master_host_in_shard_N>')
      
      for shard in "${shards[@]}" ; do
        redis-cli -h "${shard}" \
                  -p 6380 \
                  -a "<target_cluster_password>" \
                  --tls \
                  --cacert ~/.redis/YandexInternalRootCA.crt \
                  --pipe < <dump_file>
      done
      

   2. Run the script:

      bash ./load-dump.sh
      

      As you run the script, you will see messages about data insertion errors. This is normal behavior for the redis-cli command, because in a sharded cluster, each shard stores only a certain part of the data. For more information, see Sharding in Managed Service for Redis.

3. End the interactive screen session:

   exit
   

Make sure that the dump is restored completelyMake sure that the dump is restored completely

1. In the management console, select the folder containing the cluster to restore.
2. In the list of services, select Managed Service for Redis.
3. Click the cluster name and open the Monitoring tab.

Pay attention to the DB Keys chart showing the number of keys stored in the cluster. If the cluster is sharded, the chart will show the number of keys in each shard. In this case, the number of keys in the cluster is equal to the total number of keys in the shards.

The total number of keys in the cluster must be equal to the number of keys processed by the redis-dump-go utility when creating the dump.

Delete the resources you createdDelete the resources you created

Delete the resources you no longer need to avoid paying for them: