Migrating a database from a third-party Valkey™ cluster

For data migration, Valkey™ uses a logical dump. A logical dump is a file with a sequence of commands that restore the database state. There are several ways to create a dump. The following example will use redis-dump-go.

To migrate Valkey™ databases from the source cluster to the target cluster:

1. Connect to the source cluster and create a logical dump.

2. Optionally, upload the dump to the intermediate VM.

   You need to use a VM in Yandex Compute Cloud to transfer data if:

   • Your Yandex Managed Service for Valkey™ cluster is not accessible from the internet.
   • Your local hardware or Yandex Cloud cluster connection is not reliable enough.

3. Restore the dump in the target cluster.

4. Make sure the dump is fully restored.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost for this solution includes:

• Yandex Managed Service for Valkey™ target cluster fee, which covers the use of computing resources allocated to hosts, and disk space (see Valkey™ pricing).
• Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• When creating a VM for dump upload: fee for the use of computing resources, storage, operating system (for certain OS types), and optionally public IP address (see Compute Cloud pricing).

Getting startedGetting started

Set up your infrastructureSet up your infrastructure

Install additional softwareInstall additional software

1. Optionally, install SSH file transfer tools on your local machine, such as the following:

   • WinSCP
   • Putty SCP

2. Make sure GNU Screen is installed on the source cluster.

   Creating and restoring a dump may take a long time. To keep these processes running when your SSH session times out, start them with this tool. If your SSH connection drops during these operations, reconnect and resume the session using this command:

   screen -R
   

Connect to the source cluster and create a logical dumpConnect to the source cluster and create a logical dump

1. Connect to the source cluster master host over SSH.

2. Download the archive with redis-dump-go from the project page. The examples below use version 0.5.1.

   wget https://github.com/yannh/redis-dump-go/releases/download/v0.5.1/redis-dump-go_0.5.1_linux_amd64.tar.gz
   

3. Unpack the archive to the current directory:

   tar xf redis-dump-go_0.5.1_linux_amd64.tar.gz
   

4. Review the tool’s command-line options:

   ./redis-dump-go -h
   

5. If connecting to the Valkey™ cluster requires a password, enter it in the REDISDUMPGO_AUTH environment variable:

   export REDISDUMPGO_AUTH="<Valkey™_password>"
   

6. Start an interactive screen session:

   screen
   

7. Start creating a logical dump:

   ./redis-dump-go \
       -host <master_host_IP_address_or_FQDN_in_Valkey™_cluster> \
       -port <Valkey™_port> > <dump_file>
   

   Tip

   While the dump is being created, the number of processed keys will appear on the screen. Remember or write down the last output value, as you will need it to check whether the dump has been fully restored in the target cluster.

8. Once the dump is created, download it to your computer.

9. Terminate the interactive screen session:

   exit
   

Optionally, upload the dump to the intermediate VMOptionally, upload the dump to the intermediate VM

1. Connect to the intermediate VM over SSH.

2. Upload the dump from your computer to the intermediate VM in any way you like.

Restore the dump in the target clusterRestore the dump in the target cluster

1. Connect to the cluster and run an interactive screen session:

   screen
   

2. Start the dump recovery:

   Connecting without TLS

   Connecting with TLS

   Before connecting, install the required dependencies:

   sudo apt update && sudo apt install -y redis-tools
   

   Connecting via Sentinel

   host=$(redis-cli \
     -h <FQDN_of_any_Valkey™_host> \
     -p 26379 \
     sentinel \
     get-master-addr-by-name \
     no-shards-no-tls | head -n 1)
   redis-cli \
     -h ${host} \
     -p 6379 \
     -a <target_cluster_password> \
     --pipe < <dump_file>
   

   Connecting directly to the master host

   redis-cli \
     -h <master_host_FQDN> \
     -p 6379 \
     -a <target_cluster_password> \
     --pipe < <dump_file>
   

   When connecting to a non-sharded cluster, you can use special FQDNs instead of the master host FQDN.

   Connecting to a sharded cluster

   1. Create a script with commands to load the dump:

      load-dump.sh

      shards=('<FQDN_of_master_host_in_shard_1>' \
              ...
              '<FQDN_of_master_host_in_shard_N>')
      
      for shard in "${shards[@]}" ; do
        redis-cli -h "${shard}" \
                  -p 6379 \
                  -a "<target_cluster_password>" \
                  --pipe < <dump_file>
      done
      

   2. Run the script:

      bash ./load-dump.sh
      

      While the script runs, you may see data insertion error messages. This is normal behavior for the redis-cli command, as each shard in a sharded cluster stores only a portion of the data. For more information, see Sharding in Yandex Managed Service for Valkey™.

   Before connecting, install the required dependencies:

   Build redis-tools with TLS support using one of the two methods:

   • From a repository

     1. Add a repository:

        sudo apt-add-repository ppa:redislabs/redis
        

        The repository’s packages come pre-built with BUILD_TLS=yes.

     2. Install the tool:

        sudo apt update && sudo apt install -y redis-tools
        

   • Manually

     Go to the directory you want to download the distribution to. Download the stable tool version, then build and install it:

     wget https://download.redis.io/redis-stable.tar.gz && \
     tar -xzvf redis-stable.tar.gz && \
     cd redis-stable && \
     make BUILD_TLS=yes && \
     sudo make install && \
     sudo cp ./src/redis-cli /usr/bin/
     

   Connecting via Sentinel

   host=$(redis-cli \
          -h <FQDN_of_any_Valkey™_host> \
          -p 26379 \
          sentinel \
          get-master-addr-by-name \
          no-shards-tls | head -n 1)
   redis-cli \
       -h ${host} \
       -p 6380 \
       -a <target_cluster_password> \
       --tls \
       --cacert ~/.redis/YandexInternalRootCA.crt \
       --pipe < <dump_file>
   

   Connecting directly to the master host

   redis-cli \
       -h c-<cluster_ID>.rw.mdb.yandexcloud.net \
       -p 6380 \
       -a <target_cluster_password> \
       --tls \
       --cacert ~/.redis/YandexInternalRootCA.crt \
       --pipe < <dump_file>
   

   When connecting to a non-sharded cluster, you can use special FQDNs instead of the master host FQDN.

   Connecting to a sharded cluster

   1. Create a script with commands to load the dump:

      load-dump.sh

      shards=('<FQDN_of_master_host_in_shard_1>' \
              ...
              '<FQDN_of_master_host_in_shard_N>')
      
      for shard in "${shards[@]}" ; do
        redis-cli -h "${shard}" \
                  -p 6380 \
                  -a "<target_cluster_password>" \
                  --tls \
                  --cacert ~/.redis/YandexInternalRootCA.crt \
                  --pipe < <dump_file>
      done
      

   2. Run the script:

      bash ./load-dump.sh
      

      While the script runs, you may see data insertion error messages. This is normal behavior for the redis-cli command, as each shard in a sharded cluster stores only a portion of the data. For more information, see Sharding in Yandex Managed Service for Valkey™.

3. Terminate the interactive screen session:

   exit
   

Make sure the dump is fully restoredMake sure the dump is fully restored

1. In the management console, navigate to the folder where you want to restore the cluster.
2. In the list of services, select Yandex Managed Service for Valkey™.
3. Click the cluster name and open the Monitoring tab.

Pay attention to the DB Keys chart showing the number of keys stored in the cluster. If the cluster is sharded, the chart will show the number of keys in each shard. In this case, the number of keys in the cluster is equal to the total number of keys across all shards.

The total number of keys in the cluster must match the number of keys processed by redis-dump-go when creating the dump.

Delete the resources you createdDelete the resources you created

Delete the resources you no longer need to avoid paying for them: