Managing transfer process

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To get a list of transfers in a folder, run the following command:

yc datatransfer transfer list

Use the list API method.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To get information about a transfer, run the following command:

yc datatransfer transfer get <transfer_ID>

You can get the transfer ID with the list of transfers in the folder.

Use the get API method and provide the transfer ID value in the transferId request parameter.

To find out the transfer ID, get a list of transfers in the folder.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a transfer:

1. View a description of the CLI create transfer command:

   yc datatransfer transfer create --help
   

2. Specify the transfer parameters in the create command:

   yc datatransfer transfer create <transfer_name> \
      --source-id=<source_endpoint_ID> \
      --target-id=<target_endpoint_ID> \
      --type=<transfer_type>
   

   Where:

   • --source-id: Source endpoint ID.
   • --target-id: Target endpoint ID.
   • --type: Transfer type:
     • snapshot-only: Copy.
     • increment-only: Replicate.
     • snapshot-and-increment: Copy and replicate.

   Note

   The transfer name must be unique within the folder. It may contain Latin letters, numbers, and hyphens. The name may be up to 63 characters long.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a transfer:

1. Create a configuration file with a description of your transfer.

   Here is an example of the configuration file structure:

   resource "yandex_datatransfer_transfer" "<transer_name_in_Terraform>" {
     folder_id   = "<folder_ID>"
     name        = "<transfer_name>"
     description = "<transfer_description>"
     source_id   = "<source_endpoint_ID>"
     target_id   = "<target_endpoint_ID>"
     type        = "<transfer_type>"
   }
   

   Where:

   • type: Transfer type. The possible values are:

     • SNAPSHOT_ONLY: Copying.
     • INCREMENT_ONLY: Replication.
     • SNAPSHOT_AND_INCREMENT: Copying and replication.

     The INCREMENT_ONLY and SNAPSHOT_AND_INCREMENT transfers will activate and run automatically upon creation.

   • To activate a SNAPSHOT_ONLY transfer once it is created:

     1. If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

        By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

     2. Add the provisioner "local-exec" section with the following transfer activation command to the configuration file:

        provisioner "local-exec" {
          command = "yc --profile <profile> datatransfer transfer activate ${yandex_datatransfer_transfer.<transfer_Terraform_resource_name>.id}"
        }
        

        In this case, copying will only take place once at the time of transfer creation.

   • To specify runtime environment settings for a SNAPSHOT_ONLY transfer, add the runtime section to the resource description:

     resource "yandex_datatransfer_transfer" "<transer_name_in_Terraform>" {
       ...
       runtime {
         yc_runtime {
           job_count = <number_of_workers>
           upload_shard_params {
             job_count     = <number_of_workers>
             process_count = <number_of_streams>
           }
         }
       }
       ...
     }
     

     Where:

     • job_count: Number of workers running in parallel.

     • upload_shard_params: Parallel copying settings.

       • job_count: Number of workers.
       • process_count: Number of threads within each worker.

   • To specify runtime environment settings for an INCREMENT_ONLY transfer, add the replication_runtime section to the resource description:

     resource "yandex_datatransfer_transfer" "<transer_name_in_Terraform>" {
       ...
       replication_runtime {
         yc_runtime {
           job_count = <number_of_workers>
           upload_shard_params {
             job_count     = <number_of_workers>
             process_count = <number_of_streams>
           }
         }
       }
       ...
     }
     

   • To specify runtime environment settings for a SNAPSHOT_AND_INCREMENT transfer, add the runtime and replication_runtime sections to the resource description:

     resource "yandex_datatransfer_transfer" "<transer_name_in_Terraform>" {
       ...
       runtime {
         yc_runtime {
           job_count = <number_of_workers>
           upload_shard_params {
             job_count     = <number_of_workers>
             process_count = <number_of_streams>
           }
         }
       }
     
       replication_runtime {
         yc_runtime {
           job_count = <number_of_workers>
           upload_shard_params {
             job_count     = <number_of_workers>
             process_count = <number_of_streams>
           }
         }
       }
       ...
     }
     

     Where:

     • runtime: Runtime environment settings for a copyng transfer.
     • replication_runtime: Runtime environment settings for a replication transfer.

2. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Confirm updating the resources.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information, see this Terraform provider guide.

Use the Transfer.Create REST API method and provide the following in the request:

• ID of the folder where the transfer should be placed, in the folderId parameter.

• Transfer name in the name parameter.

• Source endpoint ID in the sourceId parameter.

• Target endpoint ID in the targetId parameter.

• Transfer type in the type parameter. The possible values are:

  • SNAPSHOT_ONLY: Copying.
  • INCREMENT_ONLY: Replication.
  • SNAPSHOT_AND_INCREMENT: Copying and replication.

• Runtime environment settings:

  • For the SNAPSHOT_ONLY transfer type, in the runtime object.

  • For the INCREMENT_ONLY transfer type, in the replicationRuntime object.

  • For the SNAPSHOT_AND_INCREMENT transfer type:

    • In the runtime object for copying.
    • In the replicationRuntime object for replication.

Use the TransferService.Create gRPC API method and provide the following in the request:

• ID of the folder where the transfer should be placed, in the folder_id parameter.

• Transfer name in the name parameter.

• Source endpoint ID in the source_id parameter.

• Target endpoint ID in the target_id parameter.

• Transfer type in the type parameter. The possible values are:

  • SNAPSHOT_ONLY: Copying.
  • INCREMENT_ONLY: Replication.
  • SNAPSHOT_AND_INCREMENT: Copying and replication.

• Runtime environment settings:

  • For the SNAPSHOT_ONLY transfer type, in the runtime object.

  • For the INCREMENT_ONLY transfer type, in the replication_runtime object.

  • For the SNAPSHOT_AND_INCREMENT transfer type:

    • In the runtime object for copying.
    • In the replication_runtime object for replication.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To update the transfer settings:

1. View a description of the update transfer CLI command:

   yc datatransfer transfer update --help
   

2. Run the following command with the list of settings to update:

   yc datatransfer transfer update <transfer_ID> \
      --name=<transfer_name> \
      --description=<transfer_description>
   

   You can get the transfer ID with the list of transfers in the folder.

1. Open the current Terraform configuration file with the transfer description.

   For information on creating a transfer like this, please review Create transfer.

2. Edit the values in the name and the description fields (transfer name and description).

3. Update the runtime environment settings:

   • For the SNAPSHOT_ONLY transfer type, in the runtime section.

   • For the INCREMENT_ONLY transfer type, in the replication_runtime section.

     If you have already specified the runtime environment settings under runtime, there is no need to add the replication_runtime section.

   • For the SNAPSHOT_AND_INCREMENT transfer type:

     • In the runtime section for copying.
     • In the replication_runtime section for replication.

     If only the runtime section is provided, the runtime environment settings for copying and replication will be identical.

4. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

5. Confirm updating the resources.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information, see this Terraform provider guide.

Use the Transfer.Update REST API method and provide the following in the request:

• Transfer ID in the transferId parameter. To find out the ID, get a list of transfers in the folder.

• Transfer name in the name parameter.

• Transfer description in the description parameter.

• Runtime environment settings:

  • For the SNAPSHOT_ONLY transfer type, in the runtime object.

  • For the INCREMENT_ONLY transfer type, in the replicationRuntime object.

    If you have already specified the runtime environment settings in the runtime object, there is no need to provide replicationRuntime.

  • For the SNAPSHOT_AND_INCREMENT transfer type:

    • In the runtime object for copying.
    • In the replicationRuntime object for replication.

    If only the runtime section is provided, the runtime environment settings for copying and replication will be identical.

• List of transfer configuration fields to update in the updateMask parameter.

  Warning

  When you update a transfer, all parameters of the object you are modifying will take their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the updateMask parameter.

Use the TransferService.Update gRPC API method and provide the following in the request:

• Transfer ID in the transfer_id parameter. To find out the ID, get a list of transfers in the folder.

• Transfer name in the name parameter.

• Transfer description in the description parameter.

• Runtime environment settings:

  • For the SNAPSHOT_ONLY transfer type, in the runtime object.

  • For the INCREMENT_ONLY transfer type, in the replication_runtime object.

    If you have already specified the runtime environment settings in the runtime object, there is no need to provide replication_runtime.

  • For the SNAPSHOT_AND_INCREMENT transfer type:

    • In the runtime object for copying.
    • In the replication_runtime object for replication.

    If only the runtime section is provided, the runtime environment settings for copying and replication will be identical.

• List of transfer configuration fields to update in the update_mask parameter.

  Warning

  When you update a transfer, all parameters of the object you are modifying will take their defaults unless explicitly provided in the request. To avoid this, list the settings you want to change in the update_mask parameter.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To activate a transfer, run this command:

yc datatransfer transfer activate <transfer_ID>

You can get the transfer ID with the list of transfers in the folder.

Use the activate API method and provide the transfer ID in the transferId request parameter.

To find out the transfer ID, get a list of transfers in the folder.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To deactivate a transfer, run this command:

yc datatransfer transfer deactivate <transfer_ID>

You can get the transfer ID with the list of transfers in the folder.

Use the deactivate API method and provide the transfer ID in the transferId request parameter.

To find out the transfer ID, get a list of transfers in the folder.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To delete a transfer, run this command:

yc datatransfer transfer delete <transfer_ID>

You can get the transfer ID with the list of transfers in the folder.

To delete a transfer created using Terraform:

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating a transfer.

2. Delete the transfer description.

3. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

4. Type the word yes, then press Enter.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Use the delete API method and provide the transfer ID in the transferId request parameter.

To find out the transfer ID, get a list of transfers in the folder.