Managing backups in Managed Service for ClickHouse®

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a cluster backup:

1. View a description of the CLI create ClickHouse® backup command:

   yc managed-clickhouse cluster backup --help
   

2. Request the creation of a backup specifying the cluster name or ID:

   yc managed-clickhouse cluster backup <cluster_name_or_ID>
   

   You can get the cluster name and ID with a list of clusters in the folder.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.backup method and make a request, e.g., via cURL:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --header "Content-Type: application/json" \
       --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>:backup'
   

   You can get the cluster ID with a list of clusters in the folder.

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/Backup call and make a request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d '{
             "cluster_id": "<cluster_ID>"
           }' \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.clickhouse.v1.ClusterService.Backup
   

   You can get the cluster ID with a list of clusters in the folder.

4. View the server response to make sure the request was successful.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To restore a cluster from a backup:

1. View a description of the CLI restore ClickHouse® cluster command:

   yc managed-clickhouse cluster restore --help
   

2. Getting a list of available ClickHouse® cluster backups:

   yc managed-clickhouse backup list
   

   +--------------------------+---------------------+----------------------+---------------------+-------------+-------+-----------+
   |            ID            |     CREATED AT      |  SOURCE CLUSTER ID   |     STARTED AT      | SHARD NAMES | SIZE  |   TYPE    |
   +--------------------------+---------------------+----------------------+---------------------+-------------+-------+-----------+
   | c9qud5etkq19********:... | 2023-12-08 00:09:17 | c9qud5etkq19******** | 2023-12-08 00:08:06 | shard1      | 30 KB | AUTOMATED |
   | ...                      |                     |                      |                     |             |       |           |
   +--------------------------+---------------------+----------------------+---------------------+-------------+-------+-----------+
   

3. To restore an individual shard, pass the ID of a single backup:

   yc managed-clickhouse cluster restore \
      --backup-id=<backup_ID> \
      --name=<cluster_name> \
      --environment=<environment> \
      --network-name=<network_name> \
      --host type=<host_type>,`
            `zone-id=<availability_zone>,`
            `subnet-id=<subnet_ID> \
      --clickhouse-disk-size=<storage_size_in_GB> \
      --clickhouse-disk-type=<disk_type> \
      --clickhouse-resource-preset=<host_class>
   

   Where:

   • --backup-id: Backup ID.

   • --name: Cluster name.

   • --environment: Environment:

     • PRODUCTION: For stable versions of your apps.
     • PRESTABLE: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.

   • --network-name: Network name.

   • --host: Host parameters:

     • type: Host type: clickhouse or zookeeper.
     • zone-id: Availability zone.
     • subnet-id: Subnet ID. Specify if two or more subnets are created in the selected availability zone.

   • --resource-preset: Host class.

   • --disk-size: Storage size in GB.

   • --disk-type: Disk type:

     • network-hdd

     • network-ssd

     • local-ssd

     • network-ssd-nonreplicated

4. To restore the entire cluster, provide backup IDs for all cluster shards:

   yc managed-clickhouse cluster restore \
      --backup-id=<list_of_backup_IDs_for_all_shards> \
      ...
   

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.restore method and make a request, e.g., via cURL:

   1. Create a file named body.json and add the following contents to it:

      {
        "backupId": "<backup_ID>",
        "additionalBackupIds": [
          <list_of_additional_IDs_of_backups>
        ],
        "name": "<cluster_name>",
        "environment": "<environment>",
        "configSpec": { <cluster_configuration> },
        "hostSpecs": [
          {
            "type": "<host_type>",
            "zoneId": "<availability_zone>",
            "subnetId": "<subnet_ID>",
            "shardName": "<shard_name>",
            "assignPublicIp": <public_access_to_host>
          },
          { <similar_configuration_for_host_2> },
          { ... },
          { <similar_configuration_for_host_N> }
        ],
        "folderId": "<folder_ID>",
        "networkId": "<network_ID>",
        "serviceAccountId": "<service_account_ID>",
        "securityGroupIds": [
          <list_of_security_group_IDs>
        ],
        "deletionProtection": <deletion_protection>
      }
      

      Where:

      • backupId: Backup ID.

        The shard for which that backup was created will be restored.

      • additionalBackupIds: Array containing a list of additional IDs of your backups.

        The shards for which those backups were created will be restored. To restore the entire cluster, provide backup IDs for all cluster shards.

      • name: Cluster name.

      • environment: Environment:

        • PRODUCTION: For stable versions of your apps.
        • PRESTABLE: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.

      • configSpec: ClickHouse® cluster configuration. For detailed description of the parameters, see the Cluster.restore method description.

      • hostSpecs: Array with settings for the new hosts. One array element contains settings for a single host and has the following structure:

        • type: Host type.
        • zoneId: Availability zone.
        • subnetId: Subnet ID.
        • shardName: Shard name.
        • assignPublicIp: Internet access to the host via a public IP address, true or false.

      • folderId: ID of the folder to create your cluster in.

      • networkId: ID of the network to create your cluster in.

      • serviceAccountId: Service account ID.

      • securityGroupIds: Array of security group IDs.

      • deletionProtection: Option to manage protection of the cluster, its databases, and its users against accidental deletion.

        Enabled deletion protection will not prevent a manual connection and database content deletion.

      You can get the backup ID together with a list of backups in the relevant folder.

   2. Run this request:

      curl \
          --request POST \
          --header "Authorization: Bearer $IAM_TOKEN" \
          --header "Content-Type: application/json" \
          --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters:restore' \
          --data "@body.json"
      

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/Restore call and make a request, e.g., via gRPCurl:

   1. Create a file named body.json and add the following contents to it:

      {
        "backup_id": "<backup_ID>",
        "additional_backup_ids": [
          <list_of_additional_IDs_of_backups>
        ],
        "name": "<cluster_name>",
        "environment": "<environment>",
        "config_spec": { <cluster_configuration> },
        "host_specs": [
          {
            "type": "<host_type>",
            "zone_id": "<availability_zone>",
            "subnet_id": "<subnet_ID>",
            "shard_name": "<shard_name>",
            "assign_public_ip": <public_access_to_host>
          },
          { <similar_configuration_for_host_2> },
          { ... },
          { <similar_configuration_for_host_N> }
        ],
        "folder_id": "<folder_ID>",
        "network_id": "<network_ID>",
        "service_account_id": "<service_account_ID>",
        "security_group_ids": [
          <list_of_security_group_IDs>
        ],
        "deletion_protection": <deletion_protection>
      }
      

      Where:

      • backup_id: Backup ID.

        The shard for which that backup was created will be restored.

      • additional_backup_ids: Array containing a list of additional IDs of your backups.

        The shards for which those backups were created will be restored. To restore the entire cluster, provide backup IDs for all cluster shards.

      • name: Cluster name.

      • environment: Environment:

        • PRODUCTION: For stable versions of your apps.
        • PRESTABLE: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.

      • config_spec: ClickHouse® cluster configuration. For detailed description of the parameters, see the ClusterService/Restore call description.

      • host_specs: Array with settings for the new hosts. One array element contains settings for a single host and has the following structure:

        • type: Host type.
        • zone_id: Availability zone.
        • subnet_id: Subnet ID.
        • shard_name: Shard name.
        • assign_public_ip: Internet access to the host via a public IP address, true or false.

      • folder_id: ID of the folder to create your cluster in.

      • network_id: ID of the network to create your cluster in.

      • service_account_id: Service account ID.

      • security_group_ids: Array of security group IDs.

      • deletion_protection: Option to manage protection of the cluster, its databases, and its users against accidental deletion.

        Enabled deletion protection will not prevent a manual connection and database content deletion.

      You can get the backup ID together with a list of backups in the relevant folder.

   2. Run this request:

      grpcurl \
          -format json \
          -import-path ~/cloudapi/ \
          -import-path ~/cloudapi/third_party/googleapis/ \
          -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
          -rpc-header "Authorization: Bearer $IAM_TOKEN" \
          -d @ \
          mdb.api.cloud.yandex.net:443 \
          yandex.cloud.mdb.clickhouse.v1.ClusterService.Restore \
          < body.json
      

4. View the server response to make sure the request was successful.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of ClickHouse® cluster backups available in the default folder, run the command:

yc managed-clickhouse backup list

+--------------------------+---------------------+----------------------+---------------------+-------------+-------+-----------+
|            ID            |     CREATED AT      |  SOURCE CLUSTER ID   |     STARTED AT      | SHARD NAMES | SIZE  |   TYPE    |
+--------------------------+---------------------+----------------------+---------------------+-------------+-------+-----------+
| c9qud5etkq19********:... | 2023-12-08 00:09:17 | c9qud5etkq19******** | 2023-12-08 00:08:06 | shard1      | 30 KB | AUTOMATED |
| c9qud5etkq19********:... | 2023-12-07 08:17:04 | c9qud5etkq19******** | 2023-12-07 08:15:54 | shard1      | 30 KB | MANUAL    |
+--------------------------+---------------------+----------------------+---------------------+-------------+-------+-----------+

The resulting table contains the following information:

• ID of the backup.
• End time of backup creation in UTC (Coordinated Universal Time).
• ID of the cluster that the backup was created for.
• Start time of backup creation in UTC.
• Source shard name.
• Backup size.
• Backup type: Automatic (AUTOMATED) or manual (MANUAL).

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. To get a list of ClickHouse® cluster backups:

   1. Use the Cluster.listBackups method and make a request, e.g., via cURL:

      curl \
          --request GET \
          --header "Authorization: Bearer $IAM_TOKEN" \
          --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/backups'
      

      You can get the cluster ID with a list of clusters in the folder.

   2. View the server response to make sure the request was successful.

3. To get a list of backups for all the ClickHouse® clusters in a folder:

   1. Use the Backup.list method and make a request, e.g., via cURL:

      curl \
          --request GET \
          --header "Authorization: Bearer $IAM_TOKEN" \
          --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/backups' \
          --url-query folderId=<folder_ID>
      

      You can request the folder ID with a list of folders in the cloud.

   2. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. To get a list of ClickHouse® cluster backups:

   1. Use the ClusterService/ListBackups call and make a request, e.g., via gRPCurl:

      grpcurl \
          -format json \
          -import-path ~/cloudapi/ \
          -import-path ~/cloudapi/third_party/googleapis/ \
          -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
          -rpc-header "Authorization: Bearer $IAM_TOKEN" \
          -d '{
                  "cluster_id": "<cluster_ID>"
              }' \
          mdb.api.cloud.yandex.net:443 \
          yandex.cloud.mdb.clickhouse.v1.ClusterService.ListBackups
      

      You can get the cluster ID with a list of clusters in the folder.

   2. View the server response to make sure the request was successful.

4. To get a list of backups for all the ClickHouse® clusters in a folder:

   1. Use the BackupService/List call and make a request, e.g., via gRPCurl:

      grpcurl \
          -format json \
          -import-path ~/cloudapi/ \
          -import-path ~/cloudapi/third_party/googleapis/ \
          -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/backup_service.proto \
          -rpc-header "Authorization: Bearer $IAM_TOKEN" \
          -d '{
                  "folder_id": "<folder_ID>"
              }' \
          mdb.api.cloud.yandex.net:443 \
          yandex.cloud.mdb.clickhouse.v1.BackupService.List
      

      You can request the folder ID with a list of folders in the cloud.

   2. View the server response to make sure the request was successful.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get information about a ClickHouse® cluster backup, run the command:

yc managed-clickhouse backup get <backup_ID>

You can retrieve the backup ID with a list of backups.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Backup.get method and make a request, e.g., via cURL:

   curl \
       --request GET \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/backups/<backup_ID>'
   

   You can get the backup ID together with a list of backups in the relevant folder.

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the BackupService/Get call and make a request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/backup_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d '{
               "backup_id": "<backup_ID>"
           }' \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.clickhouse.v1.BackupService.Get
   

   You can get the backup ID together with a list of backups in the relevant folder.

4. View the server response to make sure the request was successful.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To set the backup start time, provide the required value in HH:MM:SS format in the --backup-window-start argument of the cluster update command:

yc managed-clickhouse cluster update <cluster_name_or_ID> \
   --backup-window-start=<backup_start_time>

You can request the cluster ID and name with a list of clusters in the folder.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.update method and make a request, e.g., via cURL:

   Warning

   The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

   curl \
       --request PATCH \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --header "Content-Type: application/json" \
       --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>' \
       --data '{
                 "updateMask": "configSpec.backupWindowStart",
                 "configSpec": {
                   "backupWindowStart": {
                     "hours": "<hours>",
                     "minutes": "<minutes>",
                     "seconds": "<seconds>",
                     "nanos": "<nanoseconds>"
                   }
                 }
               }'
   

   Where:

   • updateMask: List of parameters to update as a single string, separated by commas.

     Here only one parameter is specified: configSpec.backupWindowStart.

   • configSpec.backupWindowStart: Backup start time (UTC):

     • hours: Hours in 24-hour format
     • minutes: Minutes
     • seconds: Seconds
     • nanos: Nanoseconds

   You can get the cluster ID with a list of clusters in the folder.

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/Update call and make a request, e.g., via gRPCurl:

   Warning

   The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.

   Format for listing settings

   "update_mask": {
       "paths": [
           "<setting_1>",
           "<setting_2>",
           ...
           "<setting_N>"
       ]
   }
   

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d '{
             "cluster_id": "<cluster_ID>",
             "update_mask": {
               "paths": [
                 "config_spec.backup_window_start"
               ]
             },
             "config_spec": {
               "backup_window_start": {
                 "hours": "<hours>",
                 "minutes": "<minutes>",
                 "seconds": "<seconds>",
                 "nanos": "<nanoseconds>"
               }
             }
           }' \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.clickhouse.v1.ClusterService.Update
   

   Where:

   • update_mask: List of parameters to update as an array of paths[] strings.

     Here only one parameter is specified: config_spec.backup_window_start.

   • config_spec.backup_window_start: Backup start time (UTC):

     • hours: Hours in 24-hour format
     • minutes: Minutes
     • seconds: Seconds
     • nanos: Nanoseconds

   You can get the cluster ID with a list of clusters in the folder.

4. View the server response to make sure the request was successful.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To set the retention period for automatic backups, provide the required value in HH:MM:SS format in the --backup-retain-period-days argument of the cluster update command:

yc managed-clickhouse cluster update <cluster_name_or_ID> \
   --backup-retain-period-days=<automatic_backup_retention_period_in_days>

You can request the cluster ID and name with a list of clusters in the folder.

To set the retention period for automatic backups, use the update REST API method for the Cluster resource or the Cluster/Update gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. You can get it with a list of clusters in the folder.
• New automatic backup retention period in days in the configSpec.backupRetainPeriodDays parameter.
• List of updatable cluster configuration fields in the updateMask parameter (in this case, configSpec.backupRetainPeriodDays).

1. View the server response to make sure the request was successful.