Creating an YTsaurus cluster

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a YTsaurus cluster:

1. In the configuration file, describe the resources you are creating:

   • YTsaurus cluster: Cluster description.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_ytsaurus_cluster" "<cluster_name>" {
   
     description        = "<cluster_description>"
     name               = "<cluster_name>"
     zone_id            = yandex_vpc_subnet.<subnet_name>.zone
     subnet_id          = yandex_vpc_subnet.<subnet_name>.id
     security_group_ids = [<list_of_security_group_IDs>]
   
     spec = {
       storage = {
         hdd = {
           size_gb = 2048
           count   = 3
         }
         ssd = {
           size_gb = 465
           type    = "network-ssd-nonreplicated"
           count   = 3
         }
       }
       compute = [
         {
           preset = "c8-m32"
           disks = [
             {
               size_gb   = 93
               type      = "network-ssd-nonreplicated"
             }
           ]
           scale_policy = {
             fixed = {
               size = 3
             }
           }
         }
       ]
       tablet = {
         preset = "c8-m16"
         count  = 3
       }
       proxy = {
         http = {
           count = 2
         }
         rpc = {
           count = 2
         }
       }
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = yandex_vpc_network.<network_name>.id
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • description: Cluster description.

   • name: Cluster name.

   • zone_id: Availability zone.

   • subnet_id: Subnet ID.

   • security_group_ids: List of security group IDs.

     Warning

     You cannot edit any network settings after you create a cluster.

   • spec: Configuration of YTsaurus cluster components:

     • storage: Storage parameters:

       • hdd: HDD parameters:

         • size_gb: Disk size in GB.
         • count: Number of disks.

       • ssd: SSD parameters:

         • size_gb: Disk size in GB.
         • type: Disk type.
         • count: Number of disks.

       Warning

       You cannot change the disk type or size for a YTsaurus cluster once you create it.

     • compute: Configuration of exec nodes:

       • preset: Computing resource configuration.

       • disks: Parameters of the storage you will use to run jobs.

         • size_gb: Disk size in GB.
         • type: Disk type.

       • scale_policy.fixed.size: Number of exec nodes for the fixed scaling policy.

     • tablet: Configuration of tablet nodes:

       • preset: Computing resource configuration.
       • count: Number of tablet nodes.

       To use tablet nodes, make sure to specify the SSD storage parameters (spec.storage.ssd).

       Warning

       You cannot edit the number of tablet nodes after you create a cluster.

     • proxy: Proxy parameters:

       • http.count: Number of HTTP proxies.
       • rpc.count: Number of RPC proxies.

   For more information about the resources you can create with Terraform, see this provider guide.

2. Validate your configuration.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Create a YTsaurus cluster.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   This will create all the resources you need in the specified folder. You can check the new resources and their configuration using the management console.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Create a file named body.json and paste the following code into it:

   {
     "folderId": "<folder_ID>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "zoneId": "<availability_zone>",
     "subnetId": "<subnet_ID>",
     "securityGroupIds": [ <list_of_security_group_IDs> ],
     "spec": {
       "storage": {
         "hdd": {
           "sizeGb": "2048",
           "count": "3"
         },
         "ssd": {
           "sizeGb": "465",
           "type": "network-ssd-nonreplicated",
           "count": "3"
         }
       },
       "compute": [
         {
           "preset": "c8-m32",
           "disks": [
             {
               "type": "network-ssd-nonreplicated",
               "sizeGb": "93"
             }
           ],
           "scalePolicy": {
             "fixed": {
               "size": "3"
             }
           }
         }
       ],
       "tablet": {
         "preset": "c8-m16",
         "count": "3"
       },
       "proxy": {
         "http": {
           "count": "2"
         },
         "rpc": {
           "count": "2"
         }
       }
     }
   }
   

   Where:

   • folderId: Folder ID. You can get it with the list of folders in the cloud.

   • name: Cluster name.

   • description: Cluster description.

   • zoneId: Availability zone.

   • subnetId: Subnet ID.

   • securityGroupIds: List of security group IDs.

     Warning

     You cannot edit any network settings after you create a cluster.

   • spec: Configuration of YTsaurus cluster components:

     • storage: Storage parameters:

       • hdd: HDD parameters:

         • sizeGb: Disk size in GB.
         • count: Number of disks.

       • ssd: SSD parameters:

         • sizeGb: Disk size in GB.
         • type: Disk type.
         • count: Number of disks.

       Warning

       You cannot change the disk type or size for a YTsaurus cluster once you create it.

     • compute: Configuration of exec nodes:

       • preset: Computing resource configuration.

       • disks: Parameters of the storage you will use to run jobs.

         • type: Disk type.
         • sizeGb: Disk size in GB.

       • scalePolicy.fixed.size: Number of exec nodes for the fixed scaling policy.

     • tablet: Configuration of tablet nodes:

       • preset: Computing resource configuration.
       • count: Number of tablet nodes.

       To use tablet nodes, make sure to specify the SSD storage parameters (spec.storage.ssd).

       Warning

       You cannot edit the number of tablet nodes after you create a cluster.

     • proxy: Proxy parameters:

       • http.count: Number of HTTP proxies.
       • rpc.count: Number of RPC proxies.

3. Use the Cluster.Create method and send the following request, e.g., via cURL:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://ytsaurus.api.cloud.yandex.net/ytsaurus/v1/clusters'
       --data '@body.json'
   

4. View the server response to make sure your request was successful.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "folder_id": "<folder_ID>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "zone_id": "<availability_zone>",
     "subnet_id": "<subnet_ID>",
     "security_group_ids": [ <list_of_security_group_IDs> ],
     "spec": {
       "storage": {
         "hdd": {
           "size_gb": "2048",
           "count": "3"
         },
         "ssd": {
           "size_gb": "465",
           "type": "network-ssd-nonreplicated",
           "count": "3"
         }
       },
       "compute": [
         {
           "preset": "c8-m32",
           "disks": [
             {
               "type": "network-ssd-nonreplicated",
               "size_gb": "93"
             }
           ],
           "scale_policy": {
             "fixed": {
               "size": "3"
             }
           }
         }
       ],
       "tablet": {
         "preset": "c8-m16",
         "count": "3"
       },
       "proxy": {
         "http": {
           "count": "2"
         },
         "rpc": {
           "count": "2"
         }
       }
     }
   }
   

   Where:

   • folder_id: Folder ID. You can get it with the list of folders in the cloud.

   • name: Cluster name.

   • description: Cluster description.

   • zone_id: Availability zone.

   • subnet_id: Subnet ID.

   • security_group_ids: List of security group IDs.

     Warning

     You cannot edit any network settings after you create a cluster.

   • spec: Configuration of YTsaurus cluster components:

     • storage: Storage parameters:

       • hdd: HDD parameters:

         • size_gb: Disk size in GB.
         • count: Number of disks.

       • ssd: SSD parameters:

         • size_gb: Disk size in GB.
         • type: Disk type.
         • count: Number of disks.

       Warning

       You cannot change the disk type or size for a YTsaurus cluster once you create it.

     • compute: Configuration of exec nodes:

       • preset: Computing resource configuration.

       • disks: Parameters of the storage you will use to run jobs.

         • type: Disk type.
         • size_gb: Disk size in GB.

       • scale_policy.fixed.size: Number of exec nodes for the fixed scaling policy.

     • tablet: Configuration of tablet nodes:

       • preset: Computing resource configuration.
       • count: Number of tablet nodes.

       To use tablet nodes, make sure to specify the SSD storage parameters (spec.storage.ssd).

       Warning

       You cannot edit the number of tablet nodes after you create a cluster.

     • proxy: Proxy parameters:

       • http.count: Number of HTTP proxies.
       • rpc.count: Number of RPC proxies.

4. Use the ClusterService.Create call and send the following request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/ytsaurus/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       ytsaurus.api.cloud.yandex.net:443 \
       yandex.cloud.ytsaurus.v1.ClusterService.Create \
       < body.json
   

5. Check the server response to make sure your request was successful.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a YTsaurus cluster:

1. In the configuration file, describe the resources you are creating:

   • YTsaurus cluster: Cluster description.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_ytsaurus_cluster" "<cluster_name>" {
   
     description        = "<cluster_description>"
     name               = "<cluster_name>"
     zone_id            = yandex_vpc_subnet.<subnet_name>.zone
     subnet_id          = yandex_vpc_subnet.<subnet_name>.id
     security_group_ids = [<list_of_security_group_IDs>]
   
     labels = {
       <label_list>
     }
   
     spec = {
       storage = {
         hdd = {
           size_gb = <disk_size_in_GB>
           count   = <number_of_disks>
         }
         ssd = {
           size_gb = <disk_size_in_GB>
           type    = "<disk_type>"
           count   = <number_of_disks>
         }
       }
       compute = [
         {
           preset = "<computing_resource_configuration>"
           disks = [
             {
               size_gb   = <disk_size_in_GB>
               type      = "<disk_type>"
             }
           ]
           scale_policy = {
             fixed = {
               size = <number_of_exec_nodes>
             }
           }
           name = "default"
         }
       ]
       tablet = {
         preset = "<computing_resource_configuration>"
         count  = <number_of_tablet_nodes>
       }
       proxy = {
         http = {
           count = <number_of_HTTP_proxies>
         }
         rpc = {
           count = <number_of_RPC_proxies>
         }
       },
       odin = {
         checks_ttl = "<frequency_of_checks>"
       }
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = yandex_vpc_network.<network_name>.id
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • description: Cluster description.

   • name: Cluster name.

   • zone_id: Availability zone.

   • subnet_id: Subnet ID.

   • security_group_ids: List of security group IDs.

     Warning

     You cannot edit any network settings after you create a cluster.

   • labels: List of labels Provide labels in <key> = "<value>" format.

   • spec: Configuration of YTsaurus cluster components:

     • storage: Storage parameters:

       • hdd: HDD parameters:

         • size_gb: Disk size in GB.
         • count: Number of disks.

       • ssd: SSD parameters:

         • size_gb: Disk size in GB.
         • type: Disk type, network-ssd or network-ssd-nonreplicated.
         • count: Number of disks.

       The selected type determines the increments in which you can set your disk size when creating a cluster:

       • Network HDD and SSD storage: In increments of 1 GB.
       • Non-replicated SSD storage: In increments of 93 GB.

       An HDD is required.

       Warning

       You cannot change the disk type or size for a YTsaurus cluster once you create it.

     • compute: Configuration of exec nodes:

       • preset: Computing resource configuration.

       • disks: Parameters of the storage you will use to run jobs.

         • size_gb: Disk size in GB.
         • type: Disk type, network-hdd, network-ssd, or network-ssd-nonreplicated.

         The selected type determines the increments in which you can set your disk size when creating a cluster:

         • Network HDD and SSD storage: In increments of 1 GB.
         • Non-replicated SSD storage: In increments of 93 GB.

       • scale_policy.fixed.size: Number of exec nodes for the fixed scaling policy.

       • name: Exec pool name.

     • tablet: Configuration of tablet nodes:

       • preset: Computing resource configuration.
       • count: Number of tablet nodes.

       To use tablet nodes, make sure to specify the SSD storage parameters (spec.storage.ssd).

       Warning

       You cannot edit the number of tablet nodes after you create a cluster.

     • proxy: Proxy parameters:

       • http.count: Number of HTTP proxies.
       • rpc.count: Number of RPC proxies.

     • odin.checks_ttl: Frequency of checks by Odin, an internal monitoring tool. Provide with units of measurement: h (hours), m (minutes), s (seconds). Here is an example: 336h0m0s (2 weeks).

   For more information about the resources you can create with Terraform, see this provider guide.

2. Validate your configuration.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Create a Managed Service for YTsaurus cluster.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   This will create all the resources you need in the specified folder. You can check the new resources and their configuration using the management console.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Create a file named body.json and paste the following code into it:

   {
     "folderId": "<folder_ID>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "labels": { <label_list> },
     "zoneId": "<availability_zone>",
     "subnetId": "<subnet_ID>",
     "securityGroupIds": [ <list_of_security_group_IDs> ],
     "spec": {
       "storage": {
         "hdd": {
           "sizeGb": "<disk_size_in_GB>",
           "count": "<number_of_disks>"
         },
         "ssd": {
           "sizeGb": "<disk_size_in_GB>",
           "type": "<disk_type>",
           "count": "<number_of_disks>"
         }
       },
       "compute": [
         {
           "preset": "<computing_resource_configuration>",
           "disks": [
             {
               "type": "<disk_type>",
               "sizeGb": "<disk_size_in_GB>"
             }
           ],
           "scalePolicy": {
             "fixed": {
               "size": "<number_of_exec_nodes>"
             }
           },
           "name": "default"
         }
       ],
       "tablet": {
         "preset": "<computing_resource_configuration>",
         "count": "<number_of_tablet_nodes>"
       },
       "proxy": {
         "http": {
           "count": "<number_of_HTTP_proxies>"
         },
         "rpc": {
           "count": "<number_of_RPC_proxies>"
         }
       },
       "odin": {
         "checksTtl": "<frequency_of_checks_in_seconds>"
       }
     }
   }
   

   Where:

   • folderId: Folder ID. You can get it with the list of folders in the cloud.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels provided in "<key>": "<value>" format.

   • zoneId: Availability zone.

   • subnetId: Subnet ID.

   • securityGroupIds: List of security group IDs.

     Warning

     You cannot edit any network settings after you create a cluster.

   • spec: Configuration of YTsaurus cluster components:

     • storage: Storage parameters:

       • hdd: HDD parameters:

         • sizeGb: Disk size in GB.
         • count: Number of disks.

       • ssd: SSD parameters:

         • sizeGb: Disk size in GB.
         • type: Disk type, network-ssd or network-ssd-nonreplicated.
         • count: Number of disks.

       The selected type determines the increments in which you can set your disk size when creating a cluster:

       • Network HDD and SSD storage: In increments of 1 GB.
       • Non-replicated SSD storage: In increments of 93 GB.

       An HDD is required.

       Warning

       You cannot change the disk type or size for a YTsaurus cluster once you create it.

     • compute: Configuration of exec nodes:

       • preset: Computing resource configuration.

       • disks: Parameters of the storage you will use to run jobs.

         • type: Disk type, network-hdd, network-ssd, or network-ssd-nonreplicated.
         • sizeGb: Disk size in GB.

         The selected type determines the increments in which you can set your disk size when creating a cluster:

         • Network HDD and SSD storage: In increments of 1 GB.
         • Non-replicated SSD storage: In increments of 93 GB.

       • scalePolicy.fixed.size: Number of exec nodes for the fixed scaling policy.

       • name: Exec pool name.

     • tablet: Configuration of tablet nodes:

       • preset: Computing resource configuration.
       • count: Number of tablet nodes.

       To use tablet nodes, make sure to specify the SSD storage parameters (spec.storage.ssd).

       Warning

       You cannot edit the number of tablet nodes after you create a cluster.

     • proxy: Proxy parameters:

       • http.count: Number of HTTP proxies.
       • rpc.count: Number of RPC proxies.

     • odin.checksTtl: Frequency of checks by Odin, an internal monitoring tool. Specify it in seconds (s), e.g., 1209600s (2 weeks).

3. Use the Cluster.Create method and send the following request, e.g., via cURL:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://ytsaurus.api.cloud.yandex.net/ytsaurus/v1/clusters' \
       --data '@body.json'
   

4. View the server response to make sure your request was successful.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "folder_id": "<folder_ID>",
     "name": "<cluster_name>",
     "description": "<cluster_description>",
     "labels": { <label_list> },
     "zone_id": "<availability_zone>",
     "subnet_id": "<subnet_ID>",
     "security_group_ids": [ <list_of_security_group_IDs> ],
     "spec": {
       "storage": {
         "hdd": {
           "size_gb": "<disk_size_in_GB>",
           "count": "<number_of_disks>"
         },
         "ssd": {
           "size_gb": "<disk_size_in_GB>",
           "type": "<disk_type>",
           "count": "<number_of_disks>"
         }
       },
       "compute": [
         {
           "preset": "<computing_resource_configuration>",
           "disks": [
             {
               "type": "<disk_type>",
               "size_gb": "<disk_size_in_GB>"
             }
           ],
           "scale_policy": {
             "fixed": {
               "size": "<number_of_exec_nodes>"
             }
           },
           "name": "default"
         }
       ],
       "tablet": {
         "preset": "<computing_resource_configuration>",
         "count": "<number_of_tablet_nodes>"
       },
       "proxy": {
         "http": {
           "count": "<number_of_HTTP_proxies>"
         },
         "rpc": {
           "count": "<number_of_RPC_proxies>"
         }
       },
       "odin": {
         "checks_ttl": "<frequency_of_checks_in_seconds>"
       }
     }
   }
   

   Where:

   • folderId: Folder ID. You can get it with the list of folders in the cloud.

   • name: Cluster name.

   • description: Cluster description.

   • labels: List of labels provided in "<key>": "<value>" format.

   • zone_id: Availability zone.

   • subnet_id: Subnet ID.

   • security_group_ids: List of security group IDs.

     Warning

     You cannot edit any network settings after you create a cluster.

   • spec: Configuration of YTsaurus cluster components:

     • storage: Storage parameters:

       • hdd: HDD parameters:

         • size_gb: Disk size in GB.
         • count: Number of disks.

       • ssd: SSD parameters:

         • size_gb: Disk size in GB.
         • type: Disk type, network-ssd or network-ssd-nonreplicated.
         • count: Number of disks.

       The selected type determines the increments in which you can set your disk size when creating a cluster:

       • Network HDD and SSD storage: In increments of 1 GB.
       • Non-replicated SSD storage: In increments of 93 GB.

       An HDD is required.

       Warning

       You cannot change the disk type or size for a YTsaurus cluster once you create it.

     • compute: Configuration of exec nodes:

       • preset: Computing resource configuration.

       • disks: Parameters of the storage you will use to run jobs.

         • type: Disk type, network-hdd, network-ssd, or network-ssd-nonreplicated.
         • size_gb: Disk size in GB.

         The selected type determines the increments in which you can set your disk size when creating a cluster:

         • Network HDD and SSD storage: In increments of 1 GB.
         • Non-replicated SSD storage: In increments of 93 GB.

       • scale_policy.fixed.size: Number of exec nodes for the fixed scaling policy.

       • name: Exec pool name.

     • tablet: Configuration of tablet nodes:

       • preset: Computing resource configuration.
       • count: Number of tablet nodes.

       To use tablet nodes, make sure to specify the SSD storage parameters (spec.storage.ssd).

       Warning

       You cannot edit the number of tablet nodes after you create a cluster.

     • proxy: Proxy parameters:

       • http.count: Number of HTTP proxies.
       • rpc.count: Number of RPC proxies.

     • odin.checks_ttl: Frequency of checks by Odin, an internal monitoring tool. Specify it in seconds (s), e.g., 1209600s (2 weeks).

4. Use the ClusterService.Create call and send the following request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/ytsaurus/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       ytsaurus.api.cloud.yandex.net:443 \
       yandex.cloud.ytsaurus.v1.ClusterService.Create \
       < body.json
   

5. Check the server response to make sure your request was successful.