Creating a Greenplum® cluster

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a Managed Service for Greenplum® cluster:

1. Check whether the folder has any subnets for the cluster hosts:

   yc vpc subnet list
   

   If there are no subnets in the folder, create the required subnets in VPC.

2. View the description of the create cluster CLI command:

   yc managed-greenplum cluster create --help
   

3. Specify cluster parameters in the create command (the list of supported parameters in the example is not exhaustive):

   yc managed-greenplum cluster create <cluster_name> \
      --greenplum-version=<Greenplum_version> \
      --environment=<environment> \
      --network-name=<network_name> \
      --user-name=<username> \
      --user-password=<user_password> \
      --master-config resource-id=<host_class>,`
                     `disk-size=<storage_size_in_GB>,`
                     `disk-type=<network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
      --segment-config resource-id=<host_class>,`
                     `disk-size=<storage_size_in_GB>,`
                     `disk-type=<network-ssd-nonreplicated|local-ssd> \
      --zone-id=<availability_zone> \
      --subnet-id=<subnet_ID> \
      --assign-public-ip=<public_access_to_hosts> \
      --security-group-ids=<list_of_security_group_IDs> \
      --deletion-protection
   

   Note

   The cluster name must be unique within a folder. It may contain Latin letters, numbers, hyphens, and underscores. The name may be up to 63 characters long.

   Where:

   • --greenplum-version: Greenplum® version, 6.19.

   • --environment: Environment:

     • PRODUCTION: For stable versions of your apps.
     • PRESTABLE: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.

   • --network-name: Network name.

   • --user-name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter, number, or underscore. It must be from 1 to 32 characters long.

   • --user-password: Password. It must be from 8 to 128 characters long.

   • --master-config and --segment-config: Master and segment host configuration:

     • resource-id: Host class.
     • disk-size: Storage size in GB.
     • disk-type: Disk type:
       • network-hdd (for master hosts only)
       • network-ssd (for master hosts only)
       • local-ssd
       • network-ssd-nonreplicated.

   • --zone-id: Availability zone.

   • --subnet-id: Subnet ID. You need to specify the ID if the selected availability zone has two or more subnets.

   • --assign-public-ip: Flag used if public access to the hosts is required, true or false.

   • --security-group-ids: List of security group IDs.

   • --deletion-protection: Cluster deletion protection.

     Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.

4. To set the start time for the backup, provide the required value in HH:MM:SS format under --backup-window-start:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --backup-window-start=<backup_start_time>
   

5. To create a cluster based on dedicated host groups, specify their IDs as a comma-separated list in the --host-group-ids parameter:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --host-group-ids=<dedicated_host_group_IDs>
   

   Alert

   You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

6. To set up a maintenance window (including for disabled clusters), provide the relevant value in the --maintenance-window parameter when creating your cluster:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --maintenance-window type=<maintenance_type>,`
                          `day=<day_of_week>,`
                          `hour=<hour> \
   

   Where type is the maintenance type:

   • anytime (default): Any time.
   • weekly: On a schedule. If setting this value, specify the day of week and the hour:
     • day: Day of week in DDD format: MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: Hour (UTC) in HH format: 1 to 24.

7. To enable cluster access from different services, provide the true value in the relevant parameters when creating the cluster:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --datalens-access=<access_from_DataLens> \
      --yandexquery-access=<access_from_Yandex_Query>
   

   Available services:

   • --datalens-access: Yandex DataLens
   • --yandexquery-access: Yandex Query

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the documentation on the Terraform website or mirror website.

To create a Managed Service for Greenplum® cluster:

1. Using the command line, navigate to the folder that will contain the Terraform configuration files with an infrastructure plan. Create the directory if it does not exist.

2. If you don't have Terraform, install it and configure the Yandex Cloud provider.

3. Create a configuration file describing the cloud network and subnets.

   The cluster is hosted on a cloud network. If you already have a suitable network, you do not need to describe it again.

   Cluster hosts are located on subnets of the selected cloud network. If you already have suitable subnets, you do not need to describe them again.

   Example structure of a configuration file that describes a cloud network with a single subnet:

   resource "yandex_vpc_network" "<network_name_in_Terraform>" { name = "<network_name>" }
   resource "yandex_vpc_subnet" "<subnet_name_in_Terraform>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = yandex_vpc_network.<network_name_in_Terraform>.id
     v4_cidr_blocks = ["<subnet>"]
   }
   

4. Create a configuration file with a description of the cluster and its hosts.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_greenplum_cluster" "<cluster_name_in_Terraform>" {
     name                = "<cluster_name>"
     environment         = "<environment>"
     network_id          = yandex_vpc_network.<network_name_in_Terraform>.id
     zone                = "<availability_zone>"
     subnet_id           = yandex_vpc_subnet.<subnet_name_in_Terraform>.id
     assign_public_ip    = <public_access_to_cluster_hosts>
     deletion_protection = <cluster_deletion_protection>
     version             = "<Greenplum_version>"
     master_host_count   = <number_of_master_hosts>
     segment_host_count  = <number_of_segment_hosts>
     segment_in_host     = <number_of_segments_per_host>
     master_subcluster {
       resources {
         resource_preset_id = "<host_class>"
         disk_size          = <storage_size_in_GB>
         disk_type_id       = "<disk_type>"
       }
     }
     segment_subcluster {
       resources {
         resource_preset_id = "<host_class>"
         disk_size          = <storage_size_in_GB>
         disk_type_id       = "<disk_type>"
       }
     }
   
     access {
       data_lens    = <access_from_DataLens>
       yandex_query = <access_from_Yandex_Query>
     }
   
     user_name     = "<username>"
     user_password = "<password>"
   
     security_group_ids = ["<list_of_security_group_IDs>"]
   }
   

   Where:

   • assign_public_ip: Public access to cluster hosts, true or false.

   • deletion_protection: Cluster deletion protection, true or false.

     Enabled cluster deletion protection will not prevent a manual connection with the purpose to delete database contents.

   • version: Greenplum® version.

   • master_host_count: Number of master hosts, 2.

   • segment_host_count: Number of segment hosts, between 2 and 32.

   • segment_in_host: Number of segments per host. The maximum value of this parameter depends on the host class.

   • access.data_lens: Access to the cluster from Yandex DataLens, true or false.

   • access.yandex_query: Access to the cluster from Yandex Query, true or false.

   For more information about the resources you can create with Terraform, see the provider documentation.

5. Check that the Terraform configuration files are correct:

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

6. Create a cluster:

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

   All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Create a file named body.json and add the following contents to it:

   {
     "folderId": "<folder_ID>",
     "name": "<cluster_name>",
     "environment": "<environment>",
     "config": {
       "version": "<Greenplum®_version>",
       "access": {
         "dataLens": <access_from_DataLens>,
         "yandexQuery": <access_from_Yandex_Query>
       },
       "zoneId": "<availability_zone>",
       "subnetId": "<subnet_ID>",
       "assignPublicIp": <public_access_to_cluster_hosts>
     },
     "masterConfig": {
       "resources": {
         "resourcePresetId": "<host_class>",
         "diskSize": "<storage_size_in_bytes>",
         "diskTypeId": "<disk_type>"
       }
     },
     "segmentConfig": {
       "resources": {
         "resourcePresetId": "<host_class>",
         "diskSize": "<storage_size_in_bytes>",
         "diskTypeId": "<disk_type>"
       }
     },
     "masterHostCount": "<number_of_master_hosts>",
     "segmentHostCount": "<number_of_segment_hosts>",
     "segmentInHost": "<number_of_segments_per_host>",
     "userName": "<username>",
     "userPassword": "<user_password>",
     "networkId": "<network_ID>",
     "securityGroupIds": [
         "<security_group_1_ID>",
         "<security_group_2_ID>",
         ...
         "<security_group_N_ID>"
     ],
     "deletionProtection": <cluster_deletion_protection>,
     "configSpec": {
       "pool": {
         "mode": "<operation_mode>",
         "size": "<number_of_client_connections>",
         "clientIdleTimeout": "<client_timeout>"
       }
     },
     "cloudStorage": {
       "enable": <hybrid_storage_use>
     }
   }
   

   Where:

   • folderId: Folder ID. You can request it with a list of folders in the cloud.

   • name: Cluster name.

   • environment: Cluster environment, PRODUCTION or PRESTABLE.

   • config: Cluster settings:

     • version: Greenplum® version.

     • access: Cluster settings for access to the following Yandex Cloud services:

       • dataLens: Yandex DataLens, true or false.
       • yandexQuery: Yandex Query, true or false.

     • zoneId: Availability zone.

     • subnetId: Subnet ID.

     • assignPublicIp: Public access to cluster hosts, true or false.

     • masterConfig.resources, segmentConfig.resources: Master and segment host configuration in the cluster:

       • resourcePresetId: Host class.
       • diskSize: Disk size in bytes.
       • diskTypeId: Disk type.

   • masterHostCount: Number of master hosts, 1 or 2.

   • segmentHostCount: Number of segment hosts, from 2 to 32.

   • segmentInHost: Number of segments per host. The maximum value of this parameter depends on the host class.

   • userName: Username.

   • userPassword: User password.

   • networkId: ID of the network the cluster will be in.

   • securityGroupIds: Security group IDs.

   • deletionProtection: Cluster deletion protection, true or false.

     Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.

   • configSpec.pool: Connection pooler settings:

     • mode: Operation mode, SESSION or TRANSACTION.
     • size: Maximum number of client connections.
     • clientIdleTimeout: Idle timeout for a client connection (in ms).

   • cloudStorage.enable: Use of hybrid storage in clusters with Greenplum® 6.25 or higher. Set it to true to enable the Yandex Cloud Yezzey extension in a cluster. This extension is used to export AO and AOCO tables from disks within the Managed Service for Greenplum® cluster to a cold storage in Yandex Object Storage. This way, the data will be stored in a service bucket in a compressed and encrypted form. This is a more cost-efficient storage method.

     You cannot disable hybrid storage after you save your cluster settings.

     Note

     This feature is at the Preview stage and is free of charge.

3. Use the Cluster.Create method and make a request, e.g., via cURL:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --header "Content-Type: application/json" \
       --url 'https://mdb.api.cloud.yandex.net/managed-greenplum/v1/clusters' \
       --data "@body.json"
   

4. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Create a file named body.json and add the following contents to it:

   {
     "folder_id": "<folder_ID>",
     "name": "<cluster_name>",
     "environment": "<environment>",
     "config": {
       "version": "<Greenplum®_version>",
       "access": {
         "data_lens": <access_from_DataLens>,
         "yandex_query": <access_from_Yandex_Query>
       },
       "zone_id": "<availability_zone>",
       "subnet_id": "<subnet_ID>",
       "assign_public_ip": <public_access_to_cluster_hosts>
     },
     "master_config": {
       "resources": {
         "resource_preset_id": "<host_class>",
         "disk_size": "<storage_size_in_bytes>",
         "disk_type_id": "<disk_type>"
       }
     },
     "segment_config": {
       "resources": {
         "resource_preset_id": "<host_class>",
         "disk_size": "<storage_size_in_bytes>",
         "disk_type_id": "<disk_type>"
       }
     },
     "master_host_count": "<number_of_master_hosts>",
     "segment_host_count": "<number_of_segment_hosts>",
     "segment_in_host": "<number_of_segments_per_host>",
     "user_name": "<username>",
     "user_password": "<user_password>",
     "network_id": "<network_ID>",
     "security_group_ids": [
         "<security_group_1_ID>",
         "<security_group_2_ID>",
         ...
         "<security_group_N_ID>"
     ],
     "deletion_protection": <cluster_deletion_protection>
     "config_spec": {
       "pool": {
         "mode": "<operation_mode>",
         "size": "<number_of_client_connections>",
         "client_idle_timeout": "<client_timeout>"
       }
     },
     "cloud_storage": {
       "enable": <hybrid_storage_use>
     }
   }
   

   Where:

   • folder_id: Folder ID. You can request it with a list of folders in the cloud.

   • name: Cluster name.

   • environment: Cluster environment, PRODUCTION or PRESTABLE.

   • config: Cluster settings:

     • version: Greenplum® version.

     • access: Cluster settings for access to the following Yandex Cloud services:

       • data_lens: Yandex DataLens, true or false.
       • yandex_query: Yandex Query, true or false.

     • zone_id: Availability zone.

     • subnet_id: Subnet ID.

     • assign_public_ip: Public access to cluster hosts, true or false.

     • master_config.resources, segment_config.resources: Master and segment host configuration in the cluster:

       • resource_preset_id: Host class.
       • disk_size: Disk size in bytes.
       • disk_type_id: Disk type.

   • master_host_count: Number of master hosts, 1 or 2.

   • segment_host_count: Number of segment hosts, from 2 to 32.

   • segment_in_host: Number of segments per host. The maximum value of this parameter depends on the host class.

   • user_name: Username.

   • user_password: User password.

   • network_id: ID of the network the cluster will be in.

   • security_group_ids: Security group IDs.

   • deletion_protection: Cluster deletion protection, true or false.

     Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.

   • config_spec.pool: Connection pooler settings:

     • mode: Operation mode, SESSION or TRANSACTION.
     • size: Maximum number of client connections.
     • client_idle_timeout: Idle timeout for a client connection (in ms).

   • cloud_storage.enable: Use of hybrid storage in clusters with Greenplum® 6.25 or higher. Set it to true to enable the Yandex Cloud Yezzey extension in a cluster. This extension is used to export AO and AOCO tables from disks within the Managed Service for Greenplum® cluster to a cold storage in Yandex Object Storage. This way, the data will be stored in a service bucket in a compressed and encrypted form. This is a more cost-efficient storage method.

     You cannot disable hybrid storage after you save your cluster settings.

     Note

     This feature is at the Preview stage and is free of charge.

4. Use the ClusterService.Create call and make a request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/greenplum/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.greenplum.v1.ClusterService.Create \
       < body.json
   

5. View the server response to make sure the request was successful.