Creating a Greenplum® cluster

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a Yandex MPP Analytics for PostgreSQL cluster:

1. Check whether the folder has any subnets for the cluster hosts:

   yc vpc subnet list
   

   If your folder has no subnets, create them in VPC.

2. View the description of the CLI command for creating a cluster:

   yc managed-greenplum cluster create --help
   

3. In this command, specify the cluster properties (our example does not use all available parameters):

   yc managed-greenplum cluster create <cluster_name> \
      --greenplum-version=<Greenplum_version> \
      --environment=<environment> \
      --network-name=<network_name> \
      --user-name=<username> \
      --user-password=<user_password> \
      --master-config resource-id=<host_class>,`
                     `disk-size=<storage_size_in_GB>,`
                     `disk-type=<network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
      --segment-config resource-id=<host_class>,`
                     `disk-size=<storage_size_in_GB>,`
                     `disk-type=<network-ssd-nonreplicated|local-ssd> \
      --zone-id=<availability_zone> \
      --subnet-id=<subnet_ID> \
      --assign-public-ip=<enable_public_access_to_cluster_hosts> \
      --security-group-ids=<list_of_security_group_IDs> \
      --deletion-protection
   

   Note

   The cluster name must be unique within a folder. It may contain Latin letters, numbers, hyphens, and underscores. The name may be up to 63 characters long.

   Where:

   • --greenplum-version: Greenplum® version, 6.19.

   • --environment: Environment:

     • PRODUCTION: For stable versions of your apps.
     • PRESTABLE: For testing purposes. The prestable environment is similar to the production environment and likewise covered by an SLA, but it is the first to get new features, improvements, and bug fixes. In the prestable environment, you can test the new versions for compatibility with your application.

   • --network-name: Network name.

   • --user-name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter, number, or underscore. It must be from 1 to 32 characters long.

   • --user-password: Password. It must be from 8 to 128 characters long.

   • --master-config and --segment-config: Master and segment host configuration:

     • resource-id: Host class.

       The segment host class and the number of segments per host affect the maximum amount of memory allocated to each Greenplum® server process. If you select a host class with small RAM and specify a large number of segments, an error may occur.

     • disk-size: Storage size in GB.

     • disk-type: Disk type:

       • network-hdd (for master hosts only)
       • network-ssd (for master hosts only)
       • local-ssd
       • network-ssd-nonreplicated

   • --zone-id: Availability zone.

   • --subnet-id: Subnet ID. You need to specify the ID if the selected availability zone has two or more subnets.

   • --assign-public-ip: Flag used if public access to the hosts is required, true or false.

   • --security-group-ids: List of security group IDs.

   • --deletion-protection: Cluster protection from accidental deletion, true or false.

     Even with deletion protection enabled, one can still connect to the cluster manually and delete the data.

4. To set the start time for the backup, provide the required value in HH:MM:SS format under --backup-window-start:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --backup-window-start=<backup_start_time>
   

5. Optionally, to create a cluster based on dedicated host groups, specify their IDs as a comma-separated list in the --master-host-group-ids and --segment-host-group-ids parameters:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --master-host-group-ids=<IDs_of_dedicated_host_groups_for_master_hosts> \
      --segment-host-group-ids=<IDs_of_dedicated_host_groups_for_segment_hosts>
   

   You can assign groups to one of the two Greenplum® host types or to both of them at once.

   You must first create a group of dedicated hosts in Yandex Compute Cloud.

   You cannot edit this setting after you create a cluster.

   If using dedicated hosts, the cluster cost is a sum of the charge for computing resources Yandex Compute Cloud and the markup Yandex MPP Analytics for PostgreSQL.

6. To set up a maintenance window (including for disabled clusters), provide the relevant value in the --maintenance-window parameter when creating your cluster:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --maintenance-window type=<maintenance_type>,`
                          `day=<day_of_week>,`
                          `hour=<hour> \
   

   Where type is the maintenance type:

   • anytime: At any time (default).
   • weekly: On a schedule. For this value, also specify the following:
     • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: Hour of day (UTC), from 1 to 24.

7. To allow access to the cluster from different services, provide the true value in the relevant parameters when creating the cluster:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --datalens-access=<allow_access_from_DataLens> \
      --yandexquery-access=<allow_access_from_Yandex_Query> \
      --websql-access=<allow_access_from_WebSQL>
   

   Available services:

   • --datalens-access: Yandex DataLens
   • --yandexquery-access: Yandex Query
   • --websql-access: Yandex WebSQL

8. To enable transferring logs to Yandex Cloud Logging, specify the following parameters when creating the cluster:

   yc managed-greenplum cluster create <cluster_name> \
      ...
      --service-account <service_account_ID> \
      --log-enabled \
      --log-command-center-enabled \
      --log-greenplum-enabled \
      --log-pooler-enabled \
      --log-folder-id <folder_ID>
   

   Where:

   • --service-account: Service account ID.

   • --log-enabled: Enables log transfer. Required for other flags responsible for transferring specific logs, e.g., --log-greenplum-enabled.

   • --log-command-center-enabled: Transferring Command Center logs.

   • --log-greenplum-enabled: Transferring Greenplum® logs.

   • --log-pooler-enabled: Transferring connection pooler logs.

   • --log-folder-id: Specify the ID of the folder whose log group you want to use.

   • --log-group-id: ID of the log group to write logs to.

     Specify either --log-folder-id or --log-group-id.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

To create a Yandex MPP Analytics for PostgreSQL cluster:

1. In the command line, navigate to the directory that contains the Terraform configuration files with the infrastructure plan. If there is no such directory, create one.

2. If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

3. Create a configuration file describing the cloud network and subnets.

   The cluster is hosted on a cloud network. If you already have a suitable network, you do not need to describe it again.

   Cluster hosts are located on the selected cloud network's subnets. If you already have suitable subnets, you do not need to describe them again.

   Below is a sample structure of a configuration file describing a single-subnet cloud network:

   resource "yandex_vpc_network" "<network_name_in_Terraform>" { name = "<network_name>" }
   resource "yandex_vpc_subnet" "<subnet_name_in_Terraform>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = yandex_vpc_network.<network_name_in_Terraform>.id
     v4_cidr_blocks = ["<subnet>"]
   }
   

4. Create a configuration file with a description of the cluster and its hosts.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_greenplum_cluster" "<cluster_name_in_Terraform>" {
     name                = "<cluster_name>"
     environment         = "<environment>"
     network_id          = yandex_vpc_network.<network_name_in_Terraform>.id
     zone                = "<availability_zone>"
     subnet_id           = yandex_vpc_subnet.<subnet_name_in_Terraform>.id
     assign_public_ip    = <enable_public_access_to_cluster_hosts>
     deletion_protection = <protect_cluster_from_deletion>
     version             = "<Greenplum_version>"
     master_host_count   = <number_of_master_hosts>
     segment_host_count  = <number_of_segment_hosts>
     segment_in_host     = <number_of_segments_per_host>
     master_subcluster {
       resources {
         resource_preset_id = "<host_class>"
         disk_size          = <storage_size_in_GB>
         disk_type_id       = "<disk_type>"
       }
     }
     segment_subcluster {
       resources {
         resource_preset_id = "<host_class>"
         disk_size          = <storage_size_in_GB>
         disk_type_id       = "<disk_type>"
       }
     }
   
     access {
       data_lens    = <allow_access_from_DataLens>
       yandex_query = <allow_access_from_Yandex_Query>
     }
   
     user_name     = "<username>"
     user_password = "<password>"
   
     security_group_ids = ["<list_of_security_group_IDs>"]
   }
   

   Where:

   • assign_public_ip: Public access to cluster hosts, true or false.

   • deletion_protection: Cluster protection from accidental deletion, true or false.

     Even with deletion protection enabled, one can still connect to the cluster manually and delete the data.

   • version: Greenplum® version.

   • master_host_count: Number of master hosts, 2.

   • segment_host_count: Number of segment hosts, between 2 and 32.

   • segment_in_host: Number of segments per host. The maximum value of this setting depends on the host class.

     The segment host class and the number of segments per host affect the maximum amount of memory allocated to each Greenplum® server process. If you select a host class with small RAM and specify a large number of segments, an error may occur.

   • access.data_lens: Access to the cluster from Yandex DataLens, true or false.

   • access.yandex_query: Access to the cluster from Yandex Query, true or false.

   For more information about the resources you can create with Terraform, see this provider article.

5. Optionally, specify dedicated host groups to place master or segment hosts on dedicated hosts:

   resource "yandex_mdb_greenplum_cluster" "<cluster_name_in_Terraform>" {
     ...
     master_host_group_ids = [<IDs_of_dedicated_host_groups_for_master_hosts>]
     segment_host_group_ids = [<IDs_of_dedicated_host_groups_for_segment_hosts>]
     ...
   }
   

   You can assign groups to one of the two Greenplum® host types or to both of them at once.

   You must first create a group of dedicated hosts in Yandex Compute Cloud.

   You cannot edit this setting after you create a cluster.

   If using dedicated hosts, the cluster cost is a sum of the charge for computing resources Yandex Compute Cloud and the markup Yandex MPP Analytics for PostgreSQL.

6. To enable transferring logs to Yandex Cloud Logging, specify the following parameters:

   resource "yandex_mdb_greenplum_cluster" "<cluster_name_in_Terraform>" {
     ...
     service_account_id="<service_account_ID>"
     logging {
       enabled                = <enable_transferring_logs>
       command_center_enabled = <transfer_Yandex_Command_Center_logs>
       greenplum_enabled      = <transfer_Greenplum®_logs>
       pooler_enabled         = <transfer_connection_pooler_logs>
       folder_id              = "<folder_ID>"
     }
   }
   

   Where:

   • service_account_id: Service account ID.

   • logging: Log transfer settings:

     • enabled: Manages log transfer, true or false. To enable parameters responsible for transferring specific logs, provide the true value.

     • command_center_enabled: Transferring Command Center logs, true or false.

     • greenplum_enabled: Transferring Greenplum® logs, true or false.

     • pooler_enabled: Transferring connection pooler logs, true or false.

     • folder_id: Specify the ID of the folder whose log group you want to use.

     • log_group_id: ID of the log group to write logs to.

       Specify either folder_id or log_group_id.

7. Make sure the Terraform configuration files are correct:

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

8. Create your cluster:

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

1. Get an IAM token for API authentication and set it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Create a file named body.json and paste the following code into it:

   {
     "folderId": "<folder_ID>",
     "name": "<cluster_name>",
     "environment": "<environment>",
     "config": {
       "version": "<Greenplum®_version>",
       "access": {
         "dataLens": <allow_access_from_DataLens>,
         "yandexQuery": <allow_access_from_Yandex_Query>
       },
       "zoneId": "<availability_zone>",
       "subnetId": "<subnet_ID>",
       "assignPublicIp": <enable_public_access_to_cluster_hosts>
     },
     "masterConfig": {
       "resources": {
         "resourcePresetId": "<host_class>",
         "diskSize": "<storage_size_in_bytes>",
         "diskTypeId": "<disk_type>"
       }
     },
     "segmentConfig": {
       "resources": {
         "resourcePresetId": "<host_class>",
         "diskSize": "<storage_size_in_bytes>",
         "diskTypeId": "<disk_type>"
       }
     },
     "masterHostCount": "<number_of_master_hosts>",
     "segmentHostCount": "<number_of_segment_hosts>",
     "segmentInHost": "<number_of_segments_per_host>",
     "userName": "<username>",
     "userPassword": "<user_password>",
     "networkId": "<network_ID>",
     "securityGroupIds": [
         "<security_group_1_ID>",
         "<security_group_2_ID>",
         ...
         "<security_group_N_ID>"
     ],
     "deletionProtection": <protect_cluster_from_deletion>,
     "configSpec": {
       "pool": {
         "mode": "<operation_mode>",
         "size": "<number_of_client_connections>",
         "clientIdleTimeout": "<client_timeout>"
       }
     },
     "cloudStorage": {
       "enable": <use_hybrid_storage>
     },
     "masterHostGroupIds": [
       "string"
     ],
     "segmentHostGroupIds": [
       "string"
     ],
     "serviceAccountId": "<service_account_ID>",
     "logging": {
       "enabled": "<enable_transferring_logs>",
       "commandCenterEnabled": "<transfer_Yandex_Command_Center_logs>",
       "greenplumEnabled": "<transfer_Greenplum®_logs>",
       "poolerEnabled": "<transfer_connection_pooler_logs>",
       "folderId": "<folder_ID>"
     }
   }
   

   Where:

   • folderId: Folder ID. You can get it with the list of folders in the cloud.

   • name: Cluster name.

   • environment: Cluster environment, PRODUCTION or PRESTABLE.

   • config: Cluster settings:

     • version: Greenplum® version.

     • access: Cluster settings for access to the following Yandex Cloud services:

       • dataLens: Yandex DataLens, true or false.
       • yandexQuery: Yandex Query, true or false.

     • zoneId: Availability zone.

     • subnetId: Subnet ID.

     • assignPublicIp: Public access to cluster hosts, true or false.

   • masterConfig.resources, segmentConfig.resources: Master and segment host configuration in the cluster:

     • resourcePresetId: Host class.
     • diskSize: Disk size, in bytes.
     • diskTypeId: Disk type.

   • masterHostCount: Number of master hosts, 1 or 2.

   • segmentHostCount: Number of segment hosts, from 2 to 32.

   • segmentInHost: Number of segments per host. The maximum value of this setting depends on the host class.

     The segment host class and the number of segments per host affect the maximum amount of memory allocated to each Greenplum® server process. If you select a host class with small RAM and specify a large number of segments, an error may occur.

   • userName: Username.

   • userPassword: User password.

   • networkId: ID of the network the cluster will be in.

   • securityGroupIds: Security group IDs.

   • deletionProtection: Cluster protection from accidental deletion, true or false.

     Even with deletion protection enabled, one can still connect to the cluster manually and delete the data.

   • configSpec.pool: Connection pooler settings:

     • mode: Operation mode, SESSION or TRANSACTION.
     • size: Maximum number of client connections.
     • clientIdleTimeout: Idle timeout for a client connection (in seconds).

   • cloudStorage.enable: Use of hybrid storage in clusters with Greenplum® 6.25 or higher. Set it to true to enable the Yandex Cloud Yezzey extension in your cluster. This extension is used to export AO and AOCO tables from disks within the Yandex MPP Analytics for PostgreSQL cluster to a cold storage in Yandex Object Storage. This way, the data will be stored in a service bucket compressed and encrypted. This is a more cost-efficient storage method.

     You cannot disable hybrid storage after you save your cluster settings.

     Note

     This feature is at the Preview stage and is free of charge.

   • masterHostGroupIds and segmentHostGroupIds: Optionally, IDs of dedicated host groups for master and segment hosts.

     You must first create a group of dedicated hosts in Yandex Compute Cloud.

     You cannot edit this setting after you create a cluster.

     If using dedicated hosts, the cluster cost is a sum of the charge for computing resources Yandex Compute Cloud and the markup Yandex MPP Analytics for PostgreSQL.

   • serviceAccountId: Service account ID.

   • logging: Settings for transferring logs to Yandex Cloud Logging:

     • enabled: Manages log transfer, true or false. To enable parameters responsible for transferring specific logs, provide the true value.

     • commandCenterEnabled: Transferring Command Center logs, true or false.

     • greenplumEnabled: Transferring Greenplum® logs, true or false.

     • poolerEnabled: Transferring connection pooler logs, true or false.

     • folderId: Specify the ID of the folder whose log group you want to use.

     • logGroupId: ID of the log group to write logs to.

       Specify either folderId or logGroupId.

3. Call the Cluster.Create method, e.g., via the following cURL request:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --header "Content-Type: application/json" \
       --url 'https://mdb.api.cloud.yandex.net/managed-greenplum/v1/clusters' \
       --data "@body.json"
   

4. Check the server response to make sure your request was successful.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Create a file named body.json and paste the following code into it:

   {
     "folder_id": "<folder_ID>",
     "name": "<cluster_name>",
     "environment": "<environment>",
     "config": {
       "version": "<Greenplum®_version>",
       "access": {
         "data_lens": <allow_access_from_DataLens>,
         "yandex_query": <allow_access_from_Yandex_Query>
       },
       "zone_id": "<availability_zone>",
       "subnet_id": "<subnet_ID>",
       "assign_public_ip": <enable_public_access_to_cluster_hosts>
     },
     "master_config": {
       "resources": {
         "resource_preset_id": "<host_class>",
         "disk_size": "<storage_size_in_bytes>",
         "disk_type_id": "<disk_type>"
       }
     },
     "segment_config": {
       "resources": {
         "resource_preset_id": "<host_class>",
         "disk_size": "<storage_size_in_bytes>",
         "disk_type_id": "<disk_type>"
       }
     },
     "master_host_count": "<number_of_master_hosts>",
     "segment_host_count": "<number_of_segment_hosts>",
     "segment_in_host": "<number_of_segments_per_host>",
     "user_name": "<username>",
     "user_password": "<user_password>",
     "network_id": "<network_ID>",
     "security_group_ids": [
         "<security_group_1_ID>",
         "<security_group_2_ID>",
         ...
         "<security_group_N_ID>"
     ],
     "deletion_protection": <protect_cluster_from_deletion>
     "config_spec": {
       "pool": {
         "mode": "<operation_mode>",
         "size": "<number_of_client_connections>",
         "client_idle_timeout": "<client_timeout>"
       }
     },
     "cloud_storage": {
       "enable": <use_hybrid_storage>
     },
     "master_host_group_ids": [
       "string"
     ],
     "segment_host_group_ids": [
       "string"
     ],
     "service_account_id": "<service_account_ID>",
     "logging": {
       "enabled": "<enable_transferring_logs>",
       "command_center_enabled": "<transfer_Yandex_Command_Center_logs>",
       "greenplum_enabled": "<transfer_Greenplum®_logs>",
       "pooler_enabled": "<transfer_connection_pooler_logs>",
       "folder_id": "<folder_ID>"
     }
   }
   

   Where:

   • folder_id: Folder ID. You can request it with the list of folders in the cloud.

   • name: Cluster name.

   • environment: Cluster environment, PRODUCTION or PRESTABLE.

   • config: Cluster settings:

     • version: Greenplum® version.

     • access: Cluster settings for access to the following Yandex Cloud services:

       • data_lens: Yandex DataLens, true or false.
       • yandex_query: Yandex Query, true or false.

     • zone_id: Availability zone.

     • subnet_id: Subnet ID.

     • assign_public_ip: Public access to cluster hosts, true or false.

   • master_config.resources, segment_config.resources: Master and segment host configuration in the cluster:

     • resource_preset_id: Host class.
     • disk_size: Disk size, in bytes.
     • disk_type_id: Disk type.

   • master_host_count: Number of master hosts, 1 or 2.

   • segment_host_count: Number of segment hosts, from 2 to 32.

   • segment_in_host: Number of segments per host. The maximum value of this setting depends on the host class.

     The segment host class and the number of segments per host affect the maximum amount of memory allocated to each Greenplum® server process. If you select a host class with small RAM and specify a large number of segments, an error may occur.

   • user_name: Username.

   • user_password: User password.

   • network_id: ID of the network the cluster will be in.

   • security_group_ids: Security group IDs.

   • deletion_protection: Cluster protection from accidental deletion, true or false.

     Even with deletion protection enabled, one can still connect to the cluster manually and delete the data.

   • config_spec.pool: Connection pooler settings:

     • mode: Operation mode, SESSION or TRANSACTION.
     • size: Maximum number of client connections.
     • client_idle_timeout: Idle timeout for a client connection (in seconds).

   • cloud_storage.enable: Use of hybrid storage in clusters with Greenplum® 6.25 or higher. Set it to true to enable the Yandex Cloud Yezzey extension in a cluster. This extension is used to export AO and AOCO tables from disks within the Yandex MPP Analytics for PostgreSQL cluster to a cold storage in Yandex Object Storage. This way, the data will be stored in a service bucket compressed and encrypted. This is a more cost-efficient storage method.

     You cannot disable hybrid storage after you save your cluster settings.

     Note

     This feature is at the Preview stage and is free of charge.

   • master_host_group_ids and segment_host_group_ids: Optionally, IDs of dedicated host groups for master and segment hosts.

     You must first create a group of dedicated hosts in Yandex Compute Cloud.

     You cannot edit this setting after you create a cluster.

     If using dedicated hosts, the cluster cost is a sum of the charge for computing resources Yandex Compute Cloud and the markup Yandex MPP Analytics for PostgreSQL.

   • service_account_id: Service account ID.

   • logging: Settings for transferring logs to Yandex Cloud Logging:

     • enabled: Manages log transfer, true or false. To enable parameters responsible for transferring specific logs, provide the true value.

     • command_center_enabled: Transferring Command Center logs, true or false.

     • greenplum_enabled: Transferring Greenplum® logs, true or false.

     • pooler_enabled: Transferring connection pooler logs, true or false.

     • folder_id: Specify the ID of the folder whose log group you want to use.

     • log_group_id: ID of the log group to write logs to.

       Specify either folder_id or log_group_id.

4. Use the ClusterService.Create call and send the following request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/greenplum/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.greenplum.v1.ClusterService.Create \
       < body.json
   

5. View the server response to make sure your request was successful.