Transferring Managed Service for Greenplum® cluster logs to Yandex Cloud Logging

Written by

Updated at May 28, 2025

Getting started
Transferring data to the default log group
Sending data to a custom log group

You can set up regular collection of Managed Service for Greenplum® cluster performance logs. Logs will be delivered to a log group in Cloud Logging. You can choose between these two types of log groups:

Getting started

Create a service account with the logging.writer role.

Transferring data to the default log group

Create a Managed Service for Greenplum® cluster with active logging and a service account created earlier:
CLI

Terraform

API
```
yc managed-greenplum cluster create <cluster_name> \
   --greenplum-version=6.25 \
   --environment=PRODUCTION \
   --network-name=<cluster_network_name> \
   --user-name=<VM_user_name> \
   --user-password=<cluster_user_password> \
   --master-config resource-id=s3-c8-m32,`
                  `disk-size=10,`
                  `disk-type=network-hdd \
   --segment-config resource-id=s3-c8-m32,`
                  `disk-size=93,`
                  `disk-type=network-ssd-nonreplicated \
   --zone-id=<availability_zone_ID> \
   --subnet-id=<cluster_subnet_ID> \
   --assign-public-ip=true \
   --security-group-ids=<cluster_security_group_IDs> \
   --service-account <cluster_service_account_ID> \
   --log-enabled \
   --log-greenplum-enabled \
   --log-command-center-enabled \
   --log-folder-id <folder_ID>
```
Where:
- --service_account_id: ID of the service account you created earlier.
- --log-greenplum-enabled: Transferring Greenplum® logs.
  
  You can set the Greenplum® logging level in the Log min messages parameter. Learn more about Greenplum® advanced settings.
- --log-command-center-enabled: Transferring Yandex Command Center logs.
  
  Command Center logs are trasferred in full; you cannot change the logging level.
- --log-folder-id: ID of the folder the Managed Service for Greenplum® cluster was created in.
1. If you do not have Terraform yet, install it.
2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.
4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
5. Specify the following parameters in the configuration file with the cluster description:
  resource "yandex_mdb_greenplum_cluster" "gp-cluster" { name = "<cluster_name>" environment = "PRODUCTION" network_id = "<network_ID>" zone = "<availability_zone_ID>" subnet_id = "<cluster_subnet_ID>" assign_public_ip = true version = "6.25" master_host_count = 2 segment_host_count = 2 segment_in_host = 2 service_account_id = "<service_account_ID>" master_subcluster { resources { resource_preset_id = "s3-c8-m32" # 8 vCPU, 32 GB RAM disk_size = 10 # GB disk_type_id = "network-hdd" } } segment_subcluster { resources { resource_preset_id = "s3-c8-m32" # 8 vCPU, 32 GB RAM disk_size = 93 # GB disk_type_id = "network-ssd-nonreplicated" } } user_name = "<VM_user_name>" user_password = "<cluster_user_password>" security_group_ids = ["<cluster_security_group_IDs>"] logging { enabled = true greenplum_enabled = true command_center_enabled = true folder_id = <folder_ID> } }
  Where:
  - service_account_id: ID of the service account you created earlier.
  - greenplum-enabled: Transferring Greenplum® logs.
    
    You can set the Greenplum® logging level in the Log min messages parameter. Learn more about Greenplum® advanced settings.
  - command-center-enabled: Transferring Yandex Command Center logs.
    
    Command Center logs are trasferred in full; you cannot change the logging level.
  - folder-id: Specify the ID of the folder whose log group you want to use.
6. Make sure the Terraform configuration files are correct using this command:
  terraform validate
  If there are any errors in the configuration files, Terraform will point them out.
7. Create the required infrastructure:
  1. Run this command to view the planned changes:
    
    terraform plan
    
    If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
  2. If everything looks correct, apply the changes:
    
    Run this command:
    
    terraform apply
    
    Confirm updating the resources.
    
    Wait for the operation to complete.
Specify the following parameters in the body of the cluster create request (Cluster.Create in the REST API or ClusterService.Create in the gRPC API):
```
{
  ...
  "serviceAccountId": "<service_account_ID>",
  "logging": {
    "enabled": "true",
    "greenplumEnabled": "true",
    "commandCenterEnabled": "true",
    "folderId": "<folder_ID>"
  }
  ...
}
```
Where:
- serviceAccountId: ID of the service account you created earlier.
- greenplumEnabled: Transferring Greenplum® logs.
  
  You can set the Greenplum® logging level in the Log min messages parameter. Learn more about Greenplum® advanced settings.
- commandCenterEnabled: Transferring Yandex Command Center logs.
  
  Command Center logs are trasferred in full; you cannot change the logging level.
- folderId: Specify the ID of the folder whose log group you want to use.

Test the transfer of cluster logs to the log group.

Management console

CLI

API

In the management console, go to the relevant folder.
Select Cloud Logging.
Click the row with the default log group.

The page that opens will show the log group records.

To view the records in JSON format, run this command:

yc logging read --group-name=default --format=json

Result:

[
  {
    "uid": "22:********:15",
    "resource": {
      "type": "mdb.greenplum.cluster",
      "id": "c9qao2tkhbdt********"
    },
    "timestamp": "2025-04-25T07:50:21.871157Z",
    "ingested_at": "2025-04-25T07:50:24.553Z",
    "saved_at": "2025-04-25T07:50:26.125189Z",
    "level": "INFO",
    "message": "2025-04-25 10:50:20.209664 MSK,\"monitor\",\"postgres\",p29034,th1258748352,\"localhost\",\"47384\",2025-04-25 07:38:20 MSK,0,con40,cmd20190,seg-1,,,,sx1,\"LOG\",\"00000\",\"statement: \n\t\treset log_min_messages\n\t\",,,,,,,0,,\"postgres.c\",1664,",
    "json_payload": {
      "cid": "c9qao2tkhbdt********",
      "gp_host_type": "master",
      "gp_preferred_role": "primary",
      "hostname": "rc1d-jds7okr4********.mdb.yandexcloud.net",
      "segment_id": "-1",
      "username": "monitor"
    },
    "stream_name": "greenplum-logs-master-primary"
  }
]

To view log group records, use the Cluster.ListLogs REST API method or the LogReadingService/Read gRPC API call.

For more information, see Reading records.

Sending data to a custom log group

Create a log group named greenplum-log-group.
Create a Managed Service for Greenplum® cluster with active logging and a service account created earlier:
CLI

Terraform

API
```
yc managed-greenplum cluster create <cluster_name> \
   --greenplum-version=6.25 \
   --environment=PRODUCTION \
   --network-name=<cluster_network_name> \
   --user-name=<VM_user_name> \
   --user-password=<cluster_user_password> \
   --master-config resource-id=s3-c8-m32,`
                  `disk-size=10,`
                  `disk-type=network-hdd \
   --segment-config resource-id=s3-c8-m32,`
                  `disk-size=93,`
                  `disk-type=network-ssd-nonreplicated \
   --zone-id=<availability_zone_ID> \
   --subnet-id=<cluster_subnet_ID> \
   --assign-public-ip=true \
   --security-group-ids=<cluster_security_group_IDs> \
   --service-account <cluster_service_account_ID> \
   --log-enabled \
   --log-greenplum-enabled \
   --log-command-center-enabled \
   --log-folder-id <folder_ID>
   --log-group-id <log_group_ID>
```
Where:
- --service_account_id: ID of the service account you created earlier.
- --log-greenplum-enabled: Transferring Greenplum® logs.
  
  You can set the Greenplum® logging level in the Log min messages parameter. Learn more about Greenplum® advanced settings.
- --log-command-center-enabled: Transferring Yandex Command Center logs.
  
  Command Center logs are trasferred in full; you cannot change the logging level.
- --log-group-id: ID of the log group to write logs to.
1. If you do not have Terraform yet, install it.
2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.
4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
5. Specify the following parameters in the configuration file with the cluster description:
  resource "yandex_mdb_greenplum_cluster" "gp-cluster" { name = "<cluster_name>" environment = "PRODUCTION" network_id = "<network_ID>" zone = "<availability_zone_ID>" subnet_id = "<cluster_subnet_ID>" assign_public_ip = true version = "6.25" master_host_count = 2 segment_host_count = 2 segment_in_host = 2 service_account_id = "<service_account_ID>" master_subcluster { resources { resource_preset_id = "s3-c8-m32" # 8 vCPU, 32 GB RAM disk_size = 10 # GB disk_type_id = "network-hdd" } } segment_subcluster { resources { resource_preset_id = "s3-c8-m32" # 8 vCPU, 32 GB RAM disk_size = 93 # GB disk_type_id = "network-ssd-nonreplicated" } } user_name = "<VM_user_name>" user_password = "<cluster_user_password>" security_group_ids = ["<cluster_security_group_IDs>"] logging { enabled = true greenplum_enabled = true command_center_enabled = true log_group_id = <log_group_ID> } }
  Where:
  - service_account_id: ID of the service account you created earlier.
  - greenplum-enabled: Transferring Greenplum® logs.
    
    You can set the Greenplum® logging level in the Log min messages parameter. Learn more about Greenplum® advanced settings.
  - command-center-enabled: Transferring Yandex Command Center logs.
    
    Command Center logs are trasferred in full; you cannot change the logging level.
  - log_group_id: ID of the log group to write logs to.
6. Make sure the Terraform configuration files are correct using this command:
  terraform validate
  If there are any errors in the configuration files, Terraform will point them out.
7. Create the required infrastructure:
  1. Run this command to view the planned changes:
    
    terraform plan
    
    If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
  2. If everything looks correct, apply the changes:
    
    Run this command:
    
    terraform apply
    
    Confirm updating the resources.
    
    Wait for the operation to complete.
Specify the following parameters in the body of the cluster create request (Cluster.Create in the REST API or ClusterService.Create in the gRPC API):
```
{
  ...
  "serviceAccountId": "<service_account_ID>",
  "logging": {
    "enabled": "true",
    "greenplumEnabled": "true",
    "commandCenterEnabled": "true",
    "logGroupId": "<log_group_ID>"
  }
  ...
}
```
Where:
- serviceAccountId: ID of the service account you created earlier.
- greenplumEnabled: Transferring Greenplum® logs.
  
  You can set the Greenplum® logging level in the Log min messages parameter. Learn more about Greenplum® advanced settings.
- commandCenterEnabled: Transferring Yandex Command Center logs.
  
  Command Center logs are trasferred in full; you cannot change the logging level.
- logGroupId: Log group ID.