Creating a OpenSearch cluster

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a Managed Service for OpenSearch cluster:

1. View the description of the create cluster CLI command:

   yc managed-opensearch cluster create --help
   

2. Specify cluster parameters in the create command (the list of supported parameters in the example is not exhaustive):

   yc managed-opensearch cluster create \
      --name <cluster_name> \
      --description <cluster_description> \
      --labels <labels> \
      --environment <environment:_production_or_prestable> \
      --network-name <network_name> \
      --security-group-ids <security_group_IDs> \
      --service-account-name <service_account_name> \
      --delete-protection \
      --maintenance schedule=<maintenance_type>,`
                   `weekday=<day_of_week>,`
                   `hour=<hour> \
      --version <OpenSearch_version> \
      --read-admin-password \
      --data-transfer-access=<true_or_false> \
      --serverless-access=<true_or_false> \
      --plugins <OpenSearch_plugins> \
      --advanced-params <additional_parameters> \
      --opensearch-node-group name=<OpenSearch_host_group_name>,`
                             `resource-preset-id=<host_class>,`
                             `disk-size=<disk_size_in_bytes>,`
                             `disk-type-id=<network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd>,`
                             `hosts-count=<number_of_hosts_in_group>,`
                             `zone-ids=<availability_zones>,`
                             `subnet-names=<subnet_names>,`
                             `assign-public-ip=<assign_public_address:_true_or_false>,`
                             `roles=<host_roles> \
      --dashboards-node-group name=<Dashboards_host_group_name>,`
                             `resource-preset-id=<host_class>,`
                             `disk-size=<disk_size_in_bytes>,`
                             `disk-type-id=<network-ssd>,`
                             `hosts-count=<number_of_hosts_in_group>,`
                             `zone-ids=<availability_zones>,`
                             `subnet-names=<subnet_names>,`
                             `assign-public-ip=<assign_public_address:_true_or_false>
   

   Where:

   • --labels: Yandex Cloud labels labels in <key>=<value> format. You can use them to logically separate resources.

   • --environment: Environment:

     • production: For stable versions of your apps.
     • prestable: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.

   • --service-account-name: Name of the service account for access to Yandex Object Storage as a repository of OpenSearch snapshots. For more information on service accounts, see the Yandex Identity and Access Management documentation.

   • --delete-protection: Cluster protection from accidental deletion by a user. With deletion protection enabled, you still can connect to a cluster manually to delete data.

   • --maintenance: Maintenance window settings:

     • To allow maintenance at any time, do not specify the --maintenance parameter in the command (default configuration) or specify --maintenance schedule=anytime.
     • To specify the preferred start time for maintenance, specify this parameter in the command: --maintenance schedule=weekly,weekday=<day_of_week>,hour=<hour_in_UTC>. In this case, maintenance will take place every week on a specified day at a specified time.

     Both enabled and disabled clusters undergo maintenance. Maintenance may involve such operations as applying patches or updating DBMS's.

   • --read-admin-password: admin user password. If you specify this parameter in the command, it will prompt you to enter a password.

   • --serverless-access: Access from Yandex Serverless Containers, true or false.

   • --plugins: OpenSearch plugins to install in the cluster.

   • --advanced-params: Additional cluster parameters. The possible values are:

     • max-clause-count: Maximum allowed number of boolean clauses per query. For more information, see the OpenSearch documentation.
     • fielddata-cache-size: JVM heap size allocated for the fielddata data structure. You can specify either an absolute value or percentage, e.g., 512mb or 50%. For more information, see the OpenSearch documentation.
     • reindex-remote-whitelist: List of remote hosts whose indexes contain documents to copy for reindexing. Specify the parameter value as <host_address>:<port>. If you need to specify more than one host, list values separated by commas. For more information, see the OpenSearch documentation.

   • --opensearch-node-group: OpenSearch host group configuration, where:

     • --resource-preset-id: Host class that defines the configuration of virtual machines the OpenSearch nodes will be deployed on. All available options are listed under Host classes.

     • disk-size: Disk size in bytes. Minimum and maximum values depend on the selected host class.

     • disk-type-id: Disk type.

     • roles: Host roles. The possible values include:

       • data: Assigns the DATA role only.
       • manager: Assigns the MANAGER role only.
       • data+manager or manager+data: Assigns both roles.

       Tip

       We do not recommend enabling public access to hosts with the MANAGER role, as this is not secure.

   • --dashboards-node-group: Dashboards host group configuration. It is configured in the same way as the OpenSearch host group, except for the host roles. You do not need to configure any roles for the Dashboards group.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the documentation on the Terraform website or mirror website.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

To create a Managed Service for OpenSearch cluster:

1. In the configuration file, describe the parameters of the resources you want to create:

   • DB cluster: Description of the Managed Service for OpenSearch cluster and its hosts

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_opensearch_cluster" "<cluster_name>" {
     name                = "<cluster_name>"
     environment         = "<environment>"
     network_id          = "<network_ID>"
     security_group_ids  = ["<list_of_security_group_IDs>"]
     deletion_protection = "<deletion_protection>"
   
     config {
   
       version        = "<OpenSearch_version>"
       admin_password = "<admin_user_password>"
   
       opensearch {
         node_groups {
           name             = "<virtual_host_group_name>"
           assign_public_ip = <public_access>
           hosts_count      = <number_of_hosts>
           zone_ids         = ["<list_of_availability_zones>"]
           subnet_ids       = ["<list_of_subnet_IDs>"]
           roles            = ["<role_list>"]
           resources {
             resource_preset_id = "<host_class>"
             disk_size          = <storage_size_in_bytes>
             disk_type_id       = "<disk_type>"
           }
         }
   
         plugins = ["<list_of_plugin_names>"]
   
       }
   
       dashboards {
         node_groups {
           name             = "<virtual_host_group_name>"
           assign_public_ip = <public_access>
           hosts_count      = <number_of_hosts>
           zone_ids         = ["<list_of_availability_zones>"]
           subnet_ids       = ["<list_of_subnet_IDs>"]
           resources {
             resource_preset_id = "<host_class>"
             disk_size          = <storage_size_in_bytes>
             disk_type_id       = "<disk_type>"
           }
         }
       }
     }
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Environment, PRESTABLE or PRODUCTION.
   • deletion_protection: Deletion protection, true or false.
   • assign_public_ip: Public access to the host, true or false.
   • roles: DATA and MANAGER host roles.
   • maintenance_window: Maintenance window settings (including those for disabled clusters):
     • type: Maintenance type. The possible values include:
       • ANYTIME: Anytime.
       • WEEKLY: On a schedule.
     • day: Day of the week in DDD format for the WEEKLY type, e.g., MON.
     • hour: Hour UTC in HH format for the WEEKLY type, e.g., 21.

   Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.

   For a complete list of available Managed Service for OpenSearch cluster configuration fields, see the Terraform provider documentation.

2. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

3. Create a cluster.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

   Timeouts

   The Terraform provider sets the following timeouts for Managed Service for OpenSearch cluster operations:

   • Creating a cluster, including restoring from a backup: 30 minutes.
   • Editing a cluster: 60 minutes.
   • Deleting a cluster: 15 minutes.

   Operations exceeding the set timeout are interrupted.

   How do I change these limits?

   Add the timeouts block to the cluster description, for example:

   resource "yandex_mdb_opensearch_cluster" "<cluster_name>" {
     ...
     timeouts {
       create = "1h30m" # 1 hour 30 minutes
       update = "2h"    # 2 hours
       delete = "30m"   # 30 minutes
     }
   }
   

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Create a file named body.json and add the following contents to it:

   {
       "folderId": "<folder_ID>",
       "name": "<cluster_name>",
       "environment": "<environment>",
       "networkId": "<network_ID>",
       "securityGroupIds": [
           "<security_group_1_ID>",
           "<security_group_2_ID>",
           ...
           "<security_group_N_ID>"
       ],
       "serviceAccountId": "<service_account_ID>",
       "deletionProtection": <deletion_protection:_true_or_false>,
       "configSpec": {
           "version": "<OpenSearch_version>",
           "adminPassword": "<admin_user_password>",
           "opensearchSpec": {
               "plugins": [
                   "<OpenSearch_pugin_1>",
                   "<OpenSearch_pugin_2>",
                   ...
                   "<OpenSearch_pugin_N>"
               ],
               "nodeGroups": [
                   {
                       "name": "<host_group_name>",
                       "resources": {
                           "resourcePresetId": "<host_class>",
                           "diskSize": "<storage_size_in_bytes>",
                           "diskTypeId": "<disk_type>"
                       },
                       "roles": ["<role_1>","<role_2>"],
                       "hostsCount": "<number_of_hosts>",
                       "zoneIds": [
                           "<availability_zone_1>",
                           "<availability_zone_2>",
                           "<availability_zone_3>"
                       ],
                       "subnetIds": [
                           "<subnet_1_ID>",
                           "<subnet_2_ID>",
                           "<subnet_3_ID>"
                       ],
                       "assignPublicIp": <public_host_address:_true_or_false>,
                       "diskSizeAutoscaling": {
                           "plannedUsageThreshold": "<scheduled_increase_percentage>",
                           "emergencyUsageThreshold": "<immediate_increase_percentage>",
                           "diskSizeLimit": "<maximum_storage_size_in_bytes>"
                       }
                   },
                   ...
               ]
           },
           "dashboardsSpec": {
               "nodeGroups": [
                   {
                       "name": "<host_group_name>",
                       "resources": {
                           "resourcePresetId": "<host_class>",
                           "diskSize": "<storage_size_in_bytes>",
                           "diskTypeId": "<disk_type>"
                       },
                       "hostsCount": "<number_of_hosts>",
                       "zoneIds": ["<availability_zone>"],
                       "subnetIds": ["<subnet_ID>"],
                       "assignPublicIp": <public_host_address:_true_or_false>,
                       "diskSizeAutoscaling": {
                           "plannedUsageThreshold": "<scheduled_increase_percentage>",
                           "emergencyUsageThreshold": "<immediate_increase_percentage>",
                           "diskSizeLimit": "<maximum_storage_size_in_bytes>"
                       }
                   }
               ]
           },
           "access": {
               "dataTransfer": <access_from_Data_Transfer:_true_or_false>,
               "serverless": <access_from_Serverless_Containers:_true_or_false>
           }
       },
       "maintenanceWindow": {
           "weeklyMaintenanceWindow": {
               "day": "<day_of_week>",
               "hour": "<hour>"
           }
       }
   }
   

   Where:

   • folderId: Folder ID. You can request it with a list of folders in the cloud.

   • name: Cluster name.

   • environment: Cluster environment, PRODUCTION or PRESTABLE.

   • networkId: ID of the network the cluster will be in.

   • securityGroupIds: Security group IDs.

   • serviceAccountId: ID of the service account used for cluster operations.

   • deletionProtection: Protection of the cluster, its databases, and users against deletion.

   • configSpec: Cluster settings:

     • version: OpenSearch version.

     • adminPassword: admin user password.

     • opensearchSpec: Settings for OpenSearch host groups:

       • plugins: List of OpenSearch plugins you should additionally install in the cluster.

       • nodeGroups: Host settings as an array of elements, one for each host group. Each element has the following structure:

         • name: Host group name.

         • resources: Cluster resources:

           • resourcePresetId: Host class.
           • diskSize: Disk size in bytes.
           • diskTypeId: Disk type.

         • roles: List of host roles. A cluster must include at least one group of DATA hosts and one group of MANAGER hosts. This can be a single group with the two roles, or different groups with the roles.

         • hostsCount: Number of hosts in the group. It must have at least one DATA host and at least three MANAGER hosts.

         • zoneIds: List of availability zones the cluster hosts are located in.

         • subnetIds: List of subnet IDs.

         • assignPublicIp: Permission to connect to the host from the internet.

         • diskSizeAutoscaling: Automatic storage size increase settings:

           • plannedUsageThreshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

             Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

             If you set this parameter, configure the maintenance window schedule in the maintenanceWindow parameter.

           • emergencyUsageThreshold: Storage utilization percentage to trigger an immediate storage increase.

             Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to plannedUsageThreshold.

           • diskSizeLimit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

     • dashboardsSpec: Settings for Dashboards host groups. Contains the nodeGroups parameter, of the same structure as opensearchSpec.nodeGroups. The roles parameter is the only exception: Dashboards hosts can only have one role, DASHBOARDS, so you do not need to specify it directly.

     • access: Cluster settings for access to the following Yandex Cloud services:

       • dataTransfer: Yandex Data Transfer
       • serverless: Yandex Serverless Containers

   • maintenance_window.weeklyMaintenanceWindow: Maintenance window schedule:

     • day: Day of week, in DDD format, for scheduled maintenance.
     • hour: Hour, in HH format, for scheduled maintenance. The values range from 1 to 24. Use the UTC time zone.

3. Use the Cluster.Create method and make a request, e.g., via cURL:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --header "Content-Type: application/json" \
       --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters' \
       --data "@body.json"
   

4. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and put it into the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Create a file named body.json and add the following contents to it:

   {
       "folder_id": "<folder_ID>",
       "name": "<cluster_name>",
       "environment": "<environment>",
       "network_id": "<network_ID>",
       "security_group_ids": [
           "<security_group_1_ID>",
           "<security_group_2_ID>",
           ...
           "<security_group_N_ID>"
       ],
       "service_account_id": "<service_account_ID>",
       "deletion_protection": <deletion_protection:_true_or_false>,
       "config_spec": {
           "version": "<OpenSearch_version>",
           "admin_password": "<admin_user_password>",
           "opensearch_spec": {
               "plugins": [
                   "<OpenSearch_pugin_1>",
                   "<OpenSearch_pugin_2>",
                   ...
                   "<OpenSearch_pugin_N>"
               ],
               "node_groups": [
                   {
                       "name": "<host_group_name>",
                       "resources": {
                           "resource_preset_id": "<host_class>",
                           "disk_size": "<storage_size_in_bytes>",
                           "disk_type_id": "<disk_type>"
                       },
                       "roles": ["<role_1>","<role_2>"],
                       "hosts_count": "<number_of_hosts>",
                       "zone_ids": [
                           "<availability_zone_1>",
                           "<availability_zone_2>",
                           "<availability_zone_3>"
                       ],
                       "subnet_ids": [
                           "<subnet_1_ID>",
                           "<subnet_2_ID>",
                           "<subnet_3_ID>"
                       ],
                       "assign_public_ip": <public_host_address:_true_or_false>,
                       "disk_size_autoscaling": {
                           "planned_usage_threshold": "<scheduled_increase_percentage>",
                           "emergency_usage_threshold": "<immediate_increase_percentage>",
                           "disk_size_limit": "<maximum_storage_size_in_bytes>"
                       }
                   },
                   ...
               ]
           },
           "dashboards_spec": {
               "node_groups": [
                   {
                       "name": "<host_group_name>",
                       "resources": {
                           "resource_preset_id": "<host_class>",
                           "disk_size": "<storage_size_in_bytes>",
                           "disk_type_id": "<disk_type>"
                       },
                       "hosts_count": "<number_of_hosts>",
                       "zone_ids": ["<availability_zone>"],
                       "subnet_ids": ["<subnet_ID>"],
                       "assign_public_ip": <public_host_address:_true_or_false>,
                       "disk_size_autoscaling": {
                           "planned_usage_threshold": "<scheduled_increase_percentage>",
                           "emergency_usage_threshold": "<immediate_increase_percentage>",
                           "disk_size_limit": "<maximum_storage_size_in_bytes>"
                       }
                   }
               ]
           },
           "access": {
               "data_transfer": <access_from_Data_Transfer:_true_or_false>,
               "serverless": <access_from_Serverless_Containers:_true_or_false>
           }
       },
       "maintenance_window": {
           "weekly_maintenance_window": {
               "day": "<day_of_week>",
               "hour": "<hour>"
           }
       }
   }
   

   Where:

   • folder_id: Folder ID. You can request it with a list of folders in the cloud.

   • name: Cluster name.

   • environment: Cluster environment, PRODUCTION or PRESTABLE.

   • network_id: ID of the network the cluster will be in.

   • security_group_ids: Security group IDs.

   • service_account_id: ID of the service account used for cluster operations.

   • deletion_protection: Protection of the cluster, its databases, and users against deletion.

   • config_spec: Cluster settings:

     • version: OpenSearch version.

     • admin_password: admin user password.

     • opensearch_spec: Settings for OpenSearch host groups:

       • plugins: List of OpenSearch plugins you should additionally install in the cluster.

       • node_groups: Host settings as an array of elements, one for each host group. Each element has the following structure:

         • name: Host group name.

         • resources: Cluster resources:

           • resource_preset_id: Host class.
           • disk_size: Disk size in bytes.
           • disk_type_id: Disk type.

         • roles: List of host roles. A cluster must include at least one group of DATA hosts and one group of MANAGER hosts. This can be a single group with the two roles, or different groups with the roles.

         • hosts_count: Number of hosts in the group. It must have at least one DATA host and at least three MANAGER hosts.

         • zone_ids: List of availability zones the cluster hosts are located in.

         • subnet_ids: List of subnet IDs.

         • assign_public_ip: Permission to connect to the host from the internet.

         • disk_size_autoscaling: Automatic storage size increase settings:

           • planned_usage_threshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

             Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

             If you set this parameter, configure the maintenance window schedule in the maintenance_window parameter.

           • emergency_usage_threshold: Storage utilization percentage to trigger an immediate storage increase.

             Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned_usage_threshold.

           • disk_size_limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

     • dashboards_spec: Settings for Dashboards host groups. Contains the node_groups parameter, of the same structure as opensearch_spec.node_groups. The roles parameter is the only exception: Dashboards hosts can only have one role, DASHBOARDS, so you do not need to specify it directly.

     • access: Cluster settings for access to the following Yandex Cloud services:

       • data_transfer: Yandex Data Transfer
       • serverless: Yandex Serverless Containers

   • maintenance_window.weekly_maintenance_window: Maintenance window schedule:

     • day: Day of week, in DDD format, for scheduled maintenance.
     • hour: Hour, in HH format, for scheduled maintenance. The values range from 1 to 24. Use the UTC time zone.

4. Use the ClusterService.Create call and make a request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d @ \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.opensearch.v1.ClusterService.Create \
       < body.json
   

5. View the server response to make sure the request was successful.

Create a Managed Service for OpenSearch cluster with the following test specifications:

• Name: my-os-clstr.
• Environment: PRODUCTION.
• OpenSearch version: 2.8.
• admin user password: osadminpwd.
• OpenSearch node group name: os-group.
• Host class: s2.micro.
• Disk size: 10737418240 (in bytes).
• Disk type: network-ssd.
• Number of hosts: 1.
• Public address: Assigned.
• Host group roles: DATA and MANAGER.
• Maintenance time: Every Monday from 13:00 till 14:00.
• Network name: mynet.
• Subnet name: mysubnet.
• Availability zone: ru-central1-a.
• Address range: 10.1.0.0/16.
• Security group name: os-sg. The security group enables connecting to the cluster host from any network (including the internet) on port 9200.

The configuration file for this cluster is as follows:

resource "yandex_mdb_opensearch_cluster" "my-os-clstr" {
  name               = "my-os-clstr"
  environment        = "PRODUCTION"
  network_id         = yandex_vpc_network.mynet.id
  security_group_ids = [yandex_vpc_security_group.os-sg.id]

  config {

    version        = "2.8"
    admin_password = "osadminpwd"

    opensearch {
      node_groups {
        name             = "os-group"
        assign_public_ip = true
        hosts_count      = 1
        zone_ids         = ["ru-central1-a"]
        subnet_ids       = [yandex_vpc_subnet.mysubnet.id]
        roles            = ["DATA", "MANAGER"]
        resources {
          resource_preset_id = "s2.micro"
          disk_size          = 10737418240
          disk_type_id       = "network-ssd"
        }
      }
    }
  }
  maintenance_window {
    type = "WEEKLY"
    day  = "MON"
    hour = 14
  }
}

resource "yandex_vpc_network" "mynet" {
  name = "mynet"
}

resource "yandex_vpc_subnet" "mysubnet" {
  name           = "mysubnet"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.mynet.id
  v4_cidr_blocks = ["10.1.0.0/16"]
}

resource "yandex_vpc_security_group" "os-sg" {
  name       = "os-sg"
  network_id = yandex_vpc_network.mynet.id

  ingress {
    description    = "Allow connections to the Managed Service for OpenSearch cluster from the Internet"
    protocol       = "TCP"
    port           = 9200
    v4_cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description    = "The rule allows all outgoing traffic"
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
    from_port      = 0
    to_port        = 65535
  }
}