Creating an OpenSearch cluster

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a Managed Service for OpenSearch cluster:

1. View the description of the create cluster CLI command:

   yc managed-opensearch cluster create --help
   

2. Specify cluster parameters in the create command (the list of supported parameters in the example is not exhaustive):

   yc managed-opensearch cluster create \
      --name <cluster_name> \
      --description <cluster_description> \
      --labels <labels> \
      --environment <environment:_production_or_prestable> \
      --network-name <network_name> \
      --security-group-ids <security_group_IDs> \
      --service-account-name <service_account_name> \
      --delete-protection \
      --maintenance schedule=<maintenance_type>,`
                   `weekday=<day_of_week>,`
                   `hour=<hour> \
      --version <OpenSearch_version> \
      --read-admin-password \
      --data-transfer-access=<true_or_false> \
      --serverless-access=<true_or_false> \
      --plugins <OpenSearch_plugins> \
      --advanced-params <additional_parameters> \
      --opensearch-node-group name=<OpenSearch_host_group_name>,`
                             `resource-preset-id=<host_class>,`
                             `disk-size=<disk_size_in_bytes>,`
                             `disk-type-id=<network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd>,`
                             `hosts-count=<number_of_hosts_in_group>,`
                             `zone-ids=<availability_zones>,`
                             `subnet-names=<subnet_names>,`
                             `assign-public-ip=<assign_public_address:_true_or_false>,`
                             `roles=<host_roles> \
      --dashboards-node-group name=<Dashboards_host_group_name>,`
                             `resource-preset-id=<host_class>,`
                             `disk-size=<disk_size_in_bytes>,`
                             `disk-type-id=<network-ssd>,`
                             `hosts-count=<number_of_hosts_in_group>,`
                             `zone-ids=<availability_zones>,`
                             `subnet-names=<subnet_names>,`
                             `assign-public-ip=<assign_public_address:_true_or_false>
   

   Where:

   • --labels: Yandex Cloud labels labels in <key>=<value> format. You can use them to logically separate resources.

   • --environment: Environment:

     • production: For stable versions of your apps.
     • prestable: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.

   • --service-account-name: Name of the service account for access to Yandex Object Storage as a repository of OpenSearch snapshots. For more information on service accounts, see the Yandex Identity and Access Management documentation.

   • --delete-protection: Cluster protection from accidental deletion by a user. With deletion protection enabled, you still can connect to a cluster manually to delete data.

   • --maintenance: Maintenance window settings:

     • To allow maintenance at any time, do not specify the --maintenance parameter in the command (default configuration) or specify --maintenance schedule=anytime.
     • To specify the preferred start time for maintenance, specify this parameter in the command: --maintenance schedule=weekly,weekday=<day_of_week>,hour=<hour_in_UTC>. In this case, maintenance will take place every week on a specified day at a specified time.

     Both enabled and disabled clusters undergo maintenance. Maintenance may involve such operations as applying patches or updating DBMS's.

   • --read-admin-password: admin user password. If you specify this parameter in the command, it will prompt you to enter a password.

   • --serverless-access: Access from Yandex Serverless Containers, true or false.

   • --plugins: OpenSearch plugins to install in the cluster.

   • --advanced-params: Additional cluster parameters. The possible values are:

     • max-clause-count: Maximum allowed number of boolean clauses per query. For more information, see the OpenSearch documentation.
     • fielddata-cache-size: JVM heap size allocated for the fielddata data structure. You can specify either an absolute value or percentage, e.g., 512mb or 50%. For more information, see the OpenSearch documentation.
     • reindex-remote-whitelist: List of remote hosts whose indexes contain documents to copy for reindexing. Specify the parameter value as <host_address>:<port>. If you need to specify more than one host, list values separated by commas. For more information, see the OpenSearch documentation.

   • --opensearch-node-group: OpenSearch host group configuration, where:

     • --resource-preset-id: Host class that defines the configuration of virtual machines the OpenSearch nodes will be deployed on. All available options are listed under Host classes.

     • disk-size: Disk size in bytes. Minimum and maximum values depend on the selected host class.

     • disk-type-id: Disk type.

     • roles: Host roles. The possible values include:

       • data: Assigns the DATA role only.
       • manager: Assigns the MANAGER role only.
       • data+manager or manager+data: Assigns both roles.

       Tip

       We do not recommend enabling public access to hosts with the MANAGER role, as this is not secure.

   • --dashboards-node-group: Dashboards host group configuration. It is configured in the same way as the OpenSearch host group, except for the host roles. You do not need to configure any roles for the Dashboards group.

Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

For more information about the provider resources, see the documentation on the Terraform website or the mirror.

If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

To create a Managed Service for OpenSearch cluster:

1. In the configuration file, describe the parameters of the resources you want to create:

   • DB cluster: Description of the Managed Service for OpenSearch cluster and its hosts

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_opensearch_cluster" "<cluster_name>" {
     name                = "<cluster_name>"
     environment         = "<environment>"
     network_id          = "<network_ID>"
     security_group_ids  = ["<list_of_security_group_IDs>"]
     deletion_protection = "<deletion_protection>"
   
     config {
   
       version        = "<OpenSearch_version>"
       admin_password = "<admin_user_password>"
   
       opensearch {
         node_groups {
           name             = "<virtual_host_group_name>"
           assign_public_ip = <public_access>
           hosts_count      = <number_of_hosts>
           zone_ids         = ["<list_of_availability_zones>"]
           subnet_ids       = ["<list_of_subnet_IDs>"]
           roles            = ["<role_list>"]
           resources {
             resource_preset_id = "<host_class>"
             disk_size          = <storage_size_in_bytes>
             disk_type_id       = "<disk_type>"
           }
         }
   
         plugins = ["<list_of_plugin_names>"]
   
       }
   
       dashboards {
         node_groups {
           name             = "<virtual_host_group_name>"
           assign_public_ip = <public_access>
           hosts_count      = <number_of_hosts>
           zone_ids         = ["<list_of_availability_zones>"]
           subnet_ids       = ["<list_of_subnet_IDs>"]
           resources {
             resource_preset_id = "<host_class>"
             disk_size          = <storage_size_in_bytes>
             disk_type_id       = "<disk_type>"
           }
         }
       }
     }
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Environment, PRESTABLE or PRODUCTION.
   • deletion_protection: Deletion protection, true or false.
   • assign_public_ip: Public access to the host, true or false.
   • roles: DATA and MANAGER host roles.
   • maintenance_window: Maintenance window settings (including those for disabled clusters):
     • type: Maintenance type. The possible values include:
       • ANYTIME: Anytime.
       • WEEKLY: On a schedule.
     • day: Day of the week in DDD format for the WEEKLY type, e.g., MON.
     • hour: Hour of the day in HH format for the WEEKLY type, e.g., 21.

   Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.

   For a complete list of available Managed Service for OpenSearch cluster configuration fields, see the Terraform provider documentation.

2. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

3. Create a cluster.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

   Timeouts

   The Terraform provider sets the following timeouts for Managed Service for OpenSearch cluster operations:

   • Creating a cluster, including restoring from a backup: 30 minutes.
   • Editing a cluster: 60 minutes.
   • Deleting a cluster: 15 minutes.

   Operations exceeding the set timeout are interrupted.

   How do I change these limits?

   Add the timeouts block to the cluster description, for example:

   resource "yandex_mdb_opensearch_cluster" "<cluster_name>" {
     ...
     timeouts {
       create = "1h30m" # 1 hour 30 minutes
       update = "2h"    # 2 hours
       delete = "30m"   # 30 minutes
     }
   }
   

To create a Managed Service for OpenSearch cluster, use the create REST API method for the Cluster resource or the ClusterService/Create gRPC API call and provide the following in the request:

• ID of the folder to host the cluster, in the folderId parameter.

• Cluster name in the name parameter.

• OpenSearch version in the configSpec.version parameter.

• admin user password in the configSpec.adminPassword parameter.

• Configuration of one or more groups of hosts with the DATA and MANAGER (optional) roles in the configSpec.opensearchSpec.nodeGroups parameter.

• Configuration of one or more groups of hosts with the DASHBOARDS role in the configSpec.dashboardsSpec.nodeGroups parameter.

• List of plugins in the configSpec.opensearchSpec.plugins parameter.

• Settings for access from other services in the configSpec.access parameter.

• Network ID in the networkId parameter.

• Security group IDs in the securityGroupIds parameter. You may also need to set up security groups to connect to the cluster.

• ID of the service account used for cluster operations in the serviceAccountId parameter.

• Cluster deletion protection settings in the deletionProtection parameter.

  Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.

• Maintenance window settings (including for disabled clusters) in the maintenanceWindow parameter.

Create a Managed Service for OpenSearch cluster with the following test specifications:

• Name: my-os-clstr.
• Environment: PRODUCTION.
• OpenSearch version: 2.8.
• admin user password: osadminpwd.
• OpenSearch node group name: os-group.
• Host class: s2.micro.
• Disk size: 10737418240 (in bytes).
• Disk type: network-ssd.
• Number of hosts: 1.
• Public address: Assigned.
• Host group roles: DATA and MANAGER.
• Maintenance time: Every Monday from 13:00 till 14:00.
• Network name: mynet.
• Subnet name: mysubnet.
• Availability zone: ru-central1-a.
• Address range: 10.1.0.0/16.
• Security group name: os-sg. The security group enables connecting to the cluster host from any network (including the internet) on port 9200.

The configuration file for this cluster is as follows:

resource "yandex_mdb_opensearch_cluster" "my-os-clstr" {
  name               = "my-os-clstr"
  environment        = "PRODUCTION"
  network_id         = yandex_vpc_network.mynet.id
  security_group_ids = [yandex_vpc_security_group.os-sg.id]

  config {

    version        = "2.8"
    admin_password = "osadminpwd"

    opensearch {
      node_groups {
        name             = "os-group"
        assign_public_ip = true
        hosts_count      = 1
        zone_ids         = ["ru-central1-a"]
        subnet_ids       = [yandex_vpc_subnet.mysubnet.id]
        roles            = ["DATA", "MANAGER"]
        resources {
          resource_preset_id = "s2.micro"
          disk_size          = 10737418240
          disk_type_id       = "network-ssd"
        }
      }
    }
  }
  maintenance_window {
    type = "WEEKLY"
    day  = "MON"
    hour = 14
  }
}

resource "yandex_vpc_network" "mynet" {
  name = "mynet"
}

resource "yandex_vpc_subnet" "mysubnet" {
  name           = "mysubnet"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.mynet.id
  v4_cidr_blocks = ["10.1.0.0/16"]
}

resource "yandex_vpc_security_group" "os-sg" {
  name       = "os-sg"
  network_id = yandex_vpc_network.mynet.id

  ingress {
    description    = "Allow connections to the Managed Service for OpenSearch cluster from the Internet"
    protocol       = "TCP"
    port           = 9200
    v4_cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description    = "The rule allows all outgoing traffic"
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
    from_port      = 0
    to_port        = 65535
  }
}