Creating an Elasticsearch cluster

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a Managed Service for Elasticsearch cluster:

1. Check whether the folder has any subnets for the Managed Service for Elasticsearch cluster hosts:

   yc vpc subnet list
   

   If there are no subnets in the folder, create the required subnets in Yandex Virtual Private Cloud.

2. View the description of the create Managed Service for Elasticsearch cluster CLI command:

   yc managed-elasticsearch cluster create --help
   

3. Specify the Managed Service for Elasticsearch cluster parameters in the create command (the example below does not list all possible parameters):

   yc managed-elasticsearch cluster create \
     --name <cluster_name> \
     --environment <environment> \
     --network-name <network_name> \
     --host zone-id=<availability_zone>,subnet-id=<subnet_ID>,assign-public-ip=<public_access>,type=<host_type> \
     --datanode-resource-preset <Data_node_host_class> \
     --datanode-disk-size <storage_size_in_GB> \
     --datanode-disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
     --masternode-resource-preset <Master_node_host_class> \
     --masternode-disk-size <Master_node_storage_size_in_GB> \
     --masternode-disk-type <network-ssd> \
     --security-group-ids <list_of_security_group_IDs> \
     --version <Elasticsearch_version> \
     --edition <Elasticsearch_edition> \
     --admin-password <admin_user_password> \
     --plugins=<plugin_1_name>,...,<plugin_N_name> \
     --deletion-protection=<deletion_protection>
   

   Where:

   • --environment: Environment, prestable or production.
   • --host: Host parameters:
     • zone-id: Availability zone.
     • subnet-id: Subnet ID. Specify if two or more subnets are created in the selected availability zone.
     • assign-public-ip: Internet access to the host via a public IP address, true or false.
     • type: Host role: datanode or masternode.
   • --datanode-resource-preset: Host class with the Data node role.
   • --datanode-disk-size: Storage size in GB for hosts with the Data node role.
   • --datanode-disk-type: Storage type for hosts with the Data node role.
   • --masternode-resource-preset: Class of hosts with the Master node role.
   • --masternode-disk-size: Storage size in GB for hosts with the Master node role.
   • --masternode-disk-type: Storage type for hosts with the Master node role.
   • --version (optional): Elasticsearch version, 7.10, 7.11, 7.12, 7.13, 7.14, 7.15, 7.16, or 7.17.
   • --edition (optional): Elasticsearch edition: basic or platinum.
   • --deletion-protection: Cluster deletion protection, true or false.

   Enter the subnet-id if the selected availability zone has more than one subnet.

   Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

   Note

   When creating a Managed Service for Elasticsearch cluster, the anytime maintenance mode is set by default. You can set a specific maintenance window when updating the Managed Service for Elasticsearch cluster settings.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the documentation on the Terraform website or mirror website.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

To create a Managed Service for Elasticsearch cluster:

1. In the configuration file, describe the parameters of the resources you want to create:

   • Database cluster: Description of the Managed Service for Elasticsearch cluster and its hosts.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_elasticsearch_cluster" "<cluster_name>" {
     name                = "<cluster_name>"
     environment         = "<environment>"
     network_id          = "<network_ID>"
     deletion_protection = "<deletion_protection>"
   
     config {
       version = "<Elasticsearch_version>"
       edition = "<Elasticsearch_edition>"
   
       admin_password = "<admin_user_password>"
   
       data_node {
         resources {
           resource_preset_id = "<host_class>"
           disk_type_id       = "<disk_type>"
           disk_size          = <storage_size_in_GB>
         }
       }
   
       master_node {
         resources {
           resource_preset_id = "<host_class>"
           disk_type_id       = "<disk_type>"
           disk_size          = <storage_size_in_GB>
         }
       }
   
       plugins = [ "<list_of_plugin_names>" ]
   
     }
   
     security_group_ids = [ "<list_of_security_group_IDs>" ]
   
     host {
       name             = "<host_name>"
       zone             = "<availability_zone>"
       type             = "<host_role>"
       assign_public_ip = <public_access>
       subnet_id        = "<subnet_ID>"
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Environment, PRESTABLE or PRODUCTION.
   • deletion_protection: Deletion protection, true or false.
   • version (optional): Elasticsearch version, 7.10, 7.11, 7.12, 7.13, 7.14, 7.15, 7.16, or 7.17.
   • edition (optional): Elasticsearch edition, basic or platinum.
   • host: Host parameters:
     • name: Host name.
     • zone: Availability zone.
     • type: Host role: DATA_NODE or MASTER_NODE.
     • assign_public_ip: Public access to the host, true or false.
     • subnet-id: Subnet ID. Specify if two or more subnets are created in the selected availability zone.

   If the Managed Service for Elasticsearch cluster deletion protection is activated, this does not protect the DB contents.

   1. To set up the maintenance window (for disabled clusters as well), add the maintenance_window block to the cluster description:

      resource "yandex_mdb_elasticsearch_cluster" "<cluster_name>" {
        ...
        maintenance_window {
          type = <maintenance_type>
          day  = <day_of_week>
          hour = <hour>
        }
        ...
      }
      

      Where:

      • type: Maintenance type. The possible values include:
        • anytime: Anytime.
        • weekly: By schedule.
      • day: Day of the week for the weekly type in DDD format, e.g., MON.
      • hour: Hour of the day for the weekly type in the HH format, e.g., 21.

   For more information about the resources you can create with Terraform, see the Terraform provider documentation.

2. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

3. Create a Managed Service for Elasticsearch cluster.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

   Time limits

   A Terraform provider sets the timeout for Managed Service for Elasticsearch cluster operations:

   • Creating a cluster, including by restoring one from a backup: 30 minutes.
   • Editing a cluster: 60 minutes.
   • Deleting a cluster: 15 minutes.

   Operations exceeding the set timeout are interrupted.

   How do I change these limits?

   Add the timeouts block to the cluster description, for example:

   resource "yandex_mdb_elasticsearch_cluster" "<cluster_name>" {
     ...
     timeouts {
       create = "1h30m" # 1 hour 30 minutes
       update = "2h"    # 2 hours
       delete = "30m"   # 30 minutes
     }
   }
   

To create a Managed Service for Elasticsearch cluster, use the create REST API method for the Cluster resource or the ClusterService/Create gRPC API call and provide the following in the request:

• ID of the folder to host the Managed Service for Elasticsearch cluster, in the folderId parameter.

• Managed Service for Elasticsearch cluster name in the name parameter.

• Elasticsearch version in the configSpec.version parameter.

• Elasticsearch edition in the configSpec.edition parameter.

• Managed Service for Elasticsearch cluster configuration in the configSpec parameter, including:

  • Class of hosts with the Master node role in the configSpec.elasticsearchSpec.masterNode.resources parameter. If you do not want to create dedicated hosts with the Master node role, do not set values for the group of configSpec.elasticsearchSpec.masterNode parameters.
  • Class of hosts with the Data node role in the configSpec.elasticsearchSpec.dataNode.resources parameter.

• Configuration of the Managed Service for Elasticsearch cluster hosts in one or more hostSpecs parameters.

• Network ID in the networkId parameter.

• Security group IDs in the securityGroupIds parameter.

• List of plugins in the configSpec.elasticsearchSpec.plugins parameter.

• Maintenance window settings (including for disabled Managed Service for Elasticsearch clusters) in the maintenanceWindow parameter.

• Settings for access from Yandex Data Transfer in the configSpec.access.dataTransfer parameter.

Create a Managed Service for Elasticsearch cluster. The configuration file for the Managed Service for Elasticsearch cluster is as follows:

resource "yandex_mdb_elasticsearch_cluster" "my-es-clstr" {
  name                = "my-es-clstr"
  environment         = "PRODUCTION"
  network_id          = yandex_vpc_network.mynet.id
  deletion_protection = "true"

  config {
    edition = "basic"
    version = "7.17"

    admin_password = "esadminpwd"

    data_node {
      resources {
        resource_preset_id = "s2.micro"
        disk_type_id       = "network-ssd"
        disk_size          = 20
      }
    }

  }

  security_group_ids = [ yandex_vpc_security_group.es-sg.id ]

  host {
    name             = "node"
    zone             = "ru-central1-a"
    type             = "DATA_NODE"
    assign_public_ip = true
    subnet_id        = yandex_vpc_subnet.mysubnet.id
  }

}

resource "yandex_vpc_network" "mynet" {
  name = "mynet"
}

resource "yandex_vpc_subnet" "mysubnet" {
  name           = "mysubnet"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.mynet.id
  v4_cidr_blocks = ["10.5.0.0/24"]
}

resource "yandex_vpc_security_group" "es-sg" {
  name       = "es-sg"
  network_id = yandex_vpc_network.mynet.id

  ingress {
    description    = "Kibana"
    port           = 443
    protocol       = "TCP"
    v4_cidr_blocks = [ "0.0.0.0/0" ]
  }

  ingress {
    description    = "Elasticsearch"
    port           = 9200
    protocol       = "TCP"
    v4_cidr_blocks = [ "0.0.0.0/0" ]
  }
}

Where the following test configuration is used:

• Name: my-es-clstr.

• Version: 7.17.

• Edition: Basic.

• Environment: PRODUCTION.

• --deletion-protection: Managed Service for Elasticsearch cluster deletion protection. You cannot delete a Managed Service for Elasticsearch cluster with this option enabled.

• Cloud ID: b1gq90dgh25bebiu75o.

• Folder ID: b1gia87mbaomkfvsleds.

• New mynet network.

• Security group: New es-sg security group allowing connections from the internet to the Managed Service for Elasticsearch cluster on ports 443 (Kibana) and 9200 (Elasticsearch).

• Hosts: One publicly available s2.micro host with the Data node role, subnet: mysubnet, availability zone: ru-central1-a. Range for mysubnet: 10.5.0.0/24.

• Network SSD storage (network-ssd): 20 GB.

• esadminpwd password for the admin user.