Creating an Elasticsearch cluster
Warning
Yandex Managed Service for Elasticsearch is unavailable as of April 11, 2024.
You can create an OpenSearch cluster in Yandex Cloud as an alternative to Elasticsearch.
A Managed Service for Elasticsearch cluster is a group of multiple linked Elasticsearch hosts. cluster provides high search performance by distributing search and indexing tasks across all hosts with the Data node role. To learn more about roles in the Managed Service for Elasticsearch cluster, see Host roles.
Note
- The number of hosts with the Data node role you can create along with an Elasticsearch cluster depends on the selected disk type and host class.
- Available disk types depend on the selected host class.
Creating a cluster
Note
As of June 13, 2022, the Gold
edition has been discontinued in Managed Service for Elasticsearch clusters. You cannot create a new cluster with this edition.
When creating a Managed Service for Elasticsearch cluster, specify parameters separately for the hosts with the Master node and the Data node roles.
You can use hosts only with the Data node role, without creating dedicated hosts for the Master node role. In this case, hosts with the Data node role combine the two roles.
To create a Managed Service for Elasticsearch cluster:
-
In the management console
, select the folder where you want to create a Managed Service for Elasticsearch cluster. -
Select Managed Service for Elasticsearch.
-
Click Create cluster.
-
Under Basic parameters:
-
Enter a name and description for the Managed Service for Elasticsearch cluster. The Managed Service for Elasticsearch cluster name must be unique within the folder.
-
Select the environment where you want to create the Managed Service for Elasticsearch cluster (you cannot change the environment once the cluster is created):
PRODUCTION
: For stable versions of your apps.PRESTABLE
: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.
-
Select the Elasticsearch version from the list.
-
Select the Elasticsearch edition.
-
Select a service account with the
storage.editor
role from the drop-down list or create a new one if you plan to use data from an Object Storage bucket with restricted access. For more information about setting up service accounts, see Configuring access to Object Storage.
-
-
Under Network settings, select the cloud network to host the Managed Service for Elasticsearch cluster and the security group for cluster network traffic. You may also need to set up security groups to connect to the Managed Service for Elasticsearch cluster.
-
Under User, specify the
admin
user password.This is a special user that is required for managing clusters and cannot be deleted. It is assigned the
superuser
role and can perform any operations on clusters.Tip
To perform routine tasks, we recommend creating individual users. For more information, see Managing users.
-
Configure hosts with the Data node role by opening the Data node tab:
-
Under Host class, select the platform, host type, and host class.
The host class defines the technical characteristics of VM instances Elasticsearch nodes are deployed on. All available options are listed under Host classes. When you change the host class for a Managed Service for Elasticsearch cluster, the specifications of all existing instances also change.
-
Under Storage:
-
Select the disk type.
The selected type determines the increments in which you can change your disk size:
- Network HDD and SSD storage: In 1 GB increments.
- Local SSD storage:
- For Intel Cascade Lake: In increments of 100 GB.
- For Intel Ice Lake: In 368 GB increments.
- Non-replicated SSD storage: In increments of 93 GB.
-
Select the storage size to use for data.
-
-
Under Hosts, select the configuration of the hosts created together with the Managed Service for Elasticsearch cluster:
-
To add a host, click Add host.
-
To change the added host, hover over its row and click
.When changing the host, you can:
-
Select the availability zone and subnet.
-
Enable public access.
Warning
You cannot enable public access to a host after creating a Managed Service for Elasticsearch cluster.
If public access is enabled for an Elasticsearch host with the Data node role, you can connect to this host, or Kibana hosted on it, over the internet. For more information, see Connecting to a cluster.
Tip
You can use Kibana even if you can't request public access to the hosts (for example, for security reasons). To do this, proxy the connections via the virtual machine in Yandex Compute Cloud that is hosted in the same network as the cluster. For more information, see Connecting to a cluster.
-
-
-
-
If necessary, configure the hosts with the Master node role by opening the Master node tab:
-
Under Host class, select the platform, host type, and host class.
-
Under Storage, configure storage the same way as for hosts with the Data node role.
-
Under Hosts, click Add hosts to add three hosts. To change one of the added hosts, hover over its row and click
.When changing the host, you can:
-
Select the availability zone and subnet.
-
Enable public access.
Tip
We do not recommend enabling public access for hosts with the Master node role as it may be unsafe.
-
-
-
Configure additional Managed Service for Elasticsearch cluster settings, if required:
-
Maintenance window: Maintenance window settings:
- To enable maintenance at any time, select arbitrary (default).
- To specify the preferred maintenance start time, select by schedule and specify the desired day of the week and UTC hour. For example, you can choose a time when the cluster is least loaded.
Maintenance operations are carried out both on enabled and disabled clusters. They may include updating the DBMS, applying patches, and so on.
-
Deletion protection: Manages protection of the cluster, its databases, and users against accidental deletion.
Cluster deletion protection will not prevent a manual connection to a cluster to delete data.
-
-
Configure the DBMS settings, if required.
-
Click Create.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To create a Managed Service for Elasticsearch cluster:
-
Check whether the folder has any subnets for the Managed Service for Elasticsearch cluster hosts:
yc vpc subnet list
If there are no subnets in the folder, create the required subnets in Yandex Virtual Private Cloud.
-
View a description of the create Managed Service for Elasticsearch cluster CLI command:
yc managed-elasticsearch cluster create --help
-
Specify the Managed Service for Elasticsearch cluster parameters in the create command (the example below does not list all possible parameters):
yc managed-elasticsearch cluster create \ --name <cluster_name> \ --environment <environment> \ --network-name <network_name> \ --host zone-id=<availability_zone>,subnet-id=<subnet_ID>,assign-public-ip=<public_access>,type=<host_type> \ --datanode-resource-preset <Data_node_host_class> \ --datanode-disk-size <storage_size_in_GB> \ --datanode-disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \ --masternode-resource-preset <Master_node_host_class> \ --masternode-disk-size <Master_node_storage_size_in_GB> \ --masternode-disk-type <network-ssd> \ --security-group-ids <list_of_security_group_IDs> \ --version <Elasticsearch_version> \ --edition <Elasticsearch_edition> \ --admin-password <admin_user_password> \ --plugins=<plugin_1_name>,...,<plugin_N_name> \ --deletion-protection=<deletion_protection>
Where:
--environment
: Environment,prestable
orproduction
.--host
: Host parameters:zone-id
: Availability zone.subnet-id
: Subnet ID. Specify if two or more subnets are created in the selected availability zone.assign-public-ip
: Internet access to the host via a public IP,true
orfalse
.type
: Host type:datanode
ormasternode
.
--datanode-resource-preset
: Host class with the Data node role.--datanode-disk-size
: Storage size in GB for hosts with the Data node role.--datanode-disk-type
: Storage type for hosts with the Data node role.--masternode-resource-preset
: Class of hosts with the Master node role.--masternode-disk-size
: Storage size in GB for hosts with the Master node role.--masternode-disk-type
: Storage type for hosts with the Master node role.--version
(optional): Elasticsearch version, 7.10, 7.11, 7.12, 7.13, 7.14, 7.15, 7.16, or 7.17.--edition
(optional): Elasticsearch edition:basic
orplatinum
.--deletion-protection
: Cluster deletion protection,true
orfalse
.
Enter the
subnet-id
if the selected availability zone has more than one subnet.Cluster deletion protection will not prevent a manual connection to a cluster to delete data.
Note
When creating a Managed Service for Elasticsearch cluster, the
anytime
maintenance mode is set by default. You can set a specific maintenance window when updating the Managed Service for Elasticsearch cluster settings.
Terraform
For more information about the provider resources, see the documentation on the Terraform
If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
To create a Managed Service for Elasticsearch cluster:
-
In the configuration file, describe the parameters of the resources you want to create:
-
Database cluster: Description of the Managed Service for Elasticsearch cluster and its hosts.
-
Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.
-
Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.
Here is an example of the configuration file structure:
resource "yandex_mdb_elasticsearch_cluster" "<cluster_name>" { name = "<cluster_name>" environment = "<environment>" network_id = "<network_ID>" deletion_protection = "<deletion_protection>" config { version = "<Elasticsearch_version>" edition = "<Elasticsearch_edition>" admin_password = "<admin_user_password>" data_node { resources { resource_preset_id = "<host_class>" disk_type_id = "<disk_type>" disk_size = <storage_size_in_GB> } } master_node { resources { resource_preset_id = "<host_class>" disk_type_id = "<disk_type>" disk_size = <storage_size_in_GB> } } plugins = [ "<list_of_plugin_names>" ] } security_group_ids = [ "<list_of_security_group_IDs>" ] host { name = "<host_name>" zone = "<availability_zone>" type = "<host_role>" assign_public_ip = <public_access> subnet_id = "<subnet_ID>" } } resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" } resource "yandex_vpc_subnet" "<subnet_name>" { name = "<subnet_name>" zone = "<availability_zone>" network_id = "<network_ID>" v4_cidr_blocks = ["<range>"] }
Where:
environment
: Environment,PRESTABLE
orPRODUCTION
.deletion_protection
: Deletion protection,true
orfalse
.version
(optional): Elasticsearch version, 7.10, 7.11, 7.12, 7.13, 7.14, 7.15, 7.16, or 7.17.edition
(optional): Elasticsearch edition,basic
orplatinum
.host
: Host parameters:name
: Host name.zone
: Availability zone.type
: Host type:DATA_NODE
orMASTER_NODE
.assign_public_ip
: Public access to the host,true
orfalse
.subnet-id
: Subnet ID. Specify if two or more subnets are created in the selected availability zone.
If the Managed Service for Elasticsearch cluster deletion protection is activated, this does not protect the DB contents.
-
To set up the maintenance window (for disabled clusters as well), add the
maintenance_window
block to the cluster description:resource "yandex_mdb_elasticsearch_cluster" "<cluster_name>" { ... maintenance_window { type = <maintenance_type> day = <day_of_week> hour = <hour> } ... }
Where:
type
: Maintenance type. The possible values include:anytime
: Anytime.weekly
: By schedule.
day
: Day of the week for theweekly
type inDDD
format, e.g.,MON
.hour
: Hour of the day for theweekly
type in theHH
format, e.g.,21
.
For more information about resources you can create with Terraform, see the Terraform provider documentation
. -
-
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
Create a Managed Service for Elasticsearch cluster.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
Time limits
A Terraform provider sets the timeout for Managed Service for Elasticsearch cluster operations:
- Creating a cluster, including by restoring one from a backup: 30 minutes.
- Editing a cluster: 60 minutes.
- Deleting a cluster: 15 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the
timeouts
block to the cluster description, for example:resource "yandex_mdb_elasticsearch_cluster" "<cluster_name>" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } }
-
To create a Managed Service for Elasticsearch cluster, use the create REST API method for the Cluster resource or the ClusterService/Create gRPC API call and provide the following in the request:
-
ID of the folder to host the Managed Service for Elasticsearch cluster, in the
folderId
parameter. -
Managed Service for Elasticsearch cluster name in the
name
parameter. -
Elasticsearch version in the
configSpec.version
parameter. -
Elasticsearch edition in the
configSpec.edition
parameter. -
Managed Service for Elasticsearch cluster configuration in the
configSpec
parameter, including:- Class of hosts with the Master node role in the
configSpec.elasticsearchSpec.masterNode.resources
parameter. If you do not want to create dedicated hosts with the Master node role, do not set values for the group ofconfigSpec.elasticsearchSpec.masterNode
parameters. - Class of hosts with the Data node role in the
configSpec.elasticsearchSpec.dataNode.resources
parameter.
- Class of hosts with the Master node role in the
-
Configuration of the Managed Service for Elasticsearch cluster hosts in one or more
hostSpecs
parameters. -
Network ID in the
networkId
parameter. -
Security group IDs in the
securityGroupIds
parameter. -
List of plugins in the
configSpec.elasticsearchSpec.plugins
parameter. -
Maintenance window settings (including for disabled Managed Service for Elasticsearch clusters) in the
maintenanceWindow
parameter. -
Settings for access from Yandex Data Transfer in the
configSpec.access.dataTransfer
parameter.
Warning
If you specified security group IDs when creating a Managed Service for Elasticsearch cluster, you may also need to set up security groups to connect to the cluster.
Examples
Creating a single-host cluster
To create a Managed Service for Elasticsearch cluster with a single host, provide a single --host
parameter.
Create a Managed Service for Elasticsearch cluster with the following test specifications:
-
Name:
my-es-clstr
. -
Version:
7.17
. -
Edition:
Platinum
. -
Environment:
PRODUCTION
. -
Network:
default
. -
Security group ID:
enpp2s8l3irh********
. -
With a single publicly available
s2.micro
host with the Data node role in theb0rcctk2rvtr8efcch64
subnet, in theru-central1-a
availability zone. -
Network SSD storage (
network-ssd
): 20 GB. -
esadminpwd
password for theadmin
user. -
Protection against accidental Managed Service for Elasticsearch cluster deletion: Enabled.
Run the following command:
yc managed-elasticsearch cluster create \
--name my-es-clstr \
--environment production \
--network-name default \
--host zone-id=ru-central1-a,assign-public-ip=true,type=datanode \
--datanode-resource-preset s2.micro \
--datanode-disk-type=network-ssd \
--datanode-disk-size=20 \
--admin-password=esadminpwd \
--security-group-ids enpp2s8l3irh******** \
--version 7.17 \
--edition platinum \
--deletion-protection=true
Create a Managed Service for Elasticsearch cluster. The configuration file for the Managed Service for Elasticsearch cluster is as follows:
resource "yandex_mdb_elasticsearch_cluster" "my-es-clstr" {
name = "my-es-clstr"
environment = "PRODUCTION"
network_id = yandex_vpc_network.mynet.id
deletion_protection = "true"
config {
edition = "basic"
version = "7.17"
admin_password = "esadminpwd"
data_node {
resources {
resource_preset_id = "s2.micro"
disk_type_id = "network-ssd"
disk_size = 20
}
}
}
security_group_ids = [ yandex_vpc_security_group.es-sg.id ]
host {
name = "node"
zone = "ru-central1-a"
type = "DATA_NODE"
assign_public_ip = true
subnet_id = yandex_vpc_subnet.mysubnet.id
}
}
resource "yandex_vpc_network" "mynet" {
name = "mynet"
}
resource "yandex_vpc_subnet" "mysubnet" {
name = "mysubnet"
zone = "ru-central1-a"
network_id = yandex_vpc_network.mynet.id
v4_cidr_blocks = ["10.5.0.0/24"]
}
resource "yandex_vpc_security_group" "es-sg" {
name = "es-sg"
network_id = yandex_vpc_network.mynet.id
ingress {
description = "Kibana"
port = 443
protocol = "TCP"
v4_cidr_blocks = [ "0.0.0.0/0" ]
}
ingress {
description = "Elasticsearch"
port = 9200
protocol = "TCP"
v4_cidr_blocks = [ "0.0.0.0/0" ]
}
}
Where the following test configuration is used:
-
Name:
my-es-clstr
. -
Version:
7.17
. -
Edition:
Basic
. -
Environment:
PRODUCTION
. -
--deletion-protection
: Managed Service for Elasticsearch cluster deletion protection. You cannot delete a Managed Service for Elasticsearch cluster with this option enabled. -
Cloud ID:
b1gq90dgh25bebiu75o
. -
Folder ID:
b1gia87mbaomkfvsleds
. -
Network: New
mynet
network. -
Security group: New
es-sg
security group allowing connections from the internet to the Managed Service for Elasticsearch cluster on ports 443 (Kibana) and 9200 (Elasticsearch). -
Hosts: One publicly available
s2.micro
host with the Data node role, subnet:mysubnet
, availability zone:ru-central1-a
.mysubnet
will have the10.5.0.0/24
range. -
Network SSD storage (
network-ssd
): 20 GB. -
esadminpwd
password for theadmin
user.