Creating an OpenSearch cluster
A Managed Service for OpenSearch cluster is a group of multiple interlinked OpenSearch and dashboardsDATA
role. To learn more about roles in the cluster, see Host roles.
Available disk types depend on the selected host class.
For more information, see Resource relationships in the service.
Creating a cluster
When creating a cluster, you need to specify individual parameters for each host group.
To create a Managed Service for OpenSearch cluster, you need the vpc.user role and the managed-opensearch.editor role or higher. For more information on assigning roles, see the Identity and Access Management documentation.
To create a Managed Service for OpenSearch cluster:
-
In the management console
, select the folder where you want to create a cluster. -
Select Managed Service for OpenSearch.
-
Click Create cluster.
-
Under Basic parameters:
-
Enter a name for the cluster. It must be unique within the folder.
-
(Optional) Enter a cluster description.
-
Select the environment where you want to create the cluster (you cannot change the environment once the cluster is created):
PRODUCTION
: For stable versions of your apps.PRESTABLE
: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.
-
Select the OpenSearch version.
-
Select the plugins you want to install in the cluster.
-
-
Under Network settings, select the cloud network to host the cluster and security groups for cluster network traffic. You may also need to set up security groups to connect to the cluster.
-
Under Virtual node group 1, configure the
OpenSearch
host group:-
Select the host group type:
OpenSearch
-
Enter a name for the host group, which must be unique within the cluster.
-
Select the
DATA
andMANAGER
host roles. -
Select the platform, host type, and host class.
The host class defines the technical characteristics of virtual machines that OpenSearch nodes are deployed on. All available options are listed under Host classes.
-
Select the disk type and data storage size.
The selected type determines the increments in which you can change your disk size:
- Network HDD and SSD storage: In 1 GB increments.
- Local SSD storage:
- For Intel Cascade Lake: In increments of 100 GB.
- For Intel Ice Lake: In 368 GB increments.
- Non-replicated SSD storage: In increments of 93 GB.
-
(Optional) Under Automatic increase of storage size, set up automatic increase of disk size:
-
In the Increase size field, set the conditions to:
- Increase the storage size during the next maintenance window if the storage is more than the specified percent (%) full.
- Increase the storage size right away if the storage is more than the specified percent (%) full.
You can set both conditions, but the threshold for immediate increase must be higher than that for increase during the maintenance window.
-
In the Maximum storage size field, specify the maximum storage size that can be set when increasing the storage size automatically.
If the specified threshold is reached, the storage size increases differently depending on disk type:
-
For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.
-
For non-replicated SSDs, by 93 GB.
-
For local SSDs:
- Intel Cascade Lake cluster, by 100 GB.
- Intel Ice Lake cluster, by 368 GB.
If the threshold is reached again, the storage size will be automatically increased until it reaches the specified maximum. After that, you can specify a new maximum storage size manually.
Warning
- You cannot decrease the storage size.
- While resizing the storage, cluster hosts will be unavailable.
-
If you have set up the storage size to increase within the maintenance window, set up a schedule for the maintenance window.
-
-
Specify how hosts should be distributed across availability zones and subnets.
-
Select the number of hosts to create.
-
Enable Public access if you want to allow connecting to hosts over the internet.
Tip
For security reasons, we do not recommend enabling public access for hosts with the
MANAGER
role.
Warning
After creating your cluster, you can only change the host configuration using the API. However, you can also create a new host group with a different configuration if needed.
-
-
Configure the
Dashboards
host group under Virtual node group 2, if required:-
Select the platform, host type, and host class.
-
Set up storage in the same way as for
OpenSearch
hosts. -
Specify how hosts should be distributed across availability zones and subnets.
-
Select the number of hosts to create.
-
Enable Public access if you want to allow connecting to hosts over the internet.
Tip
You can use OpenSearch Dashboards even if you can't request public access to the hosts (for example, for security reasons). To do this, proxy the connections via the virtual machine in Yandex Compute Cloud that is hosted in the same network as the cluster. For more information, see Connecting to OpenSearch Dashboards.
-
-
If required, click Add virtual node group to add another host group or more.
-
Under Service settings:
-
Enter the password for the
admin
user.This is a special user that is required for managing clusters and cannot be deleted. It is assigned the
superuser
role and can perform any operations on clusters.Tip
This user is not intended for routine jobs; for those, we recommend creating regular users. For more information, see Managing OpenSearch users.
-
If required, change additional cluster settings:
-
Maintenance window: Maintenance window settings:
- To enable maintenance at any time, select arbitrary (default).
- To specify the preferred maintenance start time, select by schedule and specify the desired day of the week and UTC hour. For example, you can choose a time when the cluster is least loaded.
Maintenance operations are carried out both on enabled and disabled clusters. They may include updating the DBMS, applying patches, and so on.
-
Service account is an account to access Yandex Object Storage as a repository of OpenSearch snapshots. For more detail on service accounts, see the Yandex Identity and Access Management documentation.
-
Deletion protection: Manages protection of the cluster, its databases, and users against accidental deletion.
Cluster deletion protection will not prevent a manual connection to a cluster to delete data.
-
-
-
Click Create cluster.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To create a Managed Service for OpenSearch cluster:
-
View the description of the create cluster CLI command:
yc managed-opensearch cluster create --help
-
Specify cluster parameters in the create command (the list of supported parameters in the example is not exhaustive):
yc managed-opensearch cluster create \ --name <cluster_name> \ --description <cluster_description> \ --labels <labels> \ --environment <environment:_production_or_prestable> \ --network-name <network_name> \ --security-group-ids <security_group_IDs> \ --service-account-name <service_account_name> \ --delete-protection \ --maintenance schedule=<maintenance_type>,` `weekday=<day_of_week>,` `hour=<hour> \ --version <OpenSearch_version> \ --read-admin-password \ --data-transfer-access=<true_or_false> \ --serverless-access=<true_or_false> \ --plugins <OpenSearch_plugins> \ --advanced-params <additional_parameters> \ --opensearch-node-group name=<OpenSearch_host_group_name>,` `resource-preset-id=<host_class>,` `disk-size=<disk_size_in_bytes>,` `disk-type-id=<network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd>,` `hosts-count=<number_of_hosts_in_group>,` `zone-ids=<availability_zones>,` `subnet-names=<subnet_names>,` `assign-public-ip=<assign_public_address:_true_or_false>,` `roles=<host_roles> \ --dashboards-node-group name=<Dashboards_host_group_name>,` `resource-preset-id=<host_class>,` `disk-size=<disk_size_in_bytes>,` `disk-type-id=<network-ssd>,` `hosts-count=<number_of_hosts_in_group>,` `zone-ids=<availability_zones>,` `subnet-names=<subnet_names>,` `assign-public-ip=<assign_public_address:_true_or_false>
Where:
-
--labels
: Yandex Cloud labels labels in<key>=<value>
format. You can use them to logically separate resources. -
--environment
: Environment:production
: For stable versions of your apps.prestable
: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.
-
--service-account-name
: Name of the service account for access to Yandex Object Storage as a repository of OpenSearch snapshots. For more information on service accounts, see the Yandex Identity and Access Management documentation. -
--delete-protection
: Cluster protection from accidental deletion by a user. With deletion protection enabled, you still can connect to a cluster manually to delete data. -
--maintenance
: Maintenance window settings:- To allow maintenance at any time, do not specify the
--maintenance
parameter in the command (default configuration) or specify--maintenance schedule=anytime
. - To specify the preferred start time for maintenance, specify this parameter in the command:
--maintenance schedule=weekly,weekday=<day_of_week>,hour=<hour_in_UTC>
. In this case, maintenance will take place every week on a specified day at a specified time.
Both enabled and disabled clusters undergo maintenance. Maintenance may involve such operations as applying patches or updating DBMS's.
- To allow maintenance at any time, do not specify the
-
--read-admin-password
:admin
user password. If you specify this parameter in the command, it will prompt you to enter a password. -
--serverless-access
: Access from Yandex Serverless Containers,true
orfalse
. -
--plugins
: OpenSearch plugins to install in the cluster. -
--advanced-params
: Additional cluster parameters. The possible values are:max-clause-count
: Maximum allowed number of boolean clauses per query. For more information, see the OpenSearch documentation.fielddata-cache-size
: JVM heap size allocated for thefielddata
data structure. You can specify either an absolute value or percentage, e.g.,512mb
or50%
. For more information, see the OpenSearch documentation .reindex-remote-whitelist
: List of remote hosts whose indexes contain documents to copy for reindexing. Specify the parameter value as<host_address>:<port>
. If you need to specify more than one host, list values separated by commas. For more information, see the OpenSearch documentation .
-
--opensearch-node-group
:OpenSearch
host group configuration, where:-
--resource-preset-id
: Host class that defines the configuration of virtual machines the OpenSearch nodes will be deployed on. All available options are listed under Host classes. -
disk-size
: Disk size in bytes. Minimum and maximum values depend on the selected host class. -
disk-type-id
: Disk type. -
roles
: Host roles. The possible values include:data
: Assigns theDATA
role only.manager
: Assigns theMANAGER
role only.data+manager
ormanager+data
: Assigns both roles.
Tip
We do not recommend enabling public access to hosts with the
MANAGER
role, as this is not secure.
-
-
--dashboards-node-group
:Dashboards
host group configuration. It is configured in the same way as theOpenSearch
host group, except for the host roles. You do not need to configure any roles for theDashboards
group.
-
With Terraform
Terraform is distributed under the Business Source License
For more information about the provider resources, see the documentation on the Terraform
If you don't have Terraform, install it and configure the Yandex Cloud provider.
To create a Managed Service for OpenSearch cluster:
-
In the configuration file, describe the parameters of the resources you want to create:
-
DB cluster: Description of the Managed Service for OpenSearch cluster and its hosts
-
Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.
-
Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.
Here is an example of the configuration file structure:
resource "yandex_mdb_opensearch_cluster" "<cluster_name>" { name = "<cluster_name>" environment = "<environment>" network_id = "<network_ID>" security_group_ids = ["<list_of_security_group_IDs>"] deletion_protection = "<deletion_protection>" config { version = "<OpenSearch_version>" admin_password = "<admin_user_password>" opensearch { node_groups { name = "<virtual_host_group_name>" assign_public_ip = <public_access> hosts_count = <number_of_hosts> zone_ids = ["<list_of_availability_zones>"] subnet_ids = ["<list_of_subnet_IDs>"] roles = ["<role_list>"] resources { resource_preset_id = "<host_class>" disk_size = <storage_size_in_bytes> disk_type_id = "<disk_type>" } } plugins = ["<list_of_plugin_names>"] } dashboards { node_groups { name = "<virtual_host_group_name>" assign_public_ip = <public_access> hosts_count = <number_of_hosts> zone_ids = ["<list_of_availability_zones>"] subnet_ids = ["<list_of_subnet_IDs>"] resources { resource_preset_id = "<host_class>" disk_size = <storage_size_in_bytes> disk_type_id = "<disk_type>" } } } } maintenance_window { type = <maintenance_type> day = <day_of_week> hour = <hour> } } resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" } resource "yandex_vpc_subnet" "<subnet_name>" { name = "<subnet_name>" zone = "<availability_zone>" network_id = "<network_ID>" v4_cidr_blocks = ["<range>"] }
Where:
environment
: Environment,PRESTABLE
orPRODUCTION
.deletion_protection
: Deletion protection,true
orfalse
.assign_public_ip
: Public access to the host,true
orfalse
.roles
:DATA
andMANAGER
host roles.maintenance_window
: Maintenance window settings (including those for disabled clusters):type
: Maintenance type. The possible values include:ANYTIME
: Anytime.WEEKLY
: On a schedule.
day
: Day of the week inDDD
format for theWEEKLY
type, e.g.,MON
.hour
: Hour of the day inHH
format for theWEEKLY
type, e.g.,21
.
Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.
For a complete list of available Managed Service for OpenSearch cluster configuration fields, see the Terraform provider documentation
. -
-
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Create a cluster.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
Timeouts
The Terraform provider sets the following timeouts for Managed Service for OpenSearch cluster operations:
- Creating a cluster, including restoring from a backup: 30 minutes.
- Editing a cluster: 60 minutes.
- Deleting a cluster: 15 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the
timeouts
block to the cluster description, for example:resource "yandex_mdb_opensearch_cluster" "<cluster_name>" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } }
-
To create a Managed Service for OpenSearch cluster, use the create REST API method for the Cluster resource or the ClusterService/Create gRPC API call and provide the following in the request:
-
ID of the folder to host the cluster, in the
folderId
parameter. -
Cluster name in the
name
parameter. -
OpenSearch version in the
configSpec.version
parameter. -
admin
user password in theconfigSpec.adminPassword
parameter. -
Configuration of one or more groups of hosts with the
DATA
andMANAGER
(optional) roles in theconfigSpec.opensearchSpec.nodeGroups
parameter. -
Configuration of one or more groups of hosts with the
DASHBOARDS
role in theconfigSpec.dashboardsSpec.nodeGroups
parameter. -
List of plugins in the
configSpec.opensearchSpec.plugins
parameter. -
Settings for access from other services in the
configSpec.access
parameter. -
Network ID in the
networkId
parameter. -
Security group IDs in the
securityGroupIds
parameter. You may also need to set up security groups to connect to the cluster. -
ID of the service account used for cluster operations in the
serviceAccountId
parameter. -
Cluster deletion protection settings in the
deletionProtection
parameter.Enabled deletion protection will not prevent a manual connection with the purpose to delete database contents.
-
Maintenance window settings (including for disabled clusters) in the
maintenanceWindow
parameter.
Creating a cluster copy
You can create a OpenSearch cluster with the settings of another one you previously created. To do so, you need to import the configuration of the source OpenSearch cluster to Terraform. This way, you can either create an identical copy or use the imported configuration as the baseline and modify it as needed. Importing a configuration is a good idea when the source OpenSearch cluster has a lot of settings and you need to create a similar one.
To create an OpenSearch cluster copy:
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
In the same working directory, place a
.tf
file with the following contents:resource "yandex_mdb_opensearch_cluster" "old" { }
-
Write the ID of the initial OpenSearch cluster to the environment variable:
export OPENSEARCH_CLUSTER_ID=<cluster_ID>
You can request the ID with a list of clusters in the folder.
-
Import the settings of the initial OpenSearch cluster into the Terraform configuration:
terraform import yandex_mdb_opensearch_cluster.old ${OPENSEARCH_CLUSTER_ID}
-
Get the imported configuration:
terraform show
-
Copy it from the terminal and paste it into the
.tf
file. -
Place the file in the new
imported-cluster
directory. -
Modify the copied configuration so that you can create a new cluster from it:
- Specify the new cluster name in the
resource
string and thename
parameter. - Delete the
created_at
,health
,id
, andstatus
parameters. - Add the
admin_password
parameter to theconfig
section. - If the
maintenance_window
section hastype = "ANYTIME"
, delete thehour
parameter. - Optionally, make further changes if you need to customize the configuration.
- Specify the new cluster name in the
-
Get the authentication credentials in the
imported-cluster
directory. -
In the same directory, configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in the
imported-cluster
directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file. -
Check that the Terraform configuration files are correct:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
Timeouts
The Terraform provider sets the following timeouts for Managed Service for OpenSearch cluster operations:
- Creating a cluster, including restoring from a backup: 30 minutes.
- Editing a cluster: 60 minutes.
- Deleting a cluster: 15 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_opensearch_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
Examples
Create a Managed Service for OpenSearch cluster with the following test specifications:
-
Name:
my-os-clstr
. -
Description:
My OS cluster
. -
Label:
label-key
withlabel-value
. -
Environment:
production
. -
Network name:
default
. -
Security group ID:
enp6saqnq4ie244g67sb
. -
Service account name:
os-account
. -
Cluster deletion protection: Disabled.
-
Maintenance time: Every Monday from 13:00 till 14:00.
-
OpenSearch version:
2.8
. -
admin
user password: Specified after entering the cluster create command. -
Access to Data Transfer: Enabled.
-
Access to Serverless Containers: Enabled.
-
OpenSearch added plugin: analysis-icu.
-
OpenSearch additional parameter:
fielddata-cache-size=50%
. -
OpenSearch
node group configuration:- Group name:
os-group
. - Host class:
s2.micro
. - Disk size:
10737418240
(in bytes). - Disk type:
network-ssd
. - Number of hosts: Three.
- Availability zone:
ru-central1-a
. - Subnet:
default-ru-central1-a
. - Public address: Assigned.
- Host group roles:
DATA
andMANAGER
.
- Group name:
-
Dashboards
host group configuration:- Group name:
dashboard-group
. - Host class:
s2.micro
. - Disk size:
10737418240
(in bytes). - Disk type:
network-ssd
. - Number of hosts: One.
- Availability zone:
ru-central1-a
. - Subnet:
default-ru-central1-a
. - Public address: Assigned.
- Group name:
Run this command:
yc managed-opensearch cluster create \
--name my-os-clstr \
--description "My OS cluster" \
--labels label-key=label-value \
--environment production \
--network-name default \
--security-group-ids enp6saqnq4ie244g67sb \
--service-account-name os-account \
--delete-protection \
--maintenance schedule=weekly,`
`weekday=mon,`
`hour=14 \
--version 2.8 \
--read-admin-password \
--data-transfer-access=true \
--serverless-access=true \
--plugins analysis-icu \
--advanced-params fielddata-cache-size=50% \
--opensearch-node-group name=os-group,`
`resource-preset-id=s2.micro,`
`disk-size=10737418240,`
`disk-type-id=network-ssd,`
`hosts-count=3,`
`zone-ids=ru-central1-a,`
`subnet-names=default-ru-central1-a,`
`assign-public-ip=true,`
`roles=data+manager \
--dashboards-node-group name=dashboard-group,`
`resource-preset-id=s2.micro,`
`disk-size=10737418240,`
`disk-type-id=network-ssd,`
`hosts-count=1,`
`zone-ids=ru-central1-a,`
`subnet-names=default-ru-central1-a,`
`assign-public-ip=true
Create a Managed Service for OpenSearch cluster with the following test specifications:
- Name:
my-os-clstr
. - Environment:
PRODUCTION
. - OpenSearch version:
2.8
. admin
user password:osadminpwd
.OpenSearch
node group name:os-group
.- Host class:
s2.micro
. - Disk size:
10737418240
(in bytes). - Disk type:
network-ssd
. - Number of hosts:
1
. - Public address: Assigned.
- Host group roles:
DATA
andMANAGER
. - Maintenance time: Every Monday from 13:00 till 14:00.
- Network name:
mynet
. - Subnet name:
mysubnet
. - Availability zone:
ru-central1-a
. - Address range:
10.1.0.0/16
. - Security group name:
os-sg
. The security group enables connecting to the cluster host from any network (including the internet) on port9200
.
The configuration file for this cluster is as follows:
resource "yandex_mdb_opensearch_cluster" "my-os-clstr" {
name = "my-os-clstr"
environment = "PRODUCTION"
network_id = yandex_vpc_network.mynet.id
security_group_ids = [yandex_vpc_security_group.os-sg.id]
config {
version = "2.8"
admin_password = "osadminpwd"
opensearch {
node_groups {
name = "os-group"
assign_public_ip = true
hosts_count = 1
zone_ids = ["ru-central1-a"]
subnet_ids = [yandex_vpc_subnet.mysubnet.id]
roles = ["DATA", "MANAGER"]
resources {
resource_preset_id = "s2.micro"
disk_size = 10737418240
disk_type_id = "network-ssd"
}
}
}
}
maintenance_window {
type = "WEEKLY"
day = "MON"
hour = 14
}
}
resource "yandex_vpc_network" "mynet" {
name = "mynet"
}
resource "yandex_vpc_subnet" "mysubnet" {
name = "mysubnet"
zone = "ru-central1-a"
network_id = yandex_vpc_network.mynet.id
v4_cidr_blocks = ["10.1.0.0/16"]
}
resource "yandex_vpc_security_group" "os-sg" {
name = "os-sg"
network_id = yandex_vpc_network.mynet.id
ingress {
description = "Allow connections to the Managed Service for OpenSearch cluster from the Internet"
protocol = "TCP"
port = 9200
v4_cidr_blocks = ["0.0.0.0/0"]
}
egress {
description = "The rule allows all outgoing traffic"
protocol = "ANY"
v4_cidr_blocks = ["0.0.0.0/0"]
from_port = 0
to_port = 65535
}
}