Creating an OpenSearch cluster
A Managed Service for OpenSearch cluster is a group of multiple linked OpenSearch and dashboardsDATA
role. To learn more about roles in the cluster, see Host roles.
Available disk types depend on the selected host class.
For more information, see Resource relationships in Managed Service for OpenSearch.
Creating a cluster
When creating a cluster, you need to specify individual parameters for each host group.
To create a cluster:
-
In the management console
, select the folder where you want to create a cluster. -
Select Managed Service for OpenSearch.
-
Click Create cluster.
-
Under Basic parameters:
-
Enter a name for the cluster. It must be unique within the folder.
-
(Optional) Enter a cluster description.
-
Select the environment where you want to create the cluster (you cannot change the environment once the cluster is created):
PRODUCTION
: For stable versions of your apps.PRESTABLE
: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.
-
Select the OpenSearch version.
-
Select the plugins you want to install in the cluster.
-
-
Under Network settings, select the cloud network to host the cluster in and security groups for cluster network traffic. You may also need to set up security groups to connect to the cluster.
-
Under Virtual node group 1, configure the
OpenSearch
host group:-
Select the host group type:
OpenSearch
-
Enter a name for the host group, which must be unique within the cluster.
-
Select the host roles
DATA
andMANAGER
. -
Select the platform, host type, and host class.
The host class defines the technical characteristics of virtual machines that OpenSearch nodes are deployed on. All available options are listed under Host classes.
-
Select the disk type and data storage size.
The selected type determines the increments in which you can change your disk size:
- Network HDD and SSD storage: In 1 GB increments.
- Local SSD storage:
- For Intel Cascade Lake: In increments of 100 GB.
- For Intel Ice Lake: In 368 GB increments.
- Non-replicated SSD storage: In increments of 93 GB.
-
Specify how hosts should be distributed across availability zones and subnets.
-
Select the number of hosts to create.
-
Enable Public access if you want to allow connecting to hosts over the internet.
Tip
It is not recommended to enable public access for hosts with the
MANAGER
role, because this might be unsafe.
Warning
After creating your cluster, you can only change the host configuration using the API. However, you can also create a new host group with a different configuration if needed.
-
-
If needed, configure the
Dashboards
host group under Virtual node group 2:-
Select the platform, host type, and host class.
-
Set up storage in the same way as for
OpenSearch
hosts. -
Specify how hosts should be distributed across availability zones and subnets.
-
Select the number of hosts to create.
-
Enable Public access if you want to allow connecting to hosts over the internet.
Tip
You can use OpenSearch Dashboards even if you can't request public access to the hosts (for example, for security reasons). To do this, proxy the connections via the virtual machine in Yandex Compute Cloud that is hosted in the same network as the cluster. For more information, see Connecting to OpenSearch Dashboards.
-
-
If required, click Add virtual node group to add another host group or more.
-
Under Service settings:
-
Enter the password for the
admin
user.This is a special user that is required for managing clusters and cannot be deleted. It is assigned the
superuser
role and can perform any operations on clusters.Tip
This user is not intended for routine jobs; for those, we recommend creating regular users. For more information, see Managing OpenSearch users.
-
Configure additional cluster settings, if required:
-
Maintenance window: Maintenance window settings:
- To enable maintenance at any time, select arbitrary (default).
- To specify the preferred maintenance start time, select by schedule and specify the desired day of the week and UTC hour. For example, you can choose a time when the cluster is least loaded.
Maintenance operations are carried out both on enabled and disabled clusters. They may include updating the DBMS, applying patches, and so on.
-
Service account is an account to access Yandex Object Storage as a repository of OpenSearch snapshots. For more detail on service accounts, see the Yandex Identity and Access Management documentation.
-
Deletion protection: Manages cluster protection from accidental deletion by a user.
Cluster deletion protection will not prevent a manual connection to a cluster to delete data.
-
-
-
Click Create cluster.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To create a cluster:
-
View a description of the create cluster CLI command:
yc managed-opensearch cluster create --help
-
Specify cluster parameters in the create command (the list of supported parameters in the example is not exhaustive):
yc managed-opensearch cluster create \ --name <cluster_name> \ --description <cluster_description> \ --labels <labels> \ --environment <environment:_production_or_prestable> \ --network-name <network_name> \ --security-group-ids <security_group_IDs> \ --service-account-name <service_account_name> \ --delete-protection <deletion_protection:_true_or_false> \ --maintenance schedule=<maintenance_type>,` `weekday=<day_of_week>,` `hour=<hour_of_day> \ --version <OpenSearch_version> \ --read-admin-password \ --data-transfer-access=<true_or_false> \ --serverless-access=<true_or_false> \ --plugins <OpenSearch_plugins> \ --advanced-params <advanced_parameters> \ --opensearch-node-group name=<name_of_OpenSearch_host_group>,` `resource-preset-id=<host_class>,` `disk-size=<disk_size_in_bytes>,` `disk-type-id=<disk_type>,` `hosts-count=<number_of_hosts_per_group>,` `zone-ids=<availability_zones>,` `subnet-names=<subnet_names>,` `assign-public-ip=<assign_public_IP:_true_or_false>,` `roles=<host_roles> \ --dashboards-node-group name=<name_of_Dashboards_host_group>,` `resource-preset-id=<host_class>,` `disk-size=<disk_size_in_bytes>,` `disk-type-id=<disk_type>,` `hosts-count=<number_of_hosts_per_group>,` `zone-ids=<availability_zones>,` `subnet-names=<subnet_names>,` `assign-public-ip=<assign_public_IP:_true_or_false>
Where:
-
--labels
: Yandex Cloud labels in<key>=<value>
format. You can use them to logically separate resources. -
--environment
: Environment:production
: For stable versions of your applications.prestable
: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.
-
--service-account-name
: Name of the service account. -
--delete-protection
: Cluster protection against accidental deletion by a user,true
orfalse
. Cluster deletion protection will not prevent a manual connection to a cluster to delete data. -
--maintenance
: Maintenance time settings:- To allow maintenance at any time, do not specify the
--maintenance
parameter in the command (default configuration) or specify--maintenance schedule=anytime
. - To specify the preferred start time for maintenance, specify the
--maintenance schedule=weekly,weekday=<day_of_week>,hour=<hour_in_UTC>
parameter in the command. In this case, maintenance will take place every week on a specified day at a specified time.
Both enabled and disabled clusters undergo maintenance. Maintenance may involve such operations as applying patches or updating DBMS's.
- To allow maintenance at any time, do not specify the
-
--read-admin-password
:admin
user password. If you specify this parameter in the command, it will prompt you to enter a password. -
--data-transfer-access
: Access from Yandex Data Transfer,true
orfalse
. -
--serverless-access
: Access from Yandex Serverless Containers,true
orfalse
. -
--plugins
: OpenSearch plugins you want to install in the cluster. -
--advanced-params
: Additional cluster parameters. The possible values include:max-clause-count
: Maximum allowed number of boolean clauses per query. See more in the OpenSearch documentation .fielddata-cache-size
: Amount of JVM heap memory allocated for the fielddata data structure. You can specify either an absolute value or percentage, e.g.,512mb
or50%
. For more details, see the OpenSearch documentation .reindex-remote-whitelist
: List of remote hosts whose indexes contain documents to copy for reindexing. Specify the parameter value as<host_address>:<port>
. If you need to specify more than one host, list values separated by commas. For more details, see the OpenSearch documentation .
-
--opensearch-node-group
:OpenSearch
host group configuration, where:-
--resource-preset-id
: Host class that defines the configuration of virtual machines the OpenSearch nodes will be deployed on. All available options are listed under Host classes. -
disk-size
: Disk size in bytes. Minimum and maximum values depend on the selected host class. -
disk-type-id
: Disk type. -
roles
: Host roles. The possible values include:data
: Assigns theDATA
role only.manager
: Assigns theMANAGER
role only.data+manager
ormanager+data
: Assigns both roles.
Tip
We do not recommend enabling public access to hosts with the
MANAGER
role, as this is not secure.
-
-
--dashboards-node-group
:Dashboards
host group configuration. It is configured in the same way as theOpenSearch
host group, except for the host roles. You do not need to configure any roles for theDashboards
group.
-
Terraform
For more information about the provider resources, see the documentation on the Terraform
If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
To create a Managed Service for OpenSearch cluster:
-
In the configuration file, describe the parameters of the resources you want to create:
-
DB cluster: Description of the Managed Service for OpenSearch cluster and its hosts
-
Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.
-
Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.
Here is an example of the configuration file structure:
resource "yandex_mdb_opensearch_cluster" "<cluster_name>" { name = "<cluster_name>" environment = "<environment>" network_id = "<network_ID>" security_group_ids = ["<list_of_security_group_IDs>"] deletion_protection = "<deletion_protection>" config { version = "<OpenSearch_version>" admin_password = "<admin_user_password>" opensearch { node_groups { name = "<virtual_host_group_name>" assign_public_ip = <public_access> hosts_count = <number_of_hosts> zone_ids = ["<list_of_availability_zones>"] subnet_ids = ["<list_of_subnet_IDs>"] roles = ["<list_of_roles>"] resources { resource_preset_id = "<host_class>" disk_size = <storage_size_in_bytes> disk_type_id = "<disk_type>" } } plugins = ["<list_of_plugin_names>"] } dashboards { node_groups { name = "<virtual_host_group_name>" assign_public_ip = <public_access> hosts_count = <number_of_hosts> zone_ids = ["<list_of_availability_zones>"] subnet_ids = ["<list_of_subnet_IDs>"] resources { resource_preset_id = "<host_class>" disk_size = <storage_size_in_bytes> disk_type_id = "<disk_type>" } } } } } resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" } resource "yandex_vpc_subnet" "<subnet_name>" { name = "<subnet_name>" zone = "<availability_zone>" network_id = "<network_ID>" v4_cidr_blocks = ["<range>"] }
Where:
environment
: Environment,PRESTABLE
orPRODUCTION
.deletion_protection
: Deletion protection,true
orfalse
.assign_public_ip
: Public access to the host,true
orfalse
.roles
:DATA
andMANAGER
host roles.
Cluster deletion protection will not prevent a manual connection to delete the contents of a database.
To set up the maintenance window (for disabled clusters as well), add the
maintenance_window
block to the cluster description:resource "yandex_mdb_opensearch_cluster" "<cluster_name>" { ... maintenance_window { type = <maintenance_type> day = <day_of_week> hour = <hour> } ... }
Where:
type
: Maintenance type. The possible values include:anytime
: Anytime.weekly
: By schedule.
day
: Day of the week for theweekly
type inDDD
format, e.g.,MON
.hour
: Hour of the day for theweekly
type in theHH
format, e.g.,21
.
For a complete list of available Managed Service for OpenSearch cluster configuration fields, see the Terraform provider documentation
. -
-
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Create a cluster.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
Timeouts
The Terraform provider sets the following timeouts for Managed Service for OpenSearch cluster operations:
- Creating a cluster, including restoring from a backup: 30 minutes.
- Editing a cluster: 60 minutes.
- Deleting a cluster: 15 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?Add the
timeouts
block to the cluster description, for example:resource "yandex_mdb_opensearch_cluster" "<cluster_name>" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } }
-
To create a cluster, use the create REST API method for the Cluster resource or the ClusterService/Create gRPC API call and provide the following in the request:
-
ID of the folder where the cluster should be placed, in the
folderId
parameter. -
Cluster name in the
name
parameter. -
OpenSearch version in the
configSpec.version
parameter. -
admin
user password in theconfigSpec.adminPassword
parameter. -
Configuration of one or more groups of hosts with the
DATA
andMANAGER
(optional) roles in theconfigSpec.opensearchSpec.nodeGroups
parameter. -
Configuration of one or more groups of hosts with the
DASHBOARDS
role in theconfigSpec.dashboardsSpec.nodeGroups
parameter. -
List of plugins in the
configSpec.opensearchSpec.plugins
parameter. -
Settings for access from other services in the
configSpec.access
parameter. -
Network ID in the
networkId
parameter. -
Security group identifiers in the
securityGroupIds
parameter. You may also need to set up security groups to connect to the cluster. -
ID of the service account used for cluster operations in the
serviceAccountId
parameter. -
Cluster deletion protection settings in the
deletionProtection
parameter.Cluster deletion protection will not prevent a manual connection to delete the contents of a database.
-
Settings for the maintenance window (including those for disabled clusters) in the
maintenanceWindow
parameter.
Creating a cluster copy
You can create an OpenSearch cluster with the settings of another cluster created earlier. To do so, you need to import the configuration of the source OpenSearch cluster to Terraform. Thus you can either create an identical copy or use the imported configuration as the baseline and modify it as needed. Importing is a convenient option when the source OpenSearch cluster has lots of settings and you need to create a similar one.
To create an OpenSearch cluster copy:
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
In the same working directory, place a
.tf
file with the following contents:resource "yandex_mdb_opensearch_cluster" "old" { }
-
Write the ID of the initial OpenSearch cluster to the environment variable:
export OPENSEARCH_CLUSTER_ID=<cluster_ID>
You can request the ID with a list of clusters in the folder.
-
Import the settings of the initial OpenSearch cluster into the Terraform configuration:
terraform import yandex_mdb_opensearch_cluster.old ${OPENSEARCH_CLUSTER_ID}
-
Get the imported configuration:
terraform show
-
Copy it from the terminal and paste it into the
.tf
file. -
Place the file in the new
imported-cluster
directory. -
Modify the copied configuration so that you can create a new cluster from it:
- Specify a new cluster name in the
resource
string and thename
parameter. - Delete the
created_at
,health
,id
, andstatus
parameters. - In the
config
section, add theadmin_password
parameter. - If the
maintenance_window
section specifies thetype = "ANYTIME"
parameter value, delete thehour
parameter. - (Optional) Make further modifications if you need a customized copy rather than identical one.
- Specify a new cluster name in the
-
In the
imported-cluster
directory, get the authentication data. -
In the same directory, configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in the
imported-cluster
directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file. -
Check that the Terraform configuration files are correct:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
Timeouts
The Terraform provider sets the following timeouts for Managed Service for OpenSearch cluster operations:
- Creating a cluster, including restoring from a backup: 30 minutes.
- Editing a cluster: 60 minutes.
- Deleting a cluster: 15 minutes.
Operations exceeding the set timeout are interrupted.
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_opensearch_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
Examples
Create a Managed Service for OpenSearch cluster with the following test characteristics:
-
Name:
my-os-clstr
. -
Description:
My OS cluster
. -
Label:
label-key
with thelabel-value
value. -
Environment:
production
. -
Network name:
default
. -
Security group ID:
enp6saqnq4ie244g67sb
. -
Service account name:
os-account
. -
Cluster deletion protection: Disabled.
-
Maintenance time: Every Monday from 13:00 till 14:00.
-
OpenSearch version:
2.8
. -
admin
user password: To be set after entering the cluster creation command. -
Access to Data Transfer: Enabled.
-
Access to Serverless Containers: Enabled.
-
OpenSearch added plugin: analysis-icu.
-
OpenSearch advanced parameter:
fielddata-cache-size=50%
. -
OpenSearch
node group configuration:- Group name:
os-group
- Host class:
s2.micro
- Disk size:
10737418240
(in bytes) - Disk type:
network-ssd
- Number of hosts: Three
- Availability zone:
ru-central1-a
- Subnet:
default-ru-central1-a
- Public address: Assigned
- Host group roles:
DATA
andMANAGER
- Group name:
-
Configuration of the
Dashboards
host group:- Group name:
dashboard-group
- Host class:
s2.micro
- Disk size:
10737418240
(in bytes) - Disk type:
network-ssd
- Number of hosts: One
- Availability zone:
ru-central1-a
- Subnet:
default-ru-central1-a
- Public address: Assigned
- Group name:
Run this command:
yc managed-opensearch cluster create \
--name my-os-clstr \
--description "My OS cluster" \
--labels label-key=label-value \
--environment production \
--network-name default \
--security-group-ids enp6saqnq4ie244g67sb \
--service-account-name os-account \
--delete-protection=false \
--maintenance schedule=weekly,`
`weekday=mon,`
`hour=14 \
--version 2.8 \
--read-admin-password \
--data-transfer-access=true \
--serverless-access=true \
--plugins analysis-icu \
--advanced-params fielddata-cache-size=50% \
--opensearch-node-group name=os-group,`
`resource-preset-id=s2.micro,`
`disk-size=10737418240,`
`disk-type-id=network-ssd,`
`hosts-count=3,`
`zone-ids=ru-central1-a,`
`subnet-names=default-ru-central1-a,`
`assign-public-ip=true,`
`roles=data+manager \
--dashboards-node-group name=dashboard-group,`
`resource-preset-id=s2.micro,`
`disk-size=10737418240,`
`disk-type-id=network-ssd,`
`hosts-count=1,`
`zone-ids=ru-central1-a,`
`subnet-names=default-ru-central1-a,`
`assign-public-ip=true
Create a Managed Service for OpenSearch cluster with the following test characteristics:
- Name:
my-os-clstr
. - Environment:
PRODUCTION
. - OpenSearch version:
2.8
. admin
password:osadminpwd
.OpenSearch
node group name:os-group
.- Host class:
s2.micro
. - Disk size:
10737418240
(in bytes). - Disk type:
network-ssd
. - Number of hosts:
1
. - Public address: Assigned.
- Host group roles:
DATA
andMANAGER
. - Network name:
mynet
. - Subnet name:
mysubnet
. - Availability zone:
ru-central1-a
. - Address range:
10.1.0.0/16
. - Security group name:
os-sg
. The security group allows connecting to the cluster host from any network (including the internet) on port9200
.
The configuration file for this cluster is as follows:
resource "yandex_mdb_opensearch_cluster" "my-os-clstr" {
name = "my-os-clstr"
environment = "PRODUCTION"
network_id = yandex_vpc_network.mynet.id
security_group_ids = [yandex_vpc_security_group.os-sg.id]
config {
version = "2.8"
admin_password = "osadminpwd"
opensearch {
node_groups {
name = "os-group"
assign_public_ip = true
hosts_count = 1
zone_ids = ["ru-central1-a"]
subnet_ids = [yandex_vpc_subnet.mysubnet.id]
roles = ["DATA", "MANAGER"]
resources {
resource_preset_id = "s2.micro"
disk_size = 10737418240
disk_type_id = "network-ssd"
}
}
}
}
}
resource "yandex_vpc_network" "mynet" {
name = "mynet"
}
resource "yandex_vpc_subnet" "mysubnet" {
name = "mysubnet"
zone = "ru-central1-a"
network_id = yandex_vpc_network.mynet.id
v4_cidr_blocks = ["10.1.0.0/16"]
}
resource "yandex_vpc_security_group" "os-sg" {
name = "os-sg"
network_id = yandex_vpc_network.mynet.id
ingress {
description = "Allow connections to the Managed Service for OpenSearch cluster from the Internet"
protocol = "TCP"
port = 9200
v4_cidr_blocks = ["0.0.0.0/0"]
}
egress {
description = "The rule allows all outgoing traffic"
protocol = "ANY"
v4_cidr_blocks = ["0.0.0.0/0"]
from_port = 0
to_port = 65535
}
}