Using the yandex-lemmer plugin in Yandex Managed Service for OpenSearch

Written by

Yandex Cloud

Updated at February 7, 2025

Getting started
Run a search without yandex-lemmer
Run a search with yandex-lemmer
Refine your search
Delete the resources you created

With yandex-lemmer, you can add a filter to improve text search in Russian across OpenSearch documents.

To test the plugin, compare standard search results to yandex-lemmer search results and then refine your filter.

After testing, delete the created resources if you no longer need them.

Getting started

Prepare the infrastructure:
Manually

Using Terraform
1. Create a Managed Service for OpenSearch cluster in desired configuration with the following settings:
  - yandex-lemmer and analysis-icu plugins enabled.
  - Public access to a group of hosts with the DATA role enabled.
2. If using security groups in your cluster, make sure they are configured correctly and allow connecting to the cluster.
1. If you do not have Terraform yet, install it.
2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.
4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
5. Download the opensearch-yandex-lemmer.tf configuration file to the same working directory. The file describes:
  - Network.
  - Subnet.
  - Security group and rules required to connect to a Managed Service for OpenSearch cluster.
  - Managed Service for OpenSearch cluster.
6. In the opensearch-yandex-lemmer.tf file, specify these variables:
  - version: OpenSearch version.
  - admin_password: OpenSearch admin password.
7. Check that the Terraform configuration files are correct using this command:
  terraform validate
  If there are any errors in the configuration files, Terraform will point them out.
8. Create the required infrastructure:
  1. Run the command to view planned changes:
    
    terraform plan
    
    If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
  2. If you are happy with the planned changes, apply them:
    
    Run the command:
    
    terraform apply
    
    Confirm the update of resources.
    
    Wait for the operation to complete.
  All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.
Install an SSL certificate.

Check the connection to the cluster using cURL:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --request GET 'https://<FQDN_of_the_OpenSearch_host_with_the_DATA_role>:9200/'

You can obtain the host FQDN with a list of hosts in the cluster.

A message like this is displayed if the connection is successful:

{
  "name" : "....mdb.yandexcloud.net",
  "cluster_name" : "...",
  "cluster_uuid" : "...",
  "version" : {
  "distribution" : "opensearch",
  ...
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Run a search without yandex-lemmer

Add a document to the index named simple-index.

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/simple-index/_doc?pretty' \
    --data '{
              "book": "The Night It Rained",
              "author": "Eugenia Riley"
            }'

simple-index will be created automatically when you create the document. By default, indexes use the built-in text analyzer named Standard.

Perform this search:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/simple-index/_search?pretty' \
    --data '{
              "query": {
                "query_string": {
                  "query": "book: when it rains"
                }
              }
            }'

The result contains the following lines:

...
"hits" : {
  "total" : {
    "value" : 0,
    "relation" : "eq"
  },
  "max_score" : null,
  "hits" : [ ]
}
...

The book was not found because of mismatch between the source document and the search query: it rained vs. it rains.

Run a search with yandex-lemmer

Create an index named lemmer-index, specify in it the lemmer analyzer parameters, and connect it for the book field:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request PUT "https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index?pretty" \
    --data '{
              "mappings": {
                "properties": {
                  "book": {
                    "type": "text",
                    "analyzer": "lemmer"
                  }
                }
              },
              "settings": {
                "analysis": {
                  "analyzer": {
                    "lemmer": {
                      "type": "custom",
                      "tokenizer": "standard",
                      "filter": [
                        "yandex_lemmer"
                      ]
                    }
                  }
                }
              }
            }'

Add a document to lemmer-index:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index/_doc?pretty' \
    --data '{
              "book": "The Night It Rained",
              "author": "Eugenia Riley"
            }'

Perform this search:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index/_search?pretty' \
    --data '{
              "query": {
                "query_string": {
                  "query": "book: when it rains"
                }
              }
            }'

The result contains the following lines:

...
"hits" : {
  "total" : {
    "value" : 1,
    "relation" : "eq"
  },
  "max_score" : 0.9993168,
  "hits" : [
    {
      "_index" : "lemmer-index",
      "_id" : "zoGb_ZIB-3clfLPNjPbo",
      "_score" : 0.9993168,
      "_source" : {
        "book" : "The Night It Rained",
        "author" : "Eugenia Riley"
      }
    }
  ]
}
...

The book was found.

To see how the lemmer analyzer works, run this command:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/lemmer-index/_analyze?pretty' \
    --data '{"field":"book","text":"it rained"}'

Result:

{
  "tokens" : [
    {
      "token" : "rained",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "rain",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "it",
      "start_offset" : 4,
      "end_offset" : 9,
      "type" : "<ALPHANUM>",
      "position" : 1
    }
  ]
}

After getting through the yandex_lemmer filter, the rained token transforms into two tokens:

rained: Source token.
rain: Root form of rained.

After getting through the yandex_lemmer filter, the it token remains unchanged because the source token matches the root form of the word.

Refine your search

To refine a text search, add more filters to yandex-lemmer:

nfc: Performs NFC normalization (text mapping), where the same letter is represented by different characters. This filter is available if the analysis-icu plugin is enabled.
lowercase: Converts text to lower case.
yo_ye: Repalces the letter "ё" with "е". yandex-lemmer does not distinguish between these letters (it will find both ёлка and елка), but such a replacement allows you to improve the percentage of analyzer cache hits and save some disk space.

To use additional filters:

Create an index named index-with-filters and add into it the yandex-lemmer filter and additional filters for the lemmer-improved analyzer:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request PUT "https://<address_of_OpenSearch_host_with_DATA_role>:9200/index-with-filters?pretty" \
    --data '{
              "mappings": {
                "properties": {
                  "book": {
                    "type": "text",
                    "analyzer": "lemmer-improved"
                  }
                }
              },
              "settings": {
                "analysis": {
                  "filter": {
                    "nfc": {
                      "type": "icu_transform",
                      "id": "NFC"
                    },
                    "yo_ye": {
                      "type": "pattern_replace",
                      "pattern": "ё",
                      "replacement": "е"
                    }
                  },
                  "analyzer": {
                    "lemmer-improved": {
                      "type": "custom",
                      "tokenizer": "standard",
                      "filter": [
                        "nfc",
                        "lowercase",
                        "yo_ye",
                        "yandex_lemmer"
                      ]
                    }
                  }
                }
              }
            }'

Add a document to the index:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request POST "https://<address_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/index-with-filters/_doc?pretty" \
    --data '{
              "book": "Чёрный тюльпан",
              "author": "Александр Дюма"
            }'

Here, the letter "и" (code 1048) and the "˘" breve (code 774) are used instead of "й".

Perform this search:

curl \
    --user admin:<password> \
    --cacert ~/.opensearch/root.crt \
    --header 'Content-Type: application/json' \
    --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/index-with-filters2/_search?pretty' \
    --data '{
              "query": {
                "query_string": {
                  "query": "book: черный тюльпан"
                }
              }
            }'

Here, the book name includes the letter "й" (code 1049).

The result contains the following lines:

...
"hits" : {
  "total" : {
    "value" : 1,
    "relation" : "eq"
  },
  "max_score" : 0.2876821,
  "hits" : [
    {
      "_index" : "index-with-filters2",
      "_id" : "34EO_pIB-3clfLPNvvbY",
      "_score" : 0.2876821,
      "_source" : {
        "book" : "Чёрный тюльпан",
        "author" : "Александр Дюма"
      }
    }
  ]
}
...

The book was found.

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Manually

Using Terraform

Delete the Managed Service for OpenSearch cluster.

In the terminal window, go to the directory containing the infrastructure plan.

Warning

Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.
Delete resources:
1. Run this command:
```
terraform destroy
```
2. Confirm deleting the resources and wait for the operation to complete.
All the resources described in the Terraform manifests will be deleted.

Using the yandex-lemmer plugin in Yandex Managed Service for OpenSearch

Getting startedGetting started

Run a search without yandex-lemmerRun a search without yandex-lemmer

Run a search with yandex-lemmerRun a search with yandex-lemmer

Refine your searchRefine your search

Delete the resources you createdDelete the resources you created

Was the article helpful?

Getting started

Run a search without yandex-lemmer

Run a search with yandex-lemmer

Refine your search

Delete the resources you created