Discuss with expertTry it for free
© 2026 Direct Cursus Technology L.L.C.

Using Yandex Managed Service for Valkey™ with the Search module and Yandex AI Studio models for vector search in a RAG system

Written by
View in Markdown

RAG (retrieval-augmented generation) is an approach where a language model uses information retrieved from a database to answer a user's query.

This tutorial provides an example of a RAG system that uses Yandex Managed Service for Valkey™ to store data about people and perform vector search on that data using the Search module. Yandex AI Studio models are used to generate an embedding for the search query and a response. Yandex Managed Service for Valkey™ and AI Studio work together using MCP.

When processing a search query, the system components work together as follows:

  1. The Yandex AI Studio model receives the user's query and calls an MCP tool to search for data.
  2. The MCP tool converts the query into an embedding using the Yandex AI Studio vector representation model and then sends the embedding to Yandex Managed Service for Valkey™.
  3. Yandex Managed Service for Valkey™ runs a vector search using the Search module and returns the results to the MCP tool, which then sends them to the language model.
  4. The language model analyzes the retrieved data and responds to the user.

This example requires a Unix-like environment and a Yandex Managed Service for Valkey™ cluster running version 9.0 or later.

The source code for the RAG system example is available in the SourceCraft repository.

To set up your RAG system, follow these steps:

  1. Set up your infrastructure.
  2. Set up a testing environment.
  3. Prepare your test data and upload them to Yandex Managed Service for Valkey™.
  4. Test the test data in Yandex Managed Service for Valkey™.
  5. Run a search in the RAG system.

If you no longer need the resources you created, delete them.

Getting started

Sign up for Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or create a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Set up your infrastructure

  1. Create a cloud network named demo-network.

    When creating a network, three subnets in different availability zones are created automatically.

  2. In demo-network, create a security group named valkey-sg for the Yandex Managed Service for Valkey™ cluster and add the following inbound rule to the group to allow internet access to the cluster:

    • Port range: 6380.
    • Protocol: TCP.
    • Destination name: Address range.
    • IPv4 CIDR: 0.0.0.0/0.

  3. Create a Yandex Managed Service for Valkey™ cluster of any suitable configuration with the following settings:

    • Version: 9.0 or higher.
    • Network: demo-network.
    • Security group: valkey-sg.
    • TLS support: Enabled.

    When creating your cluster, enable host public access and add the Search module.

    Note

    Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. Instead of manually creating the provider configuration file, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you have not added the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the valkey-rag-search.tf configuration file to your current working directory. This file describes:

    • Network.

    • Subnets.

    • Security group.

    • Yandex Managed Service for Valkey™ cluster with public access to its hosts and the Search module.

      Note

      Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.

  6. Specify the following in the configuration file:

    • network_name: Network name.
    • sg_name: Security group name.
    • valkey_cluster_name: Cluster name.
    • valkey_password: Password to access the cluster.

  7. Validate your Terraform configuration using this command:

    terraform validate

    Terraform will display any configuration errors detected in your files.

  8. Create the required infrastructure:

    1. Run this command to view the intended changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Set up a testing environment

  1. Install Python 3.11 or higher.

  2. Install Git.

  3. Create a service account named sa-ai and assign the ai.languageModels.user role to it for using Yandex AI Studio models.

  4. Create an API key for the sa-ai service account.

  5. Clone the SourceCraft repository with the example source code to your local computer:

    git clone https://git@git.sourcecraft.dev/valkey/webinar-260624-rag-example.git

  6. Navigate to the repository directory:

    cd webinar-260624-rag-example

  7. Set up the example environment:

    1. Install the dependencies:

      make install

    2. Activate the virtual environment:

      source .venv/bin/activate

  8. Configure the Yandex Managed Service for Valkey™ and Yandex AI Studio access settings:

    1. Use the .env.example template to create a file named .env:

      cp .env.example .env

    2. Open the .env file in a text editor, such as vim:

      vim .env

    3. In the file, specify the following settings:

      Sample file:

      # Yandex AI Studio credentials
# Get an API key: https://yandex.cloud/en/docs/ai-studio/operations/get-api-key
YDS_API_KEY=AQVN1TpRw9tkjhKtj6R0mRgmHLvMsj84********
YDS_FOLDER_ID=b1g4unjqq856********

# Valkey auth (leave empty for local unauthenticated Valkey)
VALKEY_USERNAME=default
VALKEY_PASSWORD=********

  9. Configure the Yandex Managed Service for Valkey™ cluster's connection settings:

    1. Open the config.yaml file in a text editor, such as vim:

      vim config.yaml

    2. Under valkey, specify the following:

      Here is an example of the valkey section:

      # Valkey connection settings
valkey:
  host: "c-c9qlvmr5h2g9********.rw.mdb.yandexcloud.net"
  port: 6380
  db: 0
  # username + password read from .env: VALKEY_USERNAME, VALKEY_PASSWORD
  # SSL toggle for managed Valkey
  ssl: true

    In addition to the Yandex Managed Service for Valkey™ cluster's connection settings, the config.yaml file also contains settings for the following:

    • Yandex AI Studio models.
    • Index and vector search in Yandex Managed Service for Valkey™.
    • MCP server.
    • Orchestrator.
    • Language model's system prompt.
    • Logging.

    To make this example work, you only need to configure the Yandex Managed Service for Valkey™ cluster's connection settings. You can leave the other settings unchanged.

Prepare your test data and upload it to Yandex Managed Service for Valkey™

Run the following commands in the webinar-260624-rag-example repository directory:

  1. Create a file named data/people.json with your test data:

    rag-demo generate -n 10

    Result:

    Wrote 10 people to data/people.json

  2. Upload the test data to Yandex Managed Service for Valkey™:

    rag-demo index

    This will create the idx_people index for your test data in the Yandex Managed Service for Valkey™ cluster.

    Result:

      indexed person 1: Alice Smith
  indexed person 2: John Doe
  indexed person 3: Maria Garcia
  indexed person 4: Wei Zhang
  indexed person 5: James Brown
  indexed person 6: Priya Patel
  indexed person 7: Lars Nilsson
  indexed person 8: Olivia Wilson
  indexed person 9: Hassan Ali
  indexed person 10: Sophie Martin
Loaded 10 people into index idx_people

Test the test data in Yandex Managed Service for Valkey™

  1. Connect to the Yandex Managed Service for Valkey™ cluster database:

    1. Navigate to the directory you want to download the Valkey™ distribution to.

    2. Download the correct Valkey™ version, then build and install it with TLS support:

      wget https://github.com/valkey-io/valkey/archive/refs/tags/<version>.tar.gz && \
tar -xzvf <version>.tar.gz && \
cd valkey-<version> && \
make BUILD_TLS=yes && \
sudo make install && \
sudo cp ./src/valkey-cli /usr/bin/

    3. Obtain an SSL certificate:

      mkdir -p ~/.redis && \
wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document ~/.redis/YandexInternalRootCA.crt && \
chmod 0655 ~/.redis/YandexInternalRootCA.crt

      The certificate will be saved to the ~/.redis/YandexInternalRootCA.crt file.

    4. Connect to the cluster:

      • For non-sharded clusters, run this command:

        valkey-cli \
    -h c-<cluster_ID>.rw.mdb.yandexcloud.net \
    -a <password> \
    -p 6380 \
    --tls \
    --cacert ~/.redis/YandexInternalRootCA.crt

        You can get the cluster ID with the list of clusters in the folder.

      • For sharded clusters, run this command:

        valkey-cli \
    -h <FQDN_of_master_host_in_target_shard> \
    -c \
    -a <password> \
    -p 6380 \
    --tls \
    --cacert ~/.redis/YandexInternalRootCA.crt

        To learn how to get a host’s FQDN, see FQDNs of Valkey™ hosts.

    1. Install redis-cli:

      sudo apt update && sudo apt install --yes redis-tools

    2. Obtain an SSL certificate:

      mkdir -p ~/.redis && \
wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document ~/.redis/YandexInternalRootCA.crt && \
chmod 0655 ~/.redis/YandexInternalRootCA.crt

      The certificate will be saved to the ~/.redis/YandexInternalRootCA.crt file.

    3. Connect to the cluster:

      • For non-sharded clusters, run this command:

        redis-cli \
    -h c-<cluster_ID>.rw.mdb.yandexcloud.net \
    -a <password> \
    -p 6380 \
    --tls \
    --cacert ~/.redis/YandexInternalRootCA.crt

        You can get the cluster ID with the list of clusters in the folder.

      • For sharded clusters, run this command:

        redis-cli \
    -h <FQDN_of_master_host_in_target_shard> \
    -c \
    -a <password> \
    -p 6380 \
    --tls \
    --cacert ~/.redis/YandexInternalRootCA.crt

        To learn how to get a host’s FQDN, see FQDNs of Valkey™ hosts.

    Before connecting, enable Yandex WebSQL access in the cluster settings.

    To connect to a database using Yandex WebSQL:

    1. In the management console, select the folder containing the cluster.
    2. Navigate to Yandex Managed Service for Valkey™.
    3. Click the name of your cluster and select the WebSQL tab.
    4. Click Go to WebSQL in the top-right corner of the page.

  2. Get information about the idx_people index:

    FT.INFO idx_people

    Verify the parameter values in the output:

    • num_docs: 10.
    • state: ready.

  3. Get a list of keys in the cluster:

    KEYS *

    Result:

    1) "person:1"
2) "person:2"
3) "person:3"
4) "person:4"
5) "person:5"
6) "person:9"
7) "person:id_counter"
8) "person:6"
9) "person:7"
10) "person:8"
11) "person:10"

    Where person:id_counter is the service key that stores the ID counter value.

  4. View the person:1 key data:

    HGETALL person:1

Run the following commands in the webinar-260624-rag-example repository directory:

  1. Run a search for Who works with Kubernetes?:

    rag-demo ask "Who works with Kubernetes?"

    Result:

    Lars Nilsson works with Kubernetes. He is a site reliability engineer in Stockholm, Sweden, responsible for keeping large Kubernetes clusters healthy.

    Along with the model's response, the output shows detailed information on the query execution. To hide this information, specify console_level: "WARNING" in the logging section of the config.yaml file. The output will then include a summary of the query execution and the model's final response.

  2. Run a search for Who is into running?:

    rag-demo ask "Who is into running?"

    Result:

    Among the people found, **Wei Zhang** (38 years old, Tokyo, Japan) is into running and trains for marathons.

Delete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

  1. In the terminal window, go to the directory containing the infrastructure plan.

    Warning

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

  2. Delete resources:

    1. Run this command:

      terraform destroy

    2. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.

Previous
Using a Yandex Managed Service for Valkey™ cluster as a PHP session storage
Next
Distributed locks for 1C:Enterprise in a Yandex Managed Service for Valkey™ cluster