Vector Store search indexes

AI agents can perform semantic search in the context obtained from external knowledge base files. This additional context is stored in Vector Store search indexes, which are dedicated Yandex AI Studio vector storages created using the Vector Store API and where documents are represented as vectors (embeddings).

Creating an indexCreating an index

To create a search index, upload source files for the knowledge base into Vector Store and index them.

Uploading source filesUploading source files

In Vector Store, you can upload up to 10,000 files with the maximum size of 128 MB per file. A single file can be included into multiple Vector Store search indexes at the same time.

You can upload files of the following MIME types:

• application/json
• application/msword
• application/pdf
• application/vnd.ms-excel
• application/vnd.ms-excel.sheet.2
• application/vnd.ms-excel.sheet.3
• application/vnd.ms-excel.sheet.4
• application/vnd.ms-excel.workspace.3
• application/vnd.ms-excel.workspace.4
• application/vnd.ms-outlook
• application/vnd.ms-powerpoint
• application/vnd.ms-project
• application/vnd.ms-word2006ml
• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
• application/vnd.openxmlformats-officedocument.wordprocessingml.document
• application/x-latex
• application/x-ms-owner
• application/xhtml+xml
• text/csv
• text/html
• text/markdown
• text/plain
• text/xml
• application/rtf

Indexing the uploaded filesIndexing the uploaded files

When creating Vector Store search indexes, the uploaded files are automatically broken down into fragments which the embedding model, in turn, transforms into numeric vectors. A vector reflects the semantic meaning of a fragment, which allows searching the index not only by matching words, but also by semantic relevance.

When breaking the information down into fragments, the semantic meaning is ignored, so the text may be split mid-sentence. This creates incomplete context. To minimize information loss due to broken context, some text in adjacent fragments overlaps.

By default, when creating Vector Store search indexes, the standard AI Studio embedding models are used. To improve vector search quality depending on the input file features, you may use fine-tuned embedding models.

After you finish processing all the files and vectorizing all the resulting text fragments, the search index will be ready to use.

Here are the possible states of the Vector Store search index creation process:

• in_progress: Breaking the uploaded files down into fragments and vectorizing them.
• completed: Search index is ready to use.
• failed: Error occurred while creating the search index.

Using the search indexUsing the search index

You can use search indexes to implement a retrieval augmented generation (RAG) scenario based on information from external sources, in which the text generation model will provide a response using the information from the search index.

You can add a Vector Store search index to an AI agent using the Retrieval tool via the Responses API or Realtime API.

A search through a Vector Store index will return:

• List of relevant text fragments.

  A search through a Vector Store index (similarity search) returns a list with the specified number of the most semantically relevant vectors (the Top-k method).

• Relevance score for every fragment in the list.

• Fragment metadata.

  You can apply filters by metadata, i.e., additional search index attributes specified when creating a Vector Store index. Search index metadata are objects with <key>:<value> pairs. They can extend the search context with additional information, e.g., data category, update date, special labels, etc.

Managing a search indexManaging a search index

You can add new files to the existing Vector Store search indexes as well as delete files from them.

The Vector Store API enables you to set up automatic deletion of unused search indexes. To do this, in the expires_after field, specify a value for any one of these properties:

• created_at: Sets the search index lifetime since its creation.
• last_active_at: Sets the search index lifetime since its last use.

After the time specified in the expires_after field expires, the search index will be deleted automatically. You also can delete a Vector Store search index manually at any moment.

Possible use cases for Vector Store search indexesPossible use cases for Vector Store search indexes

You can integrate Vector Store search indexes with AI agents to implement various use cases:

• AI bot for customer service to answer common questions

  • The knowledge base with instructions and answers to common customer questions is uploaded to a Vector Store search index.
  • The search index is connected to the AI agent.
  • The user asks a question, e.g., How much luggage may I carry?.
  • The search finds the relevant fragment in the knowledge base: One bag up to 23 kg.
  • The AI agent returns the response with a link to the document.

• Internal corporate knowledge base

  • Regulations, policies, and product guides are uploaded to a Vector Store search index.

  • A request to the search index returns semantically relevant answers even if the wordings do not fully match.

    For example, for the dismissal keyword, the system will find documents with such wordings as employment termination or labor contract dissolution.

• Call center AI assistant

  • The knowledge base with instructions and answers to common customer questions is uploaded to a Vector Store search index.
  • During a call, the agent types or says a question, e.g., How do I reset the user password?.
  • The AI agent connected to a search index immediately provides the relevant steps from the knowledge base.

• Analysis of large documents

  • Reports, studies, or contracts are uploaded to a Vector Store search index.

  • Send your request to the search index to quickly find the relevant fragments, e.g., Show all mentions of KPIs for 2024.

    This is especially helpful when processing hundreds of pages of information.

• Chat with an AI agent based on corporate documents

  • The AI agent is connected to a Vector Store search index containing the corporate knowledge base.
  • An employee asks a question to the AI agent: Explain how the product return process works.
  • The AI agent gets relevant fragments of documentation and uses them to provide a detailed and correct response.

• Local search in applications

  • A corporate application, e.g., a CRM system or Wiki, uses a Vector Store search index to search through internal content.
  • The user asks a question in a free form and finds relevant records even if the words do not match exactly.

• Multi-language search

  • The same Vector Store search index stores documents in English and Russian.
  • For the baggage allowance request, the index will also return fragments in Russian.

• Integration with analytics

  • Call or chat transcripts are uploaded to a Vector Store search index.
  • The Vector Store API allows you to find similar customer tickets.
  • You can quickly group the obtained information, analyze it, find the most common questions, and identify problems.

Clearly, Vector Store search indexes are a versatile tool which can assist both users and chatbots with internal search, document analysis, and integration into corporate business processes (customer support, analytics, employee training, etc.).

See alsoSee also

• Source files
• Managing Vector Store search index
• Creating a RAG assistant with the Vector Store tool