Source files

For your model to use external information sources to respond to requests, upload supplementary data files through the Files API and create a search index for them. You can upload up to 10,000 files with the maximum size of 128 MB per file. The same file can be included into several search indexes at the same time.

For all AI Assistant API limitations, see Yandex AI Studio quotas and limits.

You can upload files of the following MIME types:

• application/json
• application/msword
• application/pdf
• application/vnd.ms-excel
• application/vnd.ms-excel.sheet.2
• application/vnd.ms-excel.sheet.3
• application/vnd.ms-excel.sheet.4
• application/vnd.ms-excel.workspace.3
• application/vnd.ms-excel.workspace.4
• application/vnd.ms-outlook
• application/vnd.ms-powerpoint
• application/vnd.ms-project
• application/vnd.ms-word2006ml
• application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
• application/vnd.openxmlformats-officedocument.wordprocessingml.document
• application/x-latex
• application/x-ms-owner
• application/xhtml+xml
• text/csv
• text/html
• text/markdown
• text/plain
• text/xml
• application/rtf

Markdown is the optimal search index source as many models are trained on this format and are more likely to interpret it correctly. Use the docling Python library to convert files, even those with complex formatting, into Markdown. Learn more in Creating an AI assistant with RAG from PDF files with complex formatting.

When you create a search index, you define the type of search it will support. Full-text, vector, and hybrid search types are supported. Indexing may take from a few seconds to several hours depending on the file type and size as well as service load. The files are indexed asynchronously. The response to the request to create a serach index includes the operation ID. You can use it to find out when the search index will be ready.

Once a search index is created, you can configure an assistant to utilize it. In this case, the model will consider the contents of that search index and will primarily use information from it to generate responses.

Source verificationSource verification

If the AI assistant uses search indexes with external information sources when generating a response, the model's answer contains source citations, i.e., the citations section with details about all the indexes, external files, and fragments used for the response.

When using ML SDK, source citations are available in the citations property of the run object. To get the source citations via the API, use the run ID and initiate the Run.Get REST API method or the RunService/Get gRPC API call. The section with source citations is also included in all assistant messages stored in the thread.

Structure of the citations sectionStructure of the citations section

The citations section that features links to the sources has the following structure:

• sources: Array consisting of one or more fragments of source files that were used to generate the response:

  • chunk: Information on the file fragment that was used to generate the response:

    • searchIndex: Section of fields with information about the search index that includes the source file fragment used. This section contains the ID, type, metadata (labels), and other information about the index and its settings.
    • sourceFile: Section of fields with information about the source file, whose fragment was used to generate the response. This section contains the ID, metadata (labels), and other information about the source file.
    • content: Section of fields with the fragment text that was used to generate the response.

To learn more about the citations section structure, see the API reference.

Source file and search index metadataSource file and search index metadata

For more efficient usage of source confirmation and data returned in the citations section, you can specify metadata for each source file and search index. You can use metadata to apply additional filters to results or to give more detailed and meaningful names and descriptions to the sources.

Source file metadata is specified when creating the files using the File.Create REST API method, FileService/Create gRPC API call, or Yandex Cloud ML SDK. Source file metadata is provided as objects containing <key>:<value> pairs.

Search index metadata is specified when creating the indexes using the SearchIndex.Create REST API method, SearchIndexService/Create gRPC API call, or Yandex Cloud ML SDK. Search index metadata is provided as objects containing <key>:<value> pairs.

The metadata of each source file and search index may comprise one or more <key>:<value> pairs.

For more information on using source file and search index metadata, see Creating an AI assistant for RAG with source file and index metadata preserved.

For examples of how to work with source citations using the SDK and API, and for output examples, see Creating an assistant with a search index.