Yandex Scale 2024 Product digest

We added new features to the Yandex SpeechSense service for deep analysis of voice and text communication channels with customers:

• Enterprise chat support: You can now efficiently process dialogs involving multiple operators, preserving the details of the conversation, and more thoroughly analyze the dialogs in chats, taking into account the length of the dialog, the duration of pauses, and typing speed.

• GPT dialog analysis: Yandex SpeechSense with the help of YandexGPT offers two types of dialog analysis. Semantic analysis helps assess the quality of the operator’s work and the client’s behavior during a conversation. Classical summaries define the topic, causes, outcomes, and emotional component of the dialog. Quick searches by topic allow you to find similar dialogues.

• The advanced tag builder allows you to create multi-level theme trees with primary and dependent tags. You can also set tag trigger areas, especially useful when processing objections.

• Neuro-based reports (Preview): The generative model of the service determines the key topics of dialogs with high accuracy, helps analyze the data obtained, and prepare reports on the specified neuro-based parameters. Availble in Preview mode.

The technical manual contains the main security aspects of the ML service for its integration and use in business applications.

SourceCraft is a new platform from Yandex for integrated software product development, covering all stages: source code creation, version control, testing, assembly, deployment, and maintenance.

The first element of the platform will be the Yandex Code Assistant, an AI assistant for working with code. It supports more than 30 programming languages and is compatible with popular code editors via the VS Code plugin. It analyzes the context and offers relevant code suggestions, generating 95% of prompts at a speed of up to 400 milliseconds. The service has two modes of operation: inline completion — offering suggestions as code is being written, and full-function completion — generating entire functions.

The SourceCraft platform is under development, and is scheduled to launch in early 2025. Join the SourceCraft community to keep up with the latest news and register to be added to the waiting list by filling out the form.

Escalations in Yandex Monitoring allow you to configure the sequence of notification steps when alerts are triggered. As notification methods, you can use Yandex Cloud Functions, Telegram, push notifications, email, SMS messages, and calls. Yandex Monitoring is integration with Yandex Audit Trails to track security events on the dashboard. Yandex Smart Web Security tools help you monitor traffic and log events, providing more detailed control over security and performance.

Yandex Managed Service for GitLab now includes the Yandex Managed Runners (Technical Preview) feature, which simplifies and automates continuous integration and delivery (CI/CD) processes using GitLab Runner, an agent application that runs CI/CD tasks defined in the GitLab^® project. Yandex Managed Runners are automatically deployed in the user’s infrastructure and support auto scaling, so they do not need manual configuration.

Users can also monitor instance metrics (CPU, RAM, disk space) via Yandex Monitoring, set up alerts, and create dashboards for data visualization.

• Date of last use of keys: See the date when keys were last used. This helps to identify inactive keys and improve security.

• Date of last authentication: See the date of the last authentication of a service account. Track the last time authentication was completed and monitor account activity.

• Copied API keys: Create API keys with a limited validity period or access to certain services, improving access control.

• Meta groups: New types of system groups with all users of a specific federation or the entire cloud organization are now available. This makes it possible to add access to a large number of users without opening public access to cloud resources.

• OS Login: Manage access to a virtual machine via SSH via IAM in a centralized manner. Now available in General Availability.

• Workload Identity Federation: Configure OpenID Connect to securely access the Yandex Cloud API from third-party systems without long-lived keys.

• WAF (Web Application Firewall): Protect against the exploitation of web application vulnerabilities. Now available to all users, no longer in testing, with set prices and an SLA.

• ARL (Advanced Rate Limiter): Limit loads on applications while setting limits on the number of HTTP requests for a certain period of time. It is now available to all users, has left the testing stage and now has set prices and an SLA.

• IP address black and white lists: Manage traffic and create security rules based on IP lists using reputation analysis of IP addresses.

• Sending logs to Audit Trails: Automatically send extended event logs based on WAF triggers and URLs to Audit Trails for improved monitoring and management of security incidents.

Set captcha display settings for certain IP addresses, countries, and devices. Now, for a certain section of traffic, you can configure a captcha with different parameters, i.e. types of main and additional challenges, their complexity and appearance.

We launched a new test for information security specialists working with the Yandex Cloud platform. It is designed for people who have at least a year of experience in the field of information security and who solve practical tasks for protecting cloud systems. To obtain the status of a certified specialist, you must successfully pass the text consisting of 75 questions.

We added new features to Yandex Cloud Functions, a service that allows you to run code as functions in a secure, fault-tolerant and automatically scalable environment, without having to create and maintain virtual machines. Now the maximum execution time for functions has been increased from 10 to 60 minutes, resource allocation capabilities have been expanded: the maximum number of cores has increased from 2 to 4, and memory from 4 to 8 GB. This makes it easier to work with rare and long queries.

Preheating is now available. If previously, the worker runtime was created when an incoming request was received, now it can be done in advance to reduce the startup time for functions and improve performance under high loads. You can mount ephemeral storage in functions and containers with up to 10 GB of disk space and speeds comparable to NBS. You can mount such a disk in Preview mode. This option is available on request via the support service.

We updated the Yandex MetaData Hub metadata management service. The preview of the Schema Registry component now allows users to describe data schemas and set a policy for their evolution.

In addition, the Hive Metastore feature, which saves table metadata between runs of temporary computing clusters, is now available. The key functionality of the Yandex MetaData Hub, Data Catalog, will be available in the first half of 2025. You can use it to collect data descriptions and track dependencies between them (lineage), organizing metadata management in a single catalog.

Since September 16, the Yandex Managed Service for Apache Airflow service for managing data flow orchestration is now available to all users. It is integrated with Yandex Cloud and offers additional functionalities when compared to the self-hosted versions. It now has pricing and SLA guarantees.

• Secure embedding of non-public dashboards. Developers and analysts can safely embed not just charts, but also dashboards into analytical products — in CRM or corporate portals — to display key indicators.

• Report and presentation designer. In Yandex DataLens, you can now create multi-page documents for regular meetings or to send to clients. You can now work with layers, change the page orientation and size, add text and comments. Documents can be exported in PDF format.

• Publication control. The functionality will be useful for analytics teams who need transparency when working with corporate or sensitive data. In the DataLens settings, you can view a list of published objects and disable the ability to place them for the entire instance.

• Expanded interactive features. It is now possible to pin selectors and table columns. Markup in chart captions is supported.

• Support for groups in RLS. When configuring RLS in a dataset, you can now specify not just specific users, but also groups, using the syntax. Example: @group:<group name>.

• Connection to Connection Manager. This functionality of the Yandex MetaData Hub allows you to securely store the details of database connections, manage access to them, and rotate passwords.

Users can now perform transactional data transfers between YDS streams, as well as between YDS streams and YDB row tables. We added the function of automatic scaling of data streams with an increased write speed while maintaining the read order, guaranteeing exactly-once writes and availability. Compatibility with the Apache Kafka^® protocol has also been improved.

Yandex Message Queue now supports the latest version of the AWS SQS protocol. This allows the user to seamlessly migrate from AWS SQL XML to Yandex Cloud.

We added support for Managed Service for Greenplum and Managed Service for MySQL, an IntelliSense function to simplify query writing, as well as automatic detection of data column names and types from Object Storage. Yandex Query is now compatible with YDB. This provides support for the YDB SDK, a variety of programming languages, as well as for integrations with DBeaver, DataGrip, Airflow, Jupyter Notebooks, and DataSphere.

We introduced new tools for optimizing performance: cost optimizer, spilling and massively parallel data processing with a vector engine. Together with the column table partitioning functionality, this will allow YDB to be used in scenarios where large amounts of data need to be processed. At the same time, Workload Manager helps avoid conflicts between loads, protecting regular and priority requests. Also, in the alpha version mode you can run analytical queries in PostgreSQL syntax with visualization in Apache Superset.

We added the exact method of vector search. It allows you to search for the most similar words, phrases, documents, images, and audio files. For example, to improve the quality of the voice assistant by enriching it with the data of a dialog with the user. At the same time, search results are used in large language models, semantic and multimodal search, and similarity search scenarios, as well as in the recommendation mechanism.

We added asynchronous replication, which allows for data migration between YDB databases with minimal downtime for the applications running on them. Clusters can be located in different regions.

YDB has added support for federated queries that allow you to work with data from Greenplum, Microsoft SQL Server (MSSQL), MySQL, PostgreSQL, S3.

Now you can work with string tables and YDB topics within a single transaction. This means you can transactionally transfer data from tables to topics and vice-versa, as well as between topics, so that data is not lost or duplicated even in the case of unforeseen circumstances.

The new and unique functionality of YDB Topics for the market is the ability to customize topic auto-partitioning. Now you can create a topic, where the number of partitions will increase automatically when the write speed to the topic increases. The increase in the number of partitions is transparent to the user, while maintaining guarantees of the read order, exactly-once writes, and accessibility with no effort on the part of the user.

We improved the web console by adding query settings, displaying the analytical query plan, SQL query templates, and context-sensitive hints in the editor, updated the documentation, and simplified cluster deployment.

Important and necessary improvements: CREATE VIEW, unique secondary indexes, automatic index selection, a single string API for executing DDL and DML (QueryService), serial data type.