Data Streams overview

Applications generate data that needs to be saved for further analysis or processing. Some of the data needs to be stored for a long time in cold storage that is rarely accessed, while other data should be stored in analytical databases for hot data processing.

Yandex Data Streams makes it easier to transfer user application data to Yandex Cloud storage systems.

The data is received in Data Streams as in a data bus that stores it in a fault-tolerant way across availability zones and is scaled based on the transferred amount of data. You can send the data to the bus using the Fluentd, Logstash, log4j/log4net, and other data streaming systems, as well as via HTTP over a protocol compatible with the Amazon Kinesis Data Streams API.

The data being transferred via the bus can then be saved to target systems, such as S3, ClickHouse®, etc., using Yandex Data Transfer. You can set up the transfer parameters in the Yandex Cloud management console or via the API.

If, while saving the data, you need to change either the data itself or its format, or process it in any other way (for example, delete sensitive information), you can do this using Yandex Cloud Functions functions. Cloud Functions supports a variety of programming languages such as Python, Java, PHP, and more.

BenefitsBenefits

• Support for a large number of targets, extensive customization options for streaming data.
• The solution is fully integrated into the Yandex Cloud ecosystem and supports central management of data streams using both the Yandex Cloud management console and API.
• All components are fully managed, which means they do not require any administration or dedicated DevOps engineers.

ClickHouse® is a registered trademark of ClickHouse, Inc.