Comparison with Yandex Data Streams

Yandex Cloud uses Yandex Message Queue and Yandex Data Streams for exchanging data.

The services of data buses and data streams solve similar tasks, but in different use cases:

• Yandex Message Queue is focused on messaging between components of distributed applications.
• Yandex Data Streams is designed for transmitting data streams (such as logs of application operation or user actions, database CDC streams, or data from devices) for their processing by applications.

Yandex Message QueueYandex Message Queue

Messaging buses are focused on delivering separate, independent messages to application components (handlers) and implement a task processing queue scenario. Tasks are enqueued and a handler takes them from this queue and executes them. Only one application can read messages from a single queue. A handler may have multiple instances running at the same time.

Messaging buses are suitable for processing a set of independent tasks, each of which can be performed by any handler. The main thing is that an event is processed, no matter in what order.

For example, message queues are used by web crawler. A web crawler finds page URLs and enqueues them. Parsers get these URLs, download the pages, and perform processing. A message queue acts as a task queue, increasing the fault tolerance and scalability of a system like this.

Yandex Data StreamsYandex Data Streams

Yandex Data Streams is designed to transmit data streams, rather than individual messages, to applications. The throughput of processed data streams can range from kilobytes to terabytes or more per hour.

Data in Yandex Data Streams is transmitted through streams that consist of shards. All incoming records have keys that identify which shard the data will get into. Within a single shard, the data is read as it is written.

In Yandex Data Streams, unlike Yandex Message Queue, different applications can simultaneously read data from the same stream. At the same time, each application can read data from an arbitrary position.

Reading data simultaneously by different applications may be necessary in the following cases:

• When the same data needs to be processed by multiple applications at once.
• To protect against errors. If an error occurs in the handler, you can fix it and recalculate the enqueued data properly.
• When launching a new application. When developing a new application version, you can run it in parallel with the main one and test it on real data.

You can transmit any data via data buses, but most often these are application logs, user action logs, or data from devices. Yandex Data Streams supports an HTTP API to exchange data.

Comparison of Yandex Message Queue and Yandex Data StreamsComparison of Yandex Message Queue and Yandex Data Streams