Comparison with Yandex Message Queue

Yandex Cloud uses Yandex Data Streams and Yandex Message Queue for exchanging data.

The services of data buses and data streams solve similar tasks, but in different use cases:

• Yandex Message Queue is focused on messaging between components of distributed applications.
• Yandex Data Streams is designed for transmitting data streams (such as logs of application operation or user actions, database CDC streams, or data from devices) for their processing by applications.

Yandex Message QueueYandex Message Queue

Messaging buses help deliver separate, independent messages to application components (handlers) and function as a task processing queue. Tasks are enqueued and then picked up by a handler to run. Only one application can read messages from a single queue. A handler may have multiple instances running at the same time.

Messaging buses are suitable for processing independent tasks, each of which can be run by any handler. The main thing is that an event is processed, no matter in what order.

E.g., web crawlers use message queues. A web crawler finds page URLs and enqueues them. Parsers get these URLs, download the pages, and process them. A message queue acts as a task queue, boosting the fault tolerance and scalability of such a system.

Yandex Data StreamsYandex Data Streams

Yandex Data Streams is designed to transmit data streams, rather than individual messages, to applications. The throughput of processed data streams may range from kilobytes to terabytes or more per hour.

Data in Yandex Data Streams is transmitted through streams that consist of shards. All incoming records have keys that identify which shard the data will get into. Within a single shard, the data is read as it is written.

In Yandex Data Streams, unlike Yandex Message Queue, different applications can simultaneously read data from the same stream. At the same time, each application can read data from any position.

Reading data simultaneously by different applications may be necessary in the following cases:

• When you need to process the same data by multiple applications at once.
• To protect against errors. If an error occurs in the handler, you can fix it and recalculate the enqueued data properly.
• When launching a new application. When developing a new application version, you can run it in parallel with the main one and test it on real data.

You can transmit any data via data buses, but most often these are application logs, user action logs, or data from devices. Yandex Data Streams supports an HTTP API to exchange data.

Yandex Message Queue and Yandex Data Streams comparisonYandex Message Queue and Yandex Data Streams comparison