Transferring data within microservice architectures
Modern applications are migrating from monolithic to microservice architectures. A large number of interacting components appear, which are written in different programming languages and run on various devices.
Using data streaming buses for exchanging information between these components makes it easier to develop a microservice architecture and enhances its reliability and scalability.
Benefits
Reliability
Microservice architectures are usually distributed. Reliability is a key factor in a distributed architecture. It's difficult to support a complete failure model in each application. Therefore, fault tolerance is ensured for infrastructure components with other applications relying on them and also becoming fault-tolerant.
Different lifetime
In a monolithic architecture, all components are continuously available for data exchange. Components of a microservice architecture have a different lifetime and can't rely on direct data exchange between them. Data should be either buffered or sent via a data streaming bus.
Scalability
The efficiency of applications needs to be ensured as they grow. With data streaming buses, you can simply follow a built-in interaction scenario to make your applications scalable.
Comparison with Apache Kafka®
In the table below, Data Streams is compared with Apache Kafka®, a popular data streaming bus.
| Parameter | Yandex Data Streams | Apache Kafka® |
|---|---|---|
| Supported protocol | Amazon Kinesis Data Streams API | Apache Kafka® |
| Supported SDKs | HTTP, Java, C++, Go, and more | HTTP, Java, C++, Go, and more |
| Cost | A fee for using resources: speed and retention period | A fee for reserving resources: VMs. |
| Operation complexity | No user administration is required | The system is administered by the user, Yandex Cloud is only responsible for the infrastructure |
| Scaling | Without service interruption (currently, the system can only be scaled up) | With service interruption while restarting a cluster |
| Integration with Yandex Cloud | High: Cloud Functions triggers, API Gateway, and so on | Basic |
| Security | Authorization using an IAM token and an access key | Username and password based authorization |
| Reliability | High, data is stored in three availability zones | High, customized by the user |
| Message delivery time | Less than a second | Less than a second |
| Data retention period | Set at creation, 1-24 hours | Limited by VM disk capacity |
| Customizability | Basic parameters | High if the settings are supported in Yandex Cloud |
Setup
To set up data transfer between components:
- Create a data stream Data Streams.
- Set up the AWS SDK.
- Use SDKs: