Data transfer in microservice architectures
Modern applications are migrating from monolithic to microservice architectures. A growing number of components are now working together seamlessly, written in different programming languages and running on various devices.
Using data streaming buses for exchanging information between these components makes it easier to develop a microservice architecture, while enhancing reliability and scalability.
Benefits
Reliability
Microservice architectures are typically distributed by design. Reliability is a key factor in a distributed architecture. Implementing a complete failure model in each application is challenging. Therefore, fault tolerance is only built into infrastructure components, with other apps relying on them to stay fault-tolerant.
Different lifetime
In a monolithic architecture, all components are continuously available for data exchange. Components of a microservice architecture have different lifetimes and cannot rely on direct data exchange between them. You need to either buffer the data or send it via a data streaming bus.
Scalability
As applications grow, they are expected to remain just as efficient. With data streaming buses, you can simply follow a built-in interaction scenario to make your applications scalable.
Comparison with Apache Kafka®
The table below shows a comparison of Data Streams and Apache Kafka®, a widely used data streaming platform.
| Parameter | Yandex Data Streams | Apache Kafka® |
|---|---|---|
| Supported protocol | Amazon Kinesis Data Streams API | Apache Kafka® |
| Supported SDKs | HTTP, Java, C++, Go, and more | HTTP, Java, C++, Go, and more |
| Cost | Fee for using resources: speed and retention period | Fee for reserving resources: VMs |
| Operation complexity | No user administration required | The system is administered by the user, Yandex Cloud is only responsible for the infrastructure |
| Scaling | Zero service downtime (currently, the system only supports scaling up) | Service downtime while restarting a cluster |
| Integration with Yandex Cloud | High: Cloud Functions triggers, API Gateway, and more | Basic |
| Security | Authorization with an IAM token and access key | Authorization with a username and password |
| Reliability | High reliability with data stored across three availability zones | High reliability with user customization |
| Message delivery time | Less than a second | Less than a second |
| Data retention period | Set at creation, 1 to 24 hours | Limited by VM disk capacity |
| Customizability | Basic parameters | High, as long as Yandex Cloud supports the configurations |
Setup
To set up data transfer between components:
- Create a data stream in Data Streams.
- Configure the AWS SDK.
- Use the following SDKs: