Batch processing
Batch processing is a data processing technology that aggregates information based on large arrays of data. This type of data analysis is traditional and is used for processing data stored, for example, in a DBMS.
Data volumes increase with time but that does not mean all data is accessed often. So, rarely used data is usually transferred to storage systems like Yandex Object Storage which are much more cost-effective than DBMS.
Data is stored in Yandex Object Storage as a file structure with directories and files. To store data in files, use standard storage formats: CSV, JSON, etc.
Yandex Query allows you to access data stored in Yandex Object Storage in the same way as a DBMS by making queries in an SQL dialect called YQL.
Yandex Object Storage usually stores massive amounts of data. Yandex Query analyzes how much data needs processing and runs dozens to hundreds of concurrent data processing jobs within the computing cluster. This allows maintaining a high processing speed even for large data volumes.
Use cases
- Analyzing data with Jupyter
- Analyzing data with Query
- Visualizing Yandex Object Storage data in Yandex DataLens
- Processing usage detail files in Yandex Cloud Billing
- Working with data in Yandex Managed Service for ClickHouse®
- Working with data in Yandex Managed Service for PostgreSQL
- Working with data in Yandex Object Storage