Batch processing

Batch processing is a technology for processing data that involves preparing aggregated information based on large arrays of data. This type of data analysis is traditional and is used for processing data stored, for example, in a DBMS.

Data volumes increase with time, but that doesn't mean all data is accessed often. So, rarely used data is usually transferred to storage systems like Yandex Object Storage which are much more cost-effective than DBMS.

Data is stored in Yandex Object Storage as a file structure with directories and files. To store data in files, standard storage formats are normally used: CSV, JSON, etc.

Yandex Query allows you to access data stored in Yandex Object Storage in the same way as a DBMS by making queries in an SQL dialect called YQL.

Yandex Object Storage usually stores massive amounts of data. Yandex Query analyzes how much data needs processing and runs dozens to hundreds of concurrent data processing jobs within the computing cluster. This allows maintaining a high processing speed even for large data volumes.