Processing Yandex Cloud Logging logs
Yandex Cloud Logging is a service for reading and writing logs of Yandex Cloud services and user applications.
Logs can be sent to a Yandex Data Streams stream and then processed in real time using Yandex Query. You can do the following with processed data:
- Send it to Yandex Monitoring to make charts and use it in alerting.
- Write it to a Data Streams stream and then send it to Yandex Cloud Functions for processing.
- Write it to a Data Streams stream and then transfer it to Yandex Data Transfer to be sent to various storage systems.
In this use case, you will send Cloud Logging logs to a Data Streams stream and then run a query to them using Query. The query will return the number of messages per host grouped by 10s interval.
To implement this use case:
- Create a Data Streams data stream
- Create a Cloud Logging log group
- Start sending data to the log group
- Connect Query to your data stream
- Run a data query
Getting started
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Install the Yandex Cloud command line interface.
Create a Data Streams data stream
Create a stream named cloud-logging-stream
.
Create a Cloud Logging log group
Create a log group named cloud-logging-group
. When setting the log group parameters, specify cloud-logging-stream
created in the previous step.
Start sending data to the log group
To start sending data to the log group, run this command:
while true; do yc logging write \
--group-name=cloud-logging-group \
--message="test_message" \
--timestamp="1s ago" \
--level=INFO \
--json-payload='{"request_id": "1234", "host":"test_host"}' \
--folder-id b1kmrhakmf8a********; \
sleep 1; \
done
--group-name
: Name of the log group the messages are sent to.--message
: Message text.--json_payload
: Additional message data in JSON format.--folder-id
: ID of the folder where the log group was created.
Note
You can skip the --group-name
, --message
, and --json-payload
flags and specify only the parameter values, e.g., cloud-logging-group "test_message" '{"request_id": "1234", "host":"test_host"}'
.
Connect Query to your data stream
- Create a connection named
cloud-logging-connection
of theData Streams
type. - On the binding creation page:
- Select Automatically fill settings for Cloud Logging.
- Enter the binding name:
cloud-logging-binding
. - Specify the stream:
cloud-logging-stream
. - Set
json-list
format.
- Click Create.
Run a data query
Open the query editor in the Query interface and run the query:
$cloud_logging_data =
SELECT
CAST(JSON_VALUE(data, "$.timestamp") AS Timestamp) AS `timestamp`,
JSON_VALUE(data, "$.jsonPayload.host") AS host
FROM bindings.`cloud-logging-binding`;
SELECT
host,
COUNT(*) AS message_count,
HOP_END() AS `timestamp`
FROM $cloud_logging_data
GROUP BY
HOP(`timestamp`, "PT10S", "PT10S", "PT10S"),
host
LIMIT 2;
Result:
# | host | message_count | timestamp |
---|---|---|---|
1 | "test_host" | 3 | 2023-05-09T10:34:00.000000Z |
2 | "test_host" | 4 | 2023-05-09T10:34:10.000000Z |
Note
Data from a stream source is transferred as an infinite stream. To stop data processing and output the result to the console, the data in the example is limited with the LIMIT
operator that sets the number of rows in the result.