Getting statistics on queries to Object Storage objects using Yandex Query
In this tutorial, you will learn how to get statistics on queries to Yandex Object Storage objects using Yandex Query. You will create a bucket and configure logging in Object Storage, set up a connection in Query, and get statistics using SQL queries.
To get statistics:
- Get your cloud ready.
- Create buckets.
- Enable logging.
- Configure connections in Query.
- Get statistics on queries.
If you no longer need the resources you created, delete them.
Get your cloud ready
Sign up in Yandex Cloud and create a billing account:
- Navigate to the management console
and log in to Yandex Cloud or register a new account. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one and link a cloud to it.
If you have an active billing account, you can navigate to the cloud page
Learn more about clouds and folders.
Required paid resources
The infrastructure support cost includes:
- Fee for data storage (see Yandex Object Storage pricing).
- Fee for the amount of read data (see Yandex Query pricing).
Create a service account
With a service account, Query will be able to send queries to Object Storage.
Create a service account named yq-sa
and assign the storage.viewer
and yq.editor
roles to it.
Create buckets
Create two buckets: object-bucket
and logs-bucket
. One bucket will serve as a data source, and the other, as a log storage.
To create a bucket:
- In the management console
, select the folder where you want to create a bucket. - From the list of services, select Object Storage.
- At the top right, click Create bucket.
- In the ** Name** field, enter the bucket name:
object-bucket
. - Click Create bucket.
Similarly, create a bucket named logs-bucket
.
Enable logging
To get information on queries to objects, enable logging actions on the bucket:
-
In the management console
, select the folder where you created the bucket. -
From the list of services, select Object Storage.
-
Select
object-bucket
. -
- In the left-hand panel, select
Settings. - Navigate to the Logging tab.
- Enable Write logs.
- In the Bucket for log storage field, select
logs-bucket
. - Click Save.
- In the left-hand panel, select
-
In the left-hand panel, select
Objects and upload your objects. For example, you can upload a couple of simple text files.
Configure connections in Query
To get data from Object Storage, create a connection and binding:
-
Go to Query
. -
In the left-hand panel, select Connections.
-
Click
Create new. -
Enter a name for the connection, e.g.,
bucket-logs-connection
. -
Select the Object Storage connection type and specify the Connection type parameters.
-
In the Bucket auth field, select
Private
and set these properties:- Cloud and Folder: Select the cloud and folder where you created your buckets.
- Bucket:
logs-bucket
. - Service account:
yq-sa
.
-
Click Check. After a successful test, click Create.
-
Click Automatically fill settings for and select Object Storage Access Logs from the drop-down list.
- Enter a name for the binding, e.g.,
bucket-logs-binding
. - In the Path field, specify the path to the statistics within the bucket. If the statistics are stored in the bucket root directory, specify
/
. - Click Preview to verify the settings are correct.
- Click Create to complete creating the binding.
- Enter a name for the binding, e.g.,
Get statistics on queries
Use a connection to create SQL queries and get statistics on queries to Object Storage objects:
-
Go to Query
. -
In the left-hand panel, select Connections.
-
Select
bucket-logs-connection
. -
In the editor on the right, enter this query:
SELECT `timestamp`, request_id, handler, object_key, status, request_time FROM `bucket-logs-binding` LIMIT 100;
This query will return 100 records of statistics on queries to Object Storage objects. You can remove the record limit and filter results using
WHERE
. -
Click
Run and check the results.
Query examples
Searching for queries by response code
SELECT `timestamp`, request_id, handler, object_key, status, request_time
FROM `bucket-logs-binding`
WHERE status >= 400
Searching for long-running queries
SELECT `timestamp`, request_id, handler, object_key, status, request_time
FROM `bucket-logs-binding`
WHERE request_time >= 1000
Average query processing time
This tutorial features the AVG
aggregate function.
SELECT AVG(request_time) AS `avg` FROM `bucket-logs-binding`
How to delete the resources you created
To shut down the infrastructure and stop paying for the resources you created: