Getting Yandex Object Storage object request statistics with Yandex Query
In this tutorial, you will learn how to get the statistics of requests to Yandex Object Storage objects using Yandex Query. You will create a bucket and configure logging in Object Storage, create a connection in Query, and get statistics using SQL queries.
To get statistics:
- Prepare your cloud.
- Create buckets.
- Enable logging.
- Configure connections in Query.
- Get request statistics.
If you no longer need the resources you created, delete them.
Prepare your cloud
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Required paid resources
The infrastructure support costs include:
- Data storage fee (see Yandex Object Storage pricing).
- Fee for the amount of read data (see Yandex Query pricing).
Create a service account
Using a service account, Query will be able to send requests to Object Storage.
Create a service account named yq-sa
and assign the storage.viewer
and yq.editor
roles to it.
Create buckets
Create two buckets: object-bucket
and logs-bucket
. One bucket will serve as a data source, the other, as a log storage.
To create a bucket:
- In the management console
, select the folder where you want to create a bucket. - In the list of services, select Object Storage.
- At the top right, click Create bucket.
- In the ** Name** field, enter the bucket name:
object-bucket
. - Click Create bucket.
Similarly, create a bucket named logs-bucket
.
Enable logging
To get information on the requests to objects, enable logging actions on bucket:
-
In the management console
, select the folder where you created the bucket. -
In the list of services, select Object Storage.
-
Select
object-bucket
. -
- In the left-hand panel, select
Settings. - Go to the Logging tab.
- Enable Write logs.
- In the Bucket for log storage field, select
logs-bucket
. - Click Save.
- In the left-hand panel, select
-
In the left-hand panel, select
Objects and upload your objects. For example, this could be a couple of simple text files.
Configure connections in Query
To get data from Object Storage, create a connection and binding:
-
Go to Query
. -
In the left-hand panel, select Connections.
-
Click
Create new. -
Enter a name for the connection, e.g.,
bucket-logs-connection
. -
Select the Object Storage connection type and specify the Connection type parameters.
-
In the Bucket auth field, select
Private
and set the parameters:- Cloud and Folder: Select the cloud and folder where you created your buckets.
- Bucket:
logs-bucket
. - Service account:
yq-sa
.
-
Click Check. If they are, click Create.
-
Click Automatically fill settings for and select Object Storage Access Logs from the drop-down list.
- Enter a name for the binding, e.g.,
bucket-logs-binding
. - In the Path field, specify the path to the statistics data within the bucket. If the statistics are stored in the bucket's root directory, specify
/
. - Click Preview to verify the settings are correct.
- Click Create to complete creating the binding.
- Enter a name for the binding, e.g.,
Get request statistics
Use a connection to create SQL queries and get statistics on requests to Object Storage objects:
-
Go to Query
. -
In the left-hand panel, select Connections.
-
Select
bucket-logs-connection
. -
In the editor on the right, enter the following query:
SELECT `timestamp`, request_id, handler, object_key, status, request_time FROM `bucket-logs-binding` LIMIT 100;
This query will return 100 records of query statistics for Object Storage objects. You can remove the record limit and filter results using
WHERE
. -
Click
Run and see the result.
Query examples
Searching for requests by response code
SELECT `timestamp`, request_id, handler, object_key, status, request_time
FROM `bucket-logs-binding`
WHERE status >= 400
Searching for long-running requests
SELECT `timestamp`, request_id, handler, object_key, status, request_time
FROM `bucket-logs-binding`
WHERE request_time >= 1000
Average request processing time
This example uses the AVG
aggregate function.
SELECT AVG(request_time) AS `avg` FROM `bucket-logs-binding`
How to delete the resources you created
To shut down the infrastructure and stop paying for the resources you created: