Contact UsGet started
© 2024 Direct Cursus Technology L.L.C.

Getting Yandex Object Storage object request statistics with Yandex Query

Written by
Updated at September 4, 2024

In this tutorial, you will learn how to get the statistics of requests to Yandex Object Storage objects using Yandex Query. You will create a bucket and configure logging in Object Storage, create a connection in Query, and get statistics using SQL queries.

To get statistics:

  1. Prepare your cloud.
  2. Create buckets.
  3. Enable logging.
  4. Configure connections in Query.
  5. Get request statistics.

If you no longer need the resources you created, delete them.

Prepare your cloud

Sign up for Yandex Cloud and create a billing account:

  1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

The infrastructure support costs include:

Create a service account

Using a service account, Query will be able to send requests to Object Storage.

Create a service account named yq-sa and assign the storage.viewer and yq.editor roles to it.

Create buckets

Create two buckets: object-bucket and logs-bucket. One bucket will serve as a data source, the other, as a log storage.

To create a bucket:

  1. In the management console, select the folder where you want to create a bucket.
  2. In the list of services, select Object Storage.
  3. At the top right, click Create bucket.
  4. In the ** Name** field, enter the bucket name: object-bucket.
  5. Click Create bucket.

Similarly, create a bucket named logs-bucket.

Enable logging

To get information on the requests to objects, enable logging actions on bucket:

  1. In the management console, select the folder where you created the bucket.

  2. In the list of services, select Object Storage.

  3. Select object-bucket.

  4. Enable logging:

    1. In the left-hand panel, select Settings.
    2. Go to the Logging tab.
    3. Enable Write logs.
    4. In the Bucket for log storage field, select logs-bucket.
    5. Click Save.

  5. In the left-hand panel, select Objects and upload your objects. For example, this could be a couple of simple text files.

Configure connections in Query

To get data from Object Storage, create a connection and binding:

  1. Go to Query.

  2. In the left-hand panel, select Connections.

  3. Click Create new.

  4. Enter a name for the connection, e.g., bucket-logs-connection.

  5. Select the Object Storage connection type and specify the Connection type parameters.

  6. In the Bucket auth field, select Private and set the parameters:

    • Cloud and Folder: Select the cloud and folder where you created your buckets.
    • Bucket: logs-bucket.
    • Service account: yq-sa.

  7. Click Check. If they are, click Create.

  8. Click Automatically fill settings for and select Object Storage Access Logs from the drop-down list.

    1. Enter a name for the binding, e.g., bucket-logs-binding.
    2. In the Path field, specify the path to the statistics data within the bucket. If the statistics are stored in the bucket's root directory, specify /.
    3. Click Preview to verify the settings are correct.
    4. Click Create to complete creating the binding.

Get request statistics

Use a connection to create SQL queries and get statistics on requests to Object Storage objects:

  1. Go to Query.

  2. In the left-hand panel, select Connections.

  3. Select bucket-logs-connection.

  4. In the editor on the right, enter the following query:

    SELECT `timestamp`, request_id, handler, object_key, status, request_time
FROM `bucket-logs-binding`
LIMIT 100;

    This query will return 100 records of query statistics for Object Storage objects. You can remove the record limit and filter results using WHERE.

  5. Click Run and see the result.

Query examples

Searching for requests by response code

SELECT `timestamp`, request_id, handler, object_key, status, request_time
FROM `bucket-logs-binding`
WHERE status >= 400

Searching for long-running requests

SELECT `timestamp`, request_id, handler, object_key, status, request_time
FROM `bucket-logs-binding`
WHERE request_time >= 1000

Average request processing time

This example uses the AVG aggregate function.

SELECT AVG(request_time) AS `avg` FROM `bucket-logs-binding`

How to delete the resources you created

To shut down the infrastructure and stop paying for the resources you created:

  1. Delete the binding.
  2. Delete the connection.
  3. Delete the objects from the buckets.
  4. Delete buckets.
Previous
JMeter load testing
Next
Invoking load testing from GitLab CI