Contact UsGet started

Analyzing Object Storage logs in DataLens

Written by
Improved by
Updated at October 29, 2024

For the Yandex Object Storage bucket you can enable action logging. The logs store information about operations with a bucket and the objects in it. Analysis of bucket logs can be useful, for example, if you want to understand what caused a sharp increase in load or get the overall picture of traffic distribution.

You can create visualizations for your analysis using Yandex DataLens. You must transfer previously saved logs to the ClickHouse® database, which will be used as a source for DataLens.

To analyze the logs and present the results in interactive charts:

  1. Prepare your cloud.
  2. Create a bucket for storing logs.
  3. Enable log export.
  4. Prepare the data source.
  5. Create a connection in DataLens.
  6. Create a dataset in DataLens.
  7. Create charts in DataLens.
  8. Create a dashboard in DataLens.

If you no longer need the resources you created, delete them.

Prepare your cloud

Sign up for Yandex Cloud and create a billing account:

  1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

The cost includes:

Create a bucket for storing logs

  1. In the management console, select the folder you want to create a bucket in.
  2. In the list of services, select Object Storage.
  3. Click Create bucket.
  4. In the ** Name** field, enter a name for the bucket.
  5. In the Object read access and Object listing access fields, select Restricted.
  6. Click Create bucket.

  1. If you do not have the AWS CLI yet, install and configure it.

  2. Create a bucket:

    aws --endpoint-url https://storage.yandexcloud.net \
  s3 mb s3://<bucket_name>

    Result:

    make_bucket: <bucket_name>

Note

Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g., storage.admin, for the folder where you are going to create resources.

Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

For more information about the provider resources, see the documentation on the Terraform website or the mirror.

If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

  1. Describe the parameters for creating a service account and access key in the configuration file:

    ...
// Creating a service account
resource "yandex_iam_service_account" "sa" {
  name = "<service_account_name>"
}

// Assigning a role to a service account
resource "yandex_resourcemanager_folder_iam_member" "sa-admin" {
  folder_id = "<folder_ID>"
  role      = "storage.admin"
  member    = "serviceAccount:${yandex_iam_service_account.sa.id}"
}

// Creating a static access key
resource "yandex_iam_service_account_static_access_key" "sa-static-key" {
  service_account_id = yandex_iam_service_account.sa.id
  description        = "static access key for object storage"
}

  2. Add bucket parameters to the configuration file:

    resource "yandex_storage_bucket" "bucket-logs" {
  access_key = yandex_iam_service_account_static_access_key.sa-static-key.access_key
  secret_key = yandex_iam_service_account_static_access_key.sa-static-key.secret_key
  bucket     = "<bucket_name>"
}

    For more information about the yandex_storage_bucket resource, see the Terraform provider documentation.

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Create a bucket.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

Use the create REST API method.

Enable log export

  1. In the management console, select the bucket you want to enable logging for.
  2. In the left-hand panel, select Settings.
  3. Open the Logging tab.
  4. Enable Write logs.
  5. Select Bucket for log storage.
  6. In the Prefix field, specify the s3-logs/ prefix.
  7. Click Save.

  1. Create a file named log-config.json with the following contents:

    {
 "LoggingEnabled": {
    "TargetBucket": "<bucket_name>",
    "TargetPrefix": "s3-logs/"
 }
}

  2. Run this command:

    aws s3api put-bucket-logging \
  --endpoint-url https://storage.yandexcloud.net \
  --bucket <bucket_name> \
  --bucket-logging-status file://log-config.json

    Where --bucket is the name of the bucket to enable action logging for.

To enable logging for a bucket you want to track:

  1. Open the Terraform configuration file and add the logging section to the bucket description fragment.

    resource "yandex_storage_bucket" "bucket-logs" {
  access_key = "<static_key_ID>"
  secret_key = "<secret_key>"
  bucket     = "<name_of_bucket_to_store_logs>"
}

resource "yandex_storage_bucket" "bucket" {
  access_key = "<static_key_ID>"
  secret_key = "<secret_key>"
  bucket     = "<source_bucket_name>"
  acl        = "private"

  logging {
    target_bucket = yandex_storage_bucket.bucket-logs.id
    target_prefix = "s3-logs/"
  }
}

    Where:

    • access_key: Static access key ID.
    • secret_key: Secret access key value.
    • target_bucket: Reference to the log storage bucket.
    • target_prefix: Key prefix for objects with logs.

    For more information about the yandex_storage_bucket resource parameters in Terraform, see the provider documentation.

    1. In the terminal, change to the folder where you edited the configuration file.

    2. Make sure the configuration file is correct using the command:

      terraform validate

      If the configuration is correct, the following message is returned:

      Success! The configuration is valid.

    3. Run the command:

      terraform plan

      The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

    4. Apply the configuration changes:

      terraform apply

    5. Confirm the changes: type yes in the terminal and press Enter.

    All the resources you need will then be created in the specified folder. You can check the new resources and their configuration using the management console.

Use the REST API putBucketLogging method.

Prepare the data source

Create a ClickHouse® cluster

To create a Managed Service for ClickHouse® cluster, you need the vpc.user role and the managed-clickhouse.editor role or higher. For more information on assigning roles, see the Identity and Access Management documentation.

  1. In the management console, select the folder you want to create a cluster in.

  2. In the list of services, select Managed Service for ClickHouse.

  3. In the window that opens, click Create cluster.

  4. Specify the settings for a ClickHouse® cluster:

    1. Under Basic parameters, specify s3-logs in the Cluster name field.

    2. Under Resources, select burstable in the Type field.

    3. Under Hosts, click and enable the Public access option. Click Save.

    4. Under DBMS settings:

      • In the User management via SQL field, select Disabled.
      • In the Username field, specify user.
      • In the Password field, set a password.
      • In the DB name field, specify s3_data.

      Remember the database name.

    5. Under Service settings, enable the following options:

      • DataLens access.
      • Access from the management console.

  5. Click Create cluster.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a cluster:

  1. Check whether the folder has any subnets for the cluster hosts:

    yc vpc subnet list

    If there are no subnets in the folder, create the required subnets in VPC.

  2. Specify the cluster parameters in the create command:

    yc managed-clickhouse cluster create \
   --name s3-logs \
   --environment production \
   --network-name <network_name> \
   --host type=clickhouse,zone-id=<availability_zone>,subnet-id=<subnet_ID> \
   --clickhouse-resource-preset b2.medium \
   --clickhouse-disk-type network-ssd \
   --clickhouse-disk-size 10 \
   --user name=user,password=<user_password> \
   --database name=s3_data \
   --datalens-access=true \
   --websql-access=true

  1. Add a description of the cluster and cluster hosts to the configuration file:

    resource "yandex_mdb_clickhouse_cluster" "s3-logs" {
  name                = "s3-logs"
  environment         = "PRODUCTION"
  network_id          = yandex_vpc_network.<network_name_in_Terraform>.id

  clickhouse {
    resources {
      resource_preset_id = "b2.medium"
      disk_type_id       = "network-ssd"
      disk_size          = 10
    }
  }

  database {
    name = "s3_data"
  }

  user {
    name     = "user"
    password = "<password>"
    permission {
      database_name = "s3_data"
    }
  }

  host {
    type      = "CLICKHOUSE"
    zone      = "<availability_zone>"
    subnet_id = yandex_vpc_subnet.<subnet_name_in_Terraform>.id
  }

  access {
    datalens  = true
    web_sql   = true
  }
}

    For more information about the resources you can create with Terraform, see the provider documentation.

  2. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  3. Create a cluster:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

Use the create REST API method.

After creating the cluster, you will be automatically redirected to the Clusters page.

Wait for the cluster status to change to Alive.

Change user settings

  1. Select the s3-logs cluster.
  2. Go to the Users tab.
  3. Click and select Configure.
  4. Click Additional settings → Settings.
  5. In the Date time input format field, select best_effort.
  6. Click Save.

Create a static key

To create a table with access to Object Storage, you need a static key. Create it and save the ID and secret part of the key.

Create a table in the database

  1. Select the s3-logs cluster.

  2. Go to the SQL tab.

  3. In the Password field, enter the password.

  4. Click Connect.

  5. In the window on the right, write an SQL query:

    CREATE TABLE s3_data.s3logs
(
   bucket String,              -- Bucket name.
   bytes_received Int64,       -- Size of the request in bytes.
   bytes_send Int64,           -- Response size in bytes.
   handler String,             -- Request method in this format: REST.<HTTP method>.<subject>.
   http_referer String,        -- URL of request source.
   ip String,                  -- User IP address.
   method String,              -- HTTP request method.
   object_key String,          -- Object key in URL-encoded format.
   protocol String,            -- Data transfer protocol version.
   range String,               -- HTTP header that defines range of bytes to load from object.
   requester String,           -- User ID.
   request_args String,        -- Arguments of the URL request.
   request_id String,          -- Query ID.
   request_path String,        -- Full path of the request.
   request_time Int64,         -- Request processing time in milliseconds.
   scheme String,              -- Data transfer protocol type.
                               -- The possible values are:
                               -- * http: Application layer protocol.
                               -- * https: Application layer protocol with encryption support.
   ssl_protocol String,        -- Security protocol.
   status Int64,               -- HTTP response code.
   storage_class String,       -- Storage class of the object.
   timestamp DateTime,        -- Date and time of the operation with the bucket in the YYYY-MM-DDTHH:MM:MMZ format.
   user_agent String,          -- Client application (user agent) that executed the request.
   version_id String,          -- Object version.
   vhost String                -- Virtual host of request.
                               -- The possible values are:
                               -- * storage.yandexcloud.net.
                               -- * <bucket_name>.storage.yandexcloud.net.
                               -- * website.yandexcloud.net.
                               -- * <bucket_name>.website.yandexcloud.net.
)
ENGINE = S3(
      'https://storage.yandexcloud.net/<bucket_name>/s3-logs/*',
      '<key_ID>',
      '<secret_key>',
      'JSONEachRow'
   )
SETTINGS date_time_input_format='best_effort';

  6. Click Execute.

Create a connection in DataLens

  1. Select the s3-logs cluster.

  2. Go to the DataLens tab.

  3. In the window that opens, click Create connection.

  4. Fill in the connection settings:

    1. Add a connection name: s3-logs-con.
    2. In the Cluster field, select s3-logs.
    3. In the Host name field, select the ClickHouse® host from the drop-down list.
    4. Enter the DB user name and password.

  5. Click Confirm connection.

  6. After checking the connection, click Create connection.

  7. In the window that opens, enter a name for the connection and click Create.

Create a dataset in DataLens

  1. Click Create dataset.

  2. In the new dataset, move the s3_data.s3logs table to the workspace.

  3. Go to the Fields tab.

  4. Click Add field.

  5. Create a calculated field with the file type:

    • Field name: object_type
    • Formula: SPLIT([object_key], '.', -1)

  6. Click Create.

  7. In the top-right corner, click Save.

  8. Enter the s3-dataset name for the dataset and click Create.

  9. When the dataset is saved, click Create chart in the top-right corner.

Create charts in DataLens

Create the first chart

To visualize the number of requests to a bucket using different methods, create a pie chart:

  1. For the visualization type, select Pie chart.
  2. Drag the method field from the Dimensions section to the Colors section.
  3. Drag the request_id field from the Dimensions section to the Measures section.
  4. In the top-right corner, click Save.
  5. In the window that opens, enter the S3 - Method pie name for the chart and click Save.

Create the second chart

To visualize the number of requests ratio by object type, create a bar chart:

  1. Copy the chart from the previous step:

    1. In the top-right corner, click the down arrow next to the Save button.
    2. Click Save as.
    3. In the window that opens, enter the S3 - Object type bars name for the new chart and click Save.

  2. Select the Bar chart visualization type. The method and request_id fields will automatically appear in the X and Y sections, respectively.

  3. Delete the method field from the X section and drag the object_type field there.

  4. In the top-right corner, click Save.

Create the third chart

To visualize the distribution of outgoing traffic by day, create a bar chart:

  1. Copy the chart from the previous step:

    1. In the top-right corner, click the down arrow next to the Save button.
    2. Click Save as.
    3. In the window that opens, enter the S3 - Traffic generated by days name for the new chart and click Save.

  2. Drag the object_type field from the X section to the Filters section.

  3. In the window that opens, select the types of objects to display in the chart and click Apply filters.

  4. Drag the timestamp field from the Dimensions section to the X section.

  5. Delete the request_id field from the Y section and drag the bytes_send field there.

  6. In the top-right corner, click Save.

Create a dashboard in DataLens and add charts there

  1. Go to the DataLens home page.
  2. Click Create dashboard.
  3. Enter a name for the S3 Logs Analysis dashboard and click Create.
  4. In the top-right corner, click Add and select Chart.
  5. In the Chart field, click Select and choose the S3 - Method pie chart from the list.
  6. Click Add. The chart will be displayed on the dashboard.
  7. Repeat the previous steps for the S3 - Object type bars and S3 - Traffic generated by days charts.

How to delete the resources you created

Delete the resources you no longer need to avoid paying for them:

ClickHouse® is a registered trademark of ClickHouse, Inc.

Previous
Blue-green and canary deployment of service versions
Next
Mounting buckets to the file system of Yandex Data Processing hosts