Yandex Cloud
Search
Contact UsGet started
  • Blog
  • Pricing
  • Documentation
  • All Services
  • System Status
    • Featured
    • Infrastructure & Network
    • Data Platform
    • Containers
    • Developer tools
    • Serverless
    • Security
    • Monitoring & Resources
    • ML & AI
    • Business tools
  • All Solutions
    • By industry
    • By use case
    • Economics and Pricing
    • Security
    • Technical Support
    • Customer Stories
    • Cloud credits to scale your IT product
    • Gateway to Russia
    • Cloud for Startups
    • Education and Science
    • Yandex Cloud Partner program
  • Blog
  • Pricing
  • Documentation
© 2025 Direct Cursus Technology L.L.C.
Yandex Object Storage
    • All tutorials
    • Getting statistics on object queries with S3 Select
    • Getting website traffic statistics with S3 Select
    • Getting statistics on object queries using Yandex Query
    • Generating a resource-by-resource cost breakdown report using S3 Select
    • Server-side encryption
    • Integrating an L7 load balancer with CDN and Object Storage
    • Blue-green and canary deployment of service versions
    • Analyzing logs in DataLens
    • Mounting buckets to the file system of Yandex Data Processing hosts
    • Using Object Storage in Yandex Data Processing
    • Importing data from Object Storage, processing and exporting to Managed Service for ClickHouse®
    • Mounting a bucket as a disk in Windows
    • Migrating data from Yandex Data Streams using Yandex Data Transfer
    • Using hybrid storage in Yandex Managed Service for ClickHouse®
    • Loading data from Yandex Managed Service for OpenSearch to Yandex Object Storage using Yandex Data Transfer
    • Automatically copying objects from one bucket to another
    • Recognizing audio files in a bucket on a regular basis
    • Training a model in Yandex DataSphere on data from Object Storage
    • Connecting to Object Storage from VPC
    • Migrating data to Yandex Managed Service for PostgreSQL using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for Greenplum® using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for ClickHouse® using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for YDB using Yandex Data Transfer
    • Exchanging data between Yandex Managed Service for ClickHouse® and Yandex Data Processing
    • Uploading data from Yandex Managed Service for YDB using Yandex Data Transfer
    • Hosting a static Gatsby website in Object Storage
    • Migrating a database from Managed Service for PostgreSQL to Object Storage
    • Exchanging data between Yandex Managed Service for ClickHouse® and Yandex Data Processing
    • Importing data from Yandex Managed Service for PostgreSQL to Yandex Data Processing using Sqoop
    • Importing data from Yandex Managed Service for MySQL® to Yandex Data Processing using Sqoop
    • Migrating data from Yandex Object Storage to Yandex Managed Service for MySQL® using Yandex Data Transfer
    • Migrating a database from Yandex Managed Service for MySQL® to Yandex Object Storage
    • Exporting Greenplum® data to a cold storage in Yandex Object Storage
    • Loading data from Yandex Direct to a Yandex Managed Service for ClickHouse® data mart using Yandex Cloud Functions, Yandex Object Storage, and Yandex Data Transfer
    • Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch
    • Uploading Terraform states to Object Storage
    • Locking Terraform states using Managed Service for YDB
    • Visualizing Yandex Query data
    • Publishing game updates
    • VM backups using Hystax Acura
    • Backing up to Object Storage with CloudBerry Desktop Backup
    • Backing up to Object Storage with Duplicati
    • Backing up to Object Storage with Bacula
    • Backing up to Yandex Object Storage with Veeam Backup
    • Backing up to Object Storage with Veritas Backup Exec
    • Managed Service for Kubernetes cluster backups in Object Storage
    • Developing a custom integration in API Gateway
    • URL shortener
    • Storing application runtime logs
    • Developing a skill for Alice and a website with authorization
    • Creating an interactive serverless application using WebSocket
    • Deploying a web application using the Java Servlet API
    • Developing a Telegram bot
    • Replicating logs to Object Storage using Fluent Bit
    • Replicating logs to Object Storage using Data Streams
    • Uploading audit logs to ArcSight SIEM
    • Exporting audit logs to SIEM Splunk systems
    • Creating an MLFlow server for logging experiments and artifacts
    • Operations with data using Yandex Query
    • Federated data queries using Query
    • Recognizing text in image archives using Vision OCR
    • Converting a video to a GIF in Python
    • Automating tasks using Managed Service for Apache Airflow™
    • Processing files with usage details in Yandex Cloud Billing
    • Deploying a web app with JWT authorization in API Gateway and authentication in Firebase
    • Searching for Yandex Cloud events in Yandex Query
    • Searching for Yandex Cloud events in Object Storage
    • Creating an external table from a bucket table using a configuration file
    • Migrating databases from Google BigQuery to Managed Service for ClickHouse®
  • Terraform reference
  • Monitoring metrics
  • Audit Trails events
  • Bucket logs
  • Release notes
  • FAQ

In this article:

  • Get your cloud ready
  • Required paid resources
  • Create a bucket for storing logs
  • Enable log export
  • Get the data source ready
  • Create a ClickHouse® cluster
  • Change the user settings
  • Create a static key
  • Create a table in the database
  • Create a connection in DataLens
  • Create a dataset in DataLens
  • Create charts in DataLens
  • Create the first chart
  • Create the second chart
  • Create the third chart
  • Create a dashboard in DataLens and add charts to it
  • How to delete the resources you created
  1. Tutorials
  2. Analyzing logs in DataLens

Analyzing Object Storage logs in DataLens

Written by
Yandex Cloud
Improved by
SloNN
Updated at May 7, 2025
  • Get your cloud ready
    • Required paid resources
  • Create a bucket for storing logs
  • Enable log export
  • Get the data source ready
    • Create a ClickHouse® cluster
    • Change the user settings
    • Create a static key
    • Create a table in the database
  • Create a connection in DataLens
  • Create a dataset in DataLens
  • Create charts in DataLens
    • Create the first chart
    • Create the second chart
    • Create the third chart
  • Create a dashboard in DataLens and add charts to it
  • How to delete the resources you created

For a Yandex Object Storage bucket, you can enable action logging. The logs store info on operations involving a bucket and the objects in it. For example, analyzing bucket logs can help understand what caused a steep load increase or get the overall picture of traffic distribution.

You can create visualizations for your analysis using Yandex DataLens. You must transfer previously saved logs to the ClickHouse® database, which will be used as a source for DataLens.

To analyze logs and present the results in interactive charts:

  1. Get your cloud ready.
  2. Create a bucket for storing logs.
  3. Enable log export.
  4. Get the data source ready.
  5. Create a connection in DataLens.
  6. Create a dataset in DataLens.
  7. Create charts in DataLens.
  8. Create a dashboard in DataLens.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or register a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost includes:

  • Fee for data storage in Object Storage, data operations, and outbound traffic (see Object Storage pricing).
  • Fee for a continuously running Managed Service for ClickHouse® cluster (see Managed Service for ClickHouse® pricing).

Create a bucket for storing logsCreate a bucket for storing logs

Management console
AWS CLI
Terraform
API
  1. In the management console, select the folder where you want to create a bucket.
  2. From the list of services, select Object Storage.
  3. Click Create bucket.
  4. In the ** Name** field, enter a name for the bucket.
  5. In the Object read access and Object listing access fields, select Restricted.
  6. Click Create bucket.
  1. If you do not have the AWS CLI yet, install and configure it.

  2. Create a bucket:

    aws --endpoint-url https://storage.yandexcloud.net \
      s3 mb s3://<bucket_name>
    

    Result:

    make_bucket: <bucket_name>
    

Note

Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g., storage.admin, for the folder where you are going to create resources.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the documentation on the Terraform website or mirror website.

If you do not have Terraform yet, install it and configure its Yandex Cloud provider.

  1. Describe the properties for creating a service account and access key in the configuration file:

    ...
    // Creating a service account
    resource "yandex_iam_service_account" "sa" {
      name = "<service_account_name>"
    }
    
    // Assigning a role to a service account
    resource "yandex_resourcemanager_folder_iam_member" "sa-admin" {
      folder_id = "<folder_ID>"
      role      = "storage.admin"
      member    = "serviceAccount:${yandex_iam_service_account.sa.id}"
    }
    
    // Creating a static access key
    resource "yandex_iam_service_account_static_access_key" "sa-static-key" {
      service_account_id = yandex_iam_service_account.sa.id
      description        = "static access key for object storage"
    }
    
  2. Add the bucket properties to the configuration file:

    resource "yandex_storage_bucket" "bucket-logs" {
      access_key = yandex_iam_service_account_static_access_key.sa-static-key.access_key
      secret_key = yandex_iam_service_account_static_access_key.sa-static-key.secret_key
      bucket     = "<bucket_name>"
    }
    

    For more information about the yandex_storage_bucket resource, see this Terraform overview article.

  3. Make sure the settings are correct.

    1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

    2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

  4. Create a bucket.

    1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply
        
      2. Confirm updating the resources.

      3. Wait for the operation to complete.

Use the create REST API method.

Enable log exportEnable log export

Management console
AWS CLI
Terraform
API
  1. In the management console, select the bucket you want to enable logging for.
  2. In the left-hand panel, select Settings.
  3. Open the Logging tab.
  4. Enable Write logs.
  5. Select Bucket for log storage.
  6. In the Prefix field, specify the s3-logs/ prefix.
  7. Click Save.
  1. Create a file named log-config.json with the following contents:

    {
     "LoggingEnabled": {
        "TargetBucket": "<bucket_name>",
        "TargetPrefix": "s3-logs/"
     }
    }
    
  2. Run this command:

    aws s3api put-bucket-logging \
      --endpoint-url https://storage.yandexcloud.net \
      --bucket <bucket_name> \
      --bucket-logging-status file://log-config.json
    

    Where --bucket is the name of the bucket to enable action logging for.

To enable logging for a bucket you want to track:

  1. Open the Terraform configuration file and add the logging section to the bucket description fragment.

    resource "yandex_storage_bucket" "bucket-logs" {
      access_key = "<static_key_ID>"
      secret_key = "<secret_key>"
      bucket     = "<name_of_bucket_to_store_logs>"
    }
    
    resource "yandex_storage_bucket" "bucket" {
      access_key = "<static_key_ID>"
      secret_key = "<secret_key>"
      bucket     = "<source_bucket_name>"
      acl        = "private"
    
      logging {
        target_bucket = yandex_storage_bucket.bucket-logs.id
        target_prefix = "s3-logs/"
      }
    }
    

    Where:

    • access_key: Static access key ID.
    • secret_key: Secret access key value.
    • target_bucket: Reference to the log storage bucket.
    • target_prefix: Key prefix for objects with logs.

    For more information about yandex_storage_bucket properties in Terraform, see this Terraform article.

    1. In the terminal, change to the folder where you edited the configuration file.

    2. Make sure the configuration file is correct using the command:

      terraform validate
      

      If the configuration is correct, the following message is returned:

      Success! The configuration is valid.
      
    3. Run the command:

      terraform plan
      

      The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

    4. Apply the configuration changes:

      terraform apply
      
    5. Confirm the changes: type yes in the terminal and press Enter.

    This will create all the resources you need in the specified folder. You can check the new resources and their settings using the management console.

Use the REST API putBucketLogging method.

Get the data source readyGet the data source ready

Create a ClickHouse® clusterCreate a ClickHouse® cluster

To create a Managed Service for ClickHouse® cluster, you will need the vpc.user and managed-clickhouse.editor roles or higher. To learn more about assigning roles, see this Identity and Access Management article.

Management console
Yandex Cloud CLI
Terraform
API
  1. In the management console, select the folder where you want to create a cluster.

  2. From the list of services, select Managed Service for ClickHouse.

  3. In the window that opens, click Create cluster.

  4. Specify the ClickHouse® cluster settings:

    1. Under Basic parameters, specify s3-logs in the Cluster name field.

    2. Under Resources, select burstable in the Type field.

    3. Under Hosts, click and enable the Public access option. Click Save.

    4. Under DBMS settings:

      • In the User management via SQL field, select Disabled.
      • In the Username field, specify user.
      • In the Password field, set a password.
      • In the DB name field, specify s3_data.

      Memorize the database name.

    5. Under Service settings, enable these options:

      • DataLens access.
      • Access from the management console.
  5. Click Create cluster.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a cluster:

  1. Check whether the folder has any subnets for the cluster hosts:

    yc vpc subnet list
    

    If there are no subnets in the folder, create the required subnets in VPC.

  2. Specify the cluster properties in the creation command:

    yc managed-clickhouse cluster create \
       --name s3-logs \
       --environment production \
       --network-name <network_name> \
       --host type=clickhouse,zone-id=<availability_zone>,subnet-id=<subnet_ID> \
       --clickhouse-resource-preset b2.medium \
       --clickhouse-disk-type network-ssd \
       --clickhouse-disk-size 10 \
       --user name=user,password=<user_password> \
       --database name=s3_data \
       --datalens-access=true \
       --websql-access=true
    
  1. Add the cluster description and cluster hosts to the configuration file:

    resource "yandex_mdb_clickhouse_cluster" "s3-logs" {
      name                = "s3-logs"
      environment         = "PRODUCTION"
      network_id          = yandex_vpc_network.<network_name_in_Terraform>.id
    
      clickhouse {
        resources {
          resource_preset_id = "b2.medium"
          disk_type_id       = "network-ssd"
          disk_size          = 10
        }
      }
    
      database {
        name = "s3_data"
      }
    
      user {
        name     = "user"
        password = "<password>"
        permission {
          database_name = "s3_data"
        }
      }
    
      host {
        type      = "CLICKHOUSE"
        zone      = "<availability_zone>"
        subnet_id = yandex_vpc_subnet.<subnet_name_in_Terraform>.id
      }
    
      access {
        datalens  = true
        web_sql   = true
      }
    }
    

    To learn more about the resources you can create with Terraform, see the Terraform documentation.

  2. Make sure the settings are correct.

    1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

    2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

  3. Create a cluster:

    1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply
        
      2. Confirm updating the resources.

      3. Wait for the operation to complete.

Use the create REST API method.

After creating the cluster, you will be automatically redirected to the Clusters page.

Wait until the cluster status switches to Alive.

Change the user settingsChange the user settings

Management console
  1. Select the s3-logs cluster.
  2. Navigate to the Users tab.
  3. Click and select Configure.
  4. Click Additional settings → Settings.
  5. In the Date time input format field, select best_effort.
  6. Click Save.

Create a static keyCreate a static key

To create a table with access to Object Storage, you will need a static key. Create one and save its ID and secret part.

Create a table in the databaseCreate a table in the database

Management console
  1. Select the s3-logs cluster.

  2. Navigate to the SQL tab.

  3. In the Password field, enter the password.

  4. Click Connect.

  5. In the window on the right, write this SQL query:

    CREATE TABLE s3_data.s3logs
    (
       bucket String,              -- Bucket name.
       bytes_received Int64,       -- Request size in bytes.
       bytes_send Int64,           -- Response size in bytes.
       handler String,             -- Request method in this format: REST.<HTTP method>.<subject>.
       http_referer String,        -- Request source URL.
       ip String,                  -- User IP address.
       method String,              -- HTTP request method.
       object_key String,          -- Object key in URL-encoded format.
       protocol String,            -- Data transfer protocol version.
       range String,               -- HTTP header defining the byte range to load from the object.
       requester String,           -- User ID.
       request_args String,        -- Arguments of the URL request.
       request_id String,          -- Request ID.
       request_path String,        -- Full request path.
       request_time Int64,         -- Request processing time in milliseconds.
       scheme String,              -- Data transfer protocol type.
                                   -- The possible values are as follows:
                                   -- * http: Application layer protocol.
                                   -- * https: Application layer protocol with encryption support.
       ssl_protocol String,        -- Security protocol.
       status Int64,               -- HTTP response code.
       storage_class String,       -- Object storage class.
       timestamp DateTime,         -- Date and time of the bucket operation in the YYYY-MM-DDTHH:MM:SSZ format.
       user_agent String,          -- Client application (user agent) that executed the request.
       version_id String,          -- Object version.
       vhost String                -- Virtual host of the request.
                                   -- The possible values are as follows:
                                   -- * storage.yandexcloud.net.
                                   -- * <bucket_name>.storage.yandexcloud.net.
                                   -- * website.yandexcloud.net.
                                   -- * <bucket_name>.website.yandexcloud.net.
    )
    ENGINE = S3(
          'https://storage.yandexcloud.net/<bucket_name>/s3-logs/*',
          '<key_ID>',
          '<secret_key>',
          'JSONEachRow'
       )
    SETTINGS date_time_input_format='best_effort';
    
  6. Click Execute.

Create a connection in DataLensCreate a connection in DataLens

Management console
  1. Select the s3-logs cluster.

  2. Navigate to the DataLens tab.

  3. In the window that opens, click Create connection.

  4. Fill in the connection settings:

    1. Add a connection name: s3-logs-con.
    2. In the Cluster field, select s3-logs.
    3. In the Host name field, select the ClickHouse® host from the drop-down list.
    4. Enter the DB username and password.
  5. Click Confirm connection.

  6. After checking the connection, click Create connection.

  7. In the window that opens, enter a name for the connection and click Create.

Create a dataset in DataLensCreate a dataset in DataLens

  1. Click Create dataset.

  2. In the new dataset, move the s3_data.s3logs table to the workspace.

  3. Navigate to the Fields tab.

  4. Click Add field.

  5. Create a calculated field with the file type:

    • Field name: object_type
    • Formula: SPLIT([object_key], '.', -1)
  6. Click Create.

  7. In the top-right corner, click Save.

  8. Enter the s3-dataset name for the dataset and click Create.

  9. Once the dataset is saved, click Create chart in the top-right corner.

Create charts in DataLensCreate charts in DataLens

Create the first chartCreate the first chart

To visualize the number of requests to a bucket via different methods, create a pie chart:

  1. Select Pie chart as the visualization type.
  2. Drag the method field from the Dimensions section to the Colors section.
  3. Drag the request_id field from the Dimensions section to the Measures section.
  4. In the top-right corner, click Save.
  5. In the window that opens, enter the S3 - Method pie name for the new chart and click Save.

Create the second chartCreate the second chart

To visualize the number of requests ratio by object type, create a bar chart:

  1. Copy the chart from the previous step:

    1. In the top-right corner, click the down arrow next to the Save button.
    2. Click Save as.
    3. In the window that opens, enter the S3 - Object type bars name for the new chart and click Save.
  2. Select Bar chart as the visualization type. The method and request_id fields will automatically appear in the X and Y sections, respectively.

  3. Delete the method field from the X section and drag the object_type field there.

  4. In the top-right corner, click Save.

Create the third chartCreate the third chart

To visualize the distribution of outbound traffic by day, create a bar chart:

  1. Copy the chart from the previous step:

    1. In the top-right corner, click the check mark next to the Save button.
    2. Click Save as.
    3. In the window that opens, enter the S3 - Traffic generated by days name for the new chart and click Save.
  2. Drag the object_type field from the X section to the Filters section.

  3. In the window that opens, select the types of objects to display in the chart and click Apply filters.

  4. Drag the timestamp field from the Dimensions section to the X section.

  5. Delete the request_id field from the Y section and drag the bytes_send field there.

  6. In the top-right corner, click Save.

Create a dashboard in DataLens and add charts to itCreate a dashboard in DataLens and add charts to it

  1. Go to the DataLens home page.
  2. Click Create dashboard.
  3. Enter the S3 Logs Analysis name for the dashboard and click Create.
  4. In the top-right corner, click Add and select Chart.
  5. In the Chart field, click Select and select the S3 - Method pie chart from the list.
  6. Click Add. You will now see the chart on the dashboard.
  7. Repeat the previous steps for the S3 - Object type bars and S3 - Traffic generated by days charts.

How to delete the resources you createdHow to delete the resources you created

Delete the resources you no longer need to avoid paying for them:

  • Delete the bucket.
  • Delete the s3-logs cluster.

ClickHouse® is a registered trademark of ClickHouse, Inc.

Was the article helpful?

Previous
Blue-green and canary deployment of service versions
Next
Overview
© 2025 Direct Cursus Technology L.L.C.