Managing bucket logging

To monitor bucket operations, enable logging.

Detailed information about requests to the source bucket will be saved in an object in the target bucket. The source and target buckets must be different.

Object Storage does not guarantee complete and timely logging.

By default, logging is disabled. After you enable it, Object Storage will log info about bucket operations once an hour.

Enabling loggingEnabling logging

To log bucket access:

1. Use an existing target bucket or create a new one. This bucket will store logs.

   How to create a bucket

   To create a bucket, you need the minimum storage.editor role for the folder.

   Management console

   Yandex Cloud CLI

   AWS CLI

   Terraform

   API

   1. In the management console, select the folder where you want to create a bucket.
   2. Select Object Storage.
   3. In the top panel, click Create bucket.
   4. On the bucket creation page:

      1. Enter a name for the bucket consistent with the naming requirements.

         By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.

      2. Limit the maximum bucket size, if required.

         If the value is 0, the size is not limited. It is similar to the enabled No limit option.

      3. Configure public access to read objects in the bucket, get a list of objects, and read bucket settings:

         • Restricted: Authorized Yandex Cloud users only.
         • Public: All users.

         Warning

         Public access is granted to an unlimited number of anonymous users. Use it only when other access grant mechanisms are not available.

      4. Select the default storage class:

         • Standard.
         • Cold.
         • Ice.

         "Cold" classes are for long-term storage of objects you intend to use less frequently. The "colder" the storage, the cheaper it is to store data in, but the more expensive it is to read from and write to it.

      5. Add labels, if required:

         1. Click Add label.
         2. Enter a label in key: value format.
         3. Press Enter.

      6. Click Create bucket to complete the operation.

   If you do not have the Yandex Cloud CLI yet, install and initialize it.

   The folder specified in the CLI profile is used by default. You can specify a different folder through the --folder-name or --folder-id parameter.

   1. View the description of the CLI command to create a bucket:

      yc storage bucket create --help
      

   2. Create a bucket in the default folder:

      yc storage bucket create --name <bucket_name>
      

      Where --name is the name of the bucket. This is a required parameter. For more information, see Bucket naming rules.

      By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.

      Result:

      name: example
      folder_id: b1gmit33ngp6********
      anonymous_access_flags:
      read: false
      list: false
      default_storage_class: STANDARD
      versioning: VERSIONING_DISABLED
      max_size: "53687091200"
      acl: {}
      created_at: "2022-12-16T14:05:12.196007Z"
      

      Optional parameters

      • --default-storage-class: Storage class. The possible values are:
      • standard: Standard storage. It is installed by default.
      • cold: Cold storage.
      • ice: Ice storage.

      Cold classes are designed to store objects that you plan to use less frequently for longer periods of time. The colder the storage, the cheaper it is to store data in, but the more expensive it is to read from and write to it.

      • --max-size: Maximum bucket size, in bytes. The default value is 0 (unlimited).
      • Parameters for enabling public access to a bucket:
      • --public-read: Enables public read access to bucket objects.
      • --public-list: Enables public view access to the list of bucket objects.
      • --public-config-read: Enables public read access to bucket settings.

      By default, public access to the bucket is disabled.

      Warning

      Public access is granted to an unlimited number of anonymous users. Use it only when other access grant mechanisms are not available.

      • Parameters to configure the bucket ACL:

      • --acl: Predefined ACL. For a list of possible values, see Predefined ACLs. You cannot use this parameter together with --grants.

      • --grants: This parameter configures permissions for individual users, service accounts, user groups, and public groups (a group of all internet users or a group of all authenticated Yandex Cloud users). You cannot use this parameter together with --acl. The parameter value is specified in the following format: grant-type=<permission_grantee_type>,grantee-id=<grantee_ID>,permission=<permission_type>, where:

        • grant-type: Permission grantee type. The possible values are:
        • grant-type-account: User, service account, or user group.
        • grant-type-all-authenticated-users: Public group that includes all authenticated Yandex Cloud users.
        • grant-type-all-users: Public group that includes all internet users.
        • grantee-id: ID of the user, service account, or user group you need to grant a permission to. It is specified only if grant-type=grant-type-account.
        • permission: ACL permission type. The possible values are permission-full-control, permission-write, and permission-read. Learn more about permissions in Permission types.

        To configure multiple permissions, specify the --grants parameter multiple times.

      By default, an empty ACL is created for each new bucket.

      Learn more about the yc storage bucket create command in the YC CLI reference.

   If you do not have the AWS CLI yet, install and configure it.

   To create a bucket, assign the storage.editor role to the service account used by the AWS CLI.

   In the terminal, run this command:

   aws s3api create-bucket \
     --endpoint-url=https://storage.yandexcloud.net \
     --bucket <bucket_name>
   

   Where:

   • --endpoint-url: Object Storage endpoint.
   • --bucket: Bucket name.

   Note

   By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage. For more information, see Bucket naming rules.

   Result:

   {
     "Location": "/<bucket_name>"
   }
   

   The new bucket will have the following parameters:

   • Maximum size unlimited.
   • Limited access to read objects, get a list of objects, and read bucket setting.
   • Storage class: Standard.

   Optional parameters

   You can apply a predefined ACL to a bucket or configure permissions for individual users, service accounts, user groups, and public groups, such as a group of all internet users or a group of all authenticated Yandex Cloud users. You cannot use these settings together: a bucket can have either a predefined ACL or individual permissions.

   Note

   To manage bucket ACL settings, assign the storage.admin role to the service account used by the AWS CLI.

   Predefined ACL

   aws s3api create-bucket \
     --endpoint-url=https://storage.yandexcloud.net \
     --bucket <bucket_name> \
     --acl <predefined_ACL>
   

   Where --acl is a predefined ACL. For the list of values, see Predefined ACLs.

   Individual permissions

   aws s3api create-bucket \
     --endpoint-url=https://storage.yandexcloud.net \
     --bucket <bucket_name> \
     <permission_type> <permission_grantee>
   

   Where:

   • The possible types of ACL permissions are as follows:

     • --grant-read: Permission to access the list of objects in the bucket, read various bucket settings (lifecycle, CORS, and static hosting), and read all objects in the bucket.
     • --grant-write: Permission to write, overwrite, and delete objects in the bucket. It can only be used together with --grant-read.
     • --grant-full-control: Full access to the bucket and objects in it.

     You can specify multiple permissions in a single command.

   • The possible permission grantees are as follows:

     • id=<grantee_ID>: ID of the user, service account, or user group you need to grant a permission to.
     • uri=http://acs.amazonaws.com/groups/global/AuthenticatedUsers: Public group that includes all authenticated Yandex Cloud users.
     • uri=http://acs.amazonaws.com/groups/global/AllUsers: Public group that includes all internet users.

   By default, an empty ACL is created for each new bucket.

   For more information about the aws s3api create-bucket command, see the AWS documentation.

   Note

   Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g., storage.admin, for the folder where you are going to create resources.

   With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

   Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

   For more information about the provider resources, see the documentation on the Terraform website or mirror website.

   If you don't have Terraform, install it and configure the Yandex Cloud provider.

   1. In the configuration file, define the parameters of the resources you want to create:

      terraform {
        required_providers {
          yandex = {
            source = "yandex-cloud/yandex"
          }
        }
        required_version = ">= 0.13"
      }
      
      // Configuring a provider
      provider "yandex" {
        token     = "<IAM_or_OAuth_token>"
        cloud_id  = "<cloud_ID>"
        folder_id = "<folder_ID>"
        zone      = "ru-central1-a"
      }
      
      // Creating a service account
      resource "yandex_iam_service_account" "sa" {
        name = "<service_account_name>"
      }
      
      // Assigning a role to a service account
      resource "yandex_resourcemanager_folder_iam_member" "sa-admin" {
        folder_id = "<folder_ID>"
        role      = "storage.admin"
        member    = "serviceAccount:${yandex_iam_service_account.sa.id}"
      }
      
      // Creating a static access key
      resource "yandex_iam_service_account_static_access_key" "sa-static-key" {
        service_account_id = yandex_iam_service_account.sa.id
        description        = "static access key for object storage"
      }
      
      // Creating a bucket using a key
      resource "yandex_storage_bucket" "test" {
        access_key            = yandex_iam_service_account_static_access_key.sa-static-key.access_key
        secret_key            = yandex_iam_service_account_static_access_key.sa-static-key.secret_key
        bucket                = "<bucket_name>"
        max_size              = <maximum_bucket_size>
        default_storage_class = "<storage_class>"
        anonymous_access_flags {
          read        = <true|false>
          list        = <true|false>
          config_read = <true|false>
        }
        tags = {
          <key_1> = "<value_1>"
          <key_2> = "<value_2>"
          ...
          <key_n> = "<value_n>"
        }
      }
      

      Where:

      • yandex_iam_service_account: Description of the service account to create and use the bucket:

        • name: Service account name.

        • bucket: Bucket name.

          By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.

        • max_size: Maximum bucket size, in bytes.

        • default_storage_class: Storage class. The possible values are:

          • standard: Standard storage.
          • cold: Cold storage.
          • ice: Ice storage.

          Cold classes are designed to store objects that you plan to use less frequently for longer periods of time. The colder the storage, the cheaper it is to store data in, but the more expensive it is to read from and write to it.

        • anonymous_access_flags: Access settings:

          • read: Public read access to bucket objects.
          • list: Public view access to the list of bucket objects.
          • config_read: Public read access to bucket settings.

        • tags: Bucket labels in key = "value" format.

      name: This is a required parameter. Other parameters are optional. By default, the max-size value is 0, public access to the bucket is disabled, and the storage class is set to standard.

      For more information about the yandex_storage_bucket parameters in Terraform, see the relevant Terraform article.

   2. Create the resources:

      1. In the terminal, change to the folder where you edited the configuration file.

      2. Make sure the configuration file is correct using the command:

         terraform validate
         

         If the configuration is correct, the following message is returned:

         Success! The configuration is valid.
         

      3. Run the command:

         terraform plan
         

         The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

      4. Apply the configuration changes:

         terraform apply
         

      5. Confirm the changes: type yes in the terminal and press Enter.

   Terraform will create all the required resources. You can check the new resources and their settings using the management console.

   To create a bucket, use the create REST API method for the Bucket resource, the BucketService/Create gRPC API call, or the create S3 API method.

   Note

   To write logs to the target bucket with an access policy configured, add a rule to this policy to allow any account to perform the PutObject action, and specify the log object key prefix, e.g., logs/, as a resource.

2. Enable logging for the source bucket you want to monitor.

   Management console

   AWS CLI

   Terraform

   API

   1. In the management console, select Object Storage from the list of services and go to the bucket you want to configure logging for.
   2. In the left-hand panel, select Settings.
   3. Select the Logging tab.
   4. Enable Write logs.
   5. Select Bucket for log storage.
   6. In the Prefix field, specify the prefix for log storage.
   7. Click Save.

   To enable logging via the AWS CLI:

   1. Create a file with logging settings in JSON format. Here is an example:

      {
         "LoggingEnabled": {
            "TargetBucket": "<target_bucket_name>",
            "TargetPrefix": "<key_prefix>"
         }
      }
      

      Where:

      • TargetBucket: Name of the target bucket for the logs.
      • TargetPrefix: Prefix of the key used for objects with logs, e.g., logs/.

   2. Enable logging in the bucket:

      aws s3api put-bucket-logging \
        --bucket <source_bucket_name> \
        --endpoint-url https://storage.yandexcloud.net \
        --bucket-logging-status file://<path_to_configuration_file>
      

      Where:

      • --bucket: Name of the source bucket to enable logging for.
      • --bucket-logging-status: Path to the logging settings file.

   Note

   Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g., storage.admin, for the folder where you are going to create resources.

   With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

   Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

   For more information about the provider resources, see the documentation on the Terraform website or mirror website.

   If you don't have Terraform, install it and configure the Yandex Cloud provider.

   To enable logging for the bucket you want to monitor:

   1. Open the Terraform configuration file and add the logging section to the bucket description.

      resource "yandex_storage_bucket" "log_bucket" {
        access_key = "<static_key_ID>"
        secret_key = "<secret_key>"
        bucket     = "<name_of_bucket_for_storing_logs>"
      }
      
      resource "yandex_storage_bucket" "bucket" {
        access_key = "<static_key_ID>"
        secret_key = "<secret_key>"
        bucket     = "<source_bucket_name>"
        acl        = "private"
      
        logging {
          target_bucket = yandex_storage_bucket.log_bucket.id
          target_prefix = "log/"
        }
      }
      

      Where:

      • access_key: Static access key ID.
      • secret_key: Secret access key value.
      • target_bucket: Reference to the log storage bucket.
      • target_prefix: Prefix of the key used for objects with logs, e.g., logs/.

      For more information about the yandex_storage_bucket resource parameters in Terraform, see this TF provider article.

      1. In the terminal, change to the folder where you edited the configuration file.

      2. Make sure the configuration file is correct using the command:

         terraform validate
         

         If the configuration is correct, the following message is returned:

         Success! The configuration is valid.
         

      3. Run the command:

         terraform plan
         

         The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

      4. Apply the configuration changes:

         terraform apply
         

      5. Confirm the changes: type yes in the terminal and press Enter.

      This will create all the resources you need in the specified folder. You can check the new resources and their settings using the management console.

   To enable logging for a bucket, use the putBucketLogging S3 API method.

   Here is an example of an HTTP request body:

   <BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01">
     <LoggingEnabled>
       <TargetBucket>target_bucket_name</TargetBucket>
       <TargetPrefix>key_prefix</TargetPrefix>
     </LoggingEnabled>
   </BucketLoggingStatus>
   

   Where:

   • TargetBucket: Target bucket name.
   • TargetPrefix: Prefix of the key used for objects with logs, e.g., logs/.

If you want to delete the target bucket, first disable logging or select another target bucket for storing logs. You can only delete an empty bucket.

Getting logging settingsGetting logging settings

To get the target bucket name and the log object key prefix, follow these steps:

aws s3api get-bucket-logging \
  --bucket <bucket_name> \
  --output json \
  --endpoint-url https://storage.yandexcloud.net

{
    "LoggingEnabled": {
      "TargetBucket": "<bucket_name>",
      "TargetPrefix": "<key_prefix>"
    }
}

HTTP/1.1 200
<?xml version="1.0" encoding="UTF-8"?>
<BucketLoggingStatus>
   <LoggingEnabled>
      <TargetBucket>target_bucket_name</TargetBucket>
      <TargetPrefix>logs/</TargetPrefix>
   </LoggingEnabled>
</BucketLoggingStatus>

Getting logsGetting logs

To get logs, download the object prefixed with logs/ from the target bucket:

Disabling loggingDisabling logging

To disable logging, follow these steps:

aws s3api put-bucket-logging \
    --bucket <bucket_name> \
    --endpoint-url https://storage.yandexcloud.net \
    --bucket-logging-status {}

<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01" />