Managing bucket action logging

To track operations with the bucket, enable logging.

Detailed information about requests to the source bucket will be saved in an object in the target bucket. The source and target buckets must be different.

Object Storage does not guarantee complete and timely logging.

By default, logging is disabled. After you enable it, Object Storage will save information about actions with the bucket once an hour.

Enable loggingEnable logging

To log requests to the bucket:

1. Use an existing target bucket or create a new one. This is the bucket your logs will be written to.

   How to create a bucket

   To create a bucket, you need the minimum storage.editor role for the folder.

   Management console

   Yandex Cloud CLI

   AWS CLI

   Terraform

   API

   1. In the management console, select the folder you want to create a bucket in.
   2. Select Object Storage.
   3. In the top panel, click Create bucket.
   4. On the bucket creation page:

      1. Enter a name for the bucket according to the naming requirements.

         By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.

      2. Limit the maximum bucket size, if required.

         If the value is 0, the maximum size is not limited and is similar to the enabled No limit option.

      3. Set the public public access parameters to read objects in the bucket, get a list of objects, and read bucket settings:

         • Restricted: Authorized Yandex Cloud users only
         • Public: All users

         Warning

         Public access is granted to an unlimited number of anonymous users. Use it only when other access grant mechanisms are not available.

      4. Select the default storage class:

         • Standard.
         • Cold.
         • Ice.

         "Cold" classes are for long-term storage of objects you intend to use less frequently. The "colder" the storage, the cheaper it is to store data in, but the more expensive it is to read from and write to it.

      5. Add labels, if required:

         1. Click Add label.
         2. Enter a label in key: value format.
         3. Click Enter.

      6. Click Create bucket to complete the operation.

   If you do not have the Yandex Cloud command line interface yet, install and initialize it.

   The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

   1. View a description of the CLI command to create a bucket:

      yc storage bucket create --help
      

   2. Create a bucket in the default folder:

      yc storage bucket create --name <bucket_name>
      

      Where --name is the name of the bucket. This is a required parameter. For more information, see Bucket naming rules.

      By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.

      Result:

      name: example
      folder_id: b1gmit33ngp6********
      anonymous_access_flags:
        read: false
        list: false
      default_storage_class: STANDARD
      versioning: VERSIONING_DISABLED
      max_size: "53687091200"
      acl: {}
      created_at: "2022-12-16T14:05:12.196007Z"
      

      Optional parameters

      • --default-storage-class: Storage class. Possible values:

        • standard: Standard storage. It is installed by default.
        • cold: Cold storage.
        • ice: Ice storage.

        "Cold" classes are designed to store objects that you plan to use less frequently for longer periods of time. The "colder" your storage is, the less you pay for storing data; however, the costs of reading and writing data increase.

      • --max-size: Maximum bucket size, in bytes. Default value: 0 (unlimited).

      • Parameters for enabling public access to a bucket:

        • --public-read: Enables public access to reading bucket objects.
        • --public-list: Enables public access to viewing the list of bucket objects.
        • --public-config-read: Enables public access to reading bucket settings.

        By default, no public access to buckets is allowed.

        Warning

        Public access is granted to an unlimited number of anonymous users. Use it only when other access grant mechanisms are not available.

      • Parameters to configure the bucket ACL:

        • --acl: Predefined ACL. For a list of possible values, see Predefined ACLs. You cannot use this parameter together with the --grants parameter.

        • --grants: Configures permissions for individual users, service accounts, user groups, and public groups (a group of all internet users or a group of all authenticated Yandex Cloud users). You cannot use this parameter together with the --acl parameter. The parameter value is specified in the following format: grant-type=<permission_grantee_type>,grantee-id=<user_ID>,permission=<permission_type>, where:

          • grant-type: Permission grantee type. The possible values are:
            • grant-type-account: User, service account, or user group.
            • grant-type-all-authenticated-users: Public group that includes all authenticated Yandex Cloud users.
            • grant-type-all-users: Public group that includes all internet users.
          • grantee-id: ID of the user, service account, or user group you need to grant a permission to. Specified only if grant-type=grant-type-account.
          • permission: ACL permission type. Possible values: permission-full-control, permission-write, permission-read. For more information about permissions, see Permission types.

          To configure multiple permissions, specify the --grants parameter multiple times.

        By default, an empty ACL is created for each new bucket.

      For more information about the yc storage bucket create command, see the YC CLI reference.

   If you do not have the AWS CLI yet, install and configure it.

   To create a bucket, assign the storage.editor role to the service account used by the AWS CLI.

   In the terminal, run this command:

   aws s3api create-bucket \
     --endpoint-url=https://storage.yandexcloud.net \
     --bucket <bucket_name>
   

   Where:

   • --endpoint-url: Object Storage endpoint
   • --bucket: Bucket name

   Note

   By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage. For more information, see Bucket naming rules.

   Result:

   {
     "Location": "/<bucket_name>"
   }
   

   The new bucket will have the following parameters:

   • No limitations to the maximum size.
   • Limited access to read objects, get a list of objects, and read bucket setting.
   • Storage class: Standard.

   Optional parameters

   You can apply a predefined ACL to a bucket or configure permissions for individual users, service accounts, user groups and public groups (e.g., a group of all internet users or a group of all authenticated Yandex Cloud users). These settings are not compatible: a bucket should have either a predefined ACL or a set of individual permissions.

   Note

   To manage bucket ACL settings, assign the storage.admin role to the service account used by the AWS CLI.

   Predefined ACL

   aws s3api create-bucket \
     --endpoint-url=https://storage.yandexcloud.net \
     --bucket <bucket_name> \
     --acl <predefined_ACL>
   

   Where --acl is a predefined ACL. For the list of values, see Predefined ACLs.

   Individual permissions

   aws s3api create-bucket \
     --endpoint-url=https://storage.yandexcloud.net \
     --bucket <bucket_name> \
     <permission_type> <permission_grantee>
   

   Where:

   • Possible types of ACL permissions:

     • --grant-read: Permission to access the list of objects in the bucket, read various bucket settings (lifecycle, CORS, static hosting), and read all objects in the bucket.
     • --grant-write: Permission to write, overwrite, and delete objects in the bucket. Can only be used together with --grant-read.
     • --grant-full-control: Full access to the bucket and the objects in it.

     You can set multiple permissions within the same command.

   • The possible permission grantees are:

     • id=<grantee_ID>: ID of the user, service account, or user group you need to grant a permission to.
     • uri=http://acs.amazonaws.com/groups/global/AuthenticatedUsers: Public group that includes all authenticated Yandex Cloud users.
     • uri=http://acs.amazonaws.com/groups/global/AllUsers: Public group that includes all internet users.

   By default, an empty ACL is created for each new bucket.

   For more information about the aws s3api create-bucket command, see the AWS documentation.

   Note

   Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g., storage.admin, for the folder where you are going to create resources.

   Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

   For more information about the provider resources, see the documentation on the Terraform website or the mirror.

   If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

   If you don't have Terraform, install it and configure the Yandex Cloud provider.

   1. In the configuration file, describe the parameters of the resources you want to create:

      terraform {
        required_providers {
          yandex = {
            source = "yandex-cloud/yandex"
          }
        }
        required_version = ">= 0.13"
      }
      
      // Configuring a provider
      provider "yandex" {
        token     = "<IAM_or_OAuth_token>"
        cloud_id  = "<cloud_ID>"
        folder_id = "<folder_ID>"
        zone      = "ru-central1-a"
      }
      
      // Creating a service account
      resource "yandex_iam_service_account" "sa" {
        name = "<service_account_name>"
      }
      
      // Assigning a role to a service account
      resource "yandex_resourcemanager_folder_iam_member" "sa-admin" {
        folder_id = "<folder_ID>"
        role      = "storage.admin"
        member    = "serviceAccount:${yandex_iam_service_account.sa.id}"
      }
      
      // Creating a static access key
      resource "yandex_iam_service_account_static_access_key" "sa-static-key" {
        service_account_id = yandex_iam_service_account.sa.id
        description        = "static access key for object storage"
      }
      
      // Creating a bucket using a key
      resource "yandex_storage_bucket" "test" {
        access_key            = yandex_iam_service_account_static_access_key.sa-static-key.access_key
        secret_key            = yandex_iam_service_account_static_access_key.sa-static-key.secret_key
        bucket                = "<bucket_name>"
        max_size              = <maximum_bucket_size>
        default_storage_class = "<storage_class>"
        anonymous_access_flags {
          read        = <true|false>
          list        = <true|false>
          config_read = <true|false>
        }
        tags = {
          <key_1> = "<value_1>"
          <key_2> = "<value_2>"
          ...
          <key_n> = "<value_n>"
        }
      }
      

      Where:

      • yandex_iam_service_account: Description of the service account to create and use the bucket:

        • name: Service account name

        • bucket: Bucket name

          By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.

        • max_size: Maximum bucket size, in bytes.

        • default_storage_class: Storage class. Possible values:

          • standard: Standard storage.
          • cold: Cold storage.
          • ice: Ice storage.

          "Cold" classes are designed to store objects that you plan to use less frequently for longer periods of time. The "colder" your storage is, the less you pay for storing data; however, the costs of reading and writing data increase.

        • anonymous_access_flags: Access settings:

          • read: Public access to read bucket objects.
          • list: Public access to view the list of bucket objects.
          • config_read: Public access to read bucket settings.

        • tags: Bucket labels in key = "value" format.

      name: Required parameter. Other parameters are optional. By default, the max-size value is 0, public access to the bucket is disabled, and the storage class is set to standard.

      For more information about the yandex_storage_bucket resource parameters in Terraform, see the relevant provider documentation.

   2. Create resources:

      1. In the terminal, change to the folder where you edited the configuration file.

      2. Make sure the configuration file is correct using the command:

         terraform validate
         

         If the configuration is correct, the following message is returned:

         Success! The configuration is valid.
         

      3. Run the command:

         terraform plan
         

         The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

      4. Apply the configuration changes:

         terraform apply
         

      5. Confirm the changes: type yes in the terminal and press Enter.

   Terraform will create all the required resources. You can check the new resources and their configuration using the management console.

   To create a bucket, use the create REST API method for the Bucket resource, the BucketService/Create gRPC API call, or the create S3 API method.

   Note

   To write logs to the target bucket with an access policy configured, add there a rule to allow any account to perform the PutObject action, and specify the log object key prefix, e.g., logs/, as a resource.

2. Enable logging in the source bucket that you want to track.

   Management console

   AWS CLI

   Terraform

   API

   1. In the management console, select Object Storage from the list of services and go to the bucket you want to configure logging for.
   2. In the left-hand panel, select Settings.
   3. Select the Logging tab.
   4. Enable Write logs.
   5. Select Bucket for log storage.
   6. In the Prefix field, specify the prefix with which the logs will be saved.
   7. Click Save.

   To enable logging via the AWS CLI:

   1. Create a file with logging settings in JSON format. See the examples below.

      {
         "LoggingEnabled": {
            "TargetBucket": "<target_bucket_name>",
            "TargetPrefix": "<key_prefix>"
         }
      }
      

      Where:

      • TargetBucket: Name of the target bucket for the logs.
      • TargetPrefix: Prefix of the key used for objects with logs, e.g., logs/.

   2. Enable logging in the bucket:

      aws s3api put-bucket-logging \
        --bucket <source_bucket_name> \
        --endpoint-url https://storage.yandexcloud.net \
        --bucket-logging-status file://<path_to_configuration_file>
      

      Where:

      • --bucket: Name of the source bucket to enable action logging for.
      • --bucket-logging-status: Path to the logging settings file.

   Note

   Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g., storage.admin, for the folder where you are going to create resources.

   Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

   For more information about the provider resources, see the documentation on the Terraform website or the mirror.

   If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

   If you don't have Terraform, install it and configure the Yandex Cloud provider.

   To enable logging for a bucket you want to track:

   1. Open the Terraform configuration file and add the logging section to the bucket description fragment.

      resource "yandex_storage_bucket" "log_bucket" {
        access_key = "<static_key_ID>"
        secret_key = "<secret_key>"
        bucket     = "<name_of_bucket_to_store_logs>"
      }
      
      resource "yandex_storage_bucket" "bucket" {
        access_key = "<static_key_ID>"
        secret_key = "<secret_key>"
        bucket     = "<source_bucket_name>"
        acl        = "private"
      
        logging {
          target_bucket = yandex_storage_bucket.log_bucket.id
          target_prefix = "log/"
        }
      }
      

      Where:

      • access_key: Static access key ID.
      • secret_key: Secret access key value.
      • target_bucket: Reference to the log storage bucket.
      • target_prefix: Prefix of the key used for objects with logs, e.g., logs/.

      For more information about the yandex_storage_bucket resource parameters in Terraform, see the provider documentation.

      1. In the terminal, change to the folder where you edited the configuration file.

      2. Make sure the configuration file is correct using the command:

         terraform validate
         

         If the configuration is correct, the following message is returned:

         Success! The configuration is valid.
         

      3. Run the command:

         terraform plan
         

         The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

      4. Apply the configuration changes:

         terraform apply
         

      5. Confirm the changes: type yes in the terminal and press Enter.

      All the resources you need will then be created in the specified folder. You can check the new resources and their configuration using the management console.

   To enable logging for your bucket, use the putBucketLogging S3 API method.

   Example of the HTTP request body:

   <BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01">
     <LoggingEnabled>
       <TargetBucket>target_bucket_name</TargetBucket>
       <TargetPrefix>key_prefix</TargetPrefix>
     </LoggingEnabled>
   </BucketLoggingStatus>
   

   Where:

   • TargetBucket: Target bucket name.
   • TargetPrefix: Prefix of the key used for objects with logs, e.g., logs/.

If you want to delete the target bucket, first disable logging or select another target bucket for storing logs. You can delete only an empty bucket.

Get the logging settingsGet the logging settings

To get the name of the target bucket and the prefix of the key for the log object, follow these steps:

aws s3api get-bucket-logging \
  --bucket <bucket_name> \
  --output json \
  --endpoint-url https://storage.yandexcloud.net

{
    "LoggingEnabled": {
      "TargetBucket": "<bucket_name>",
      "TargetPrefix": "<key_prefix>"
    }
}

HTTP/1.1 200
<?xml version="1.0" encoding="UTF-8"?>
<BucketLoggingStatus>
   <LoggingEnabled>
      <TargetBucket>target_bucket_name</TargetBucket>
      <TargetPrefix>logs/</TargetPrefix>
   </LoggingEnabled>
</BucketLoggingStatus>

Get the logsGet the logs

To get logs, download the object prefixed with logs/ from the target bucket:

Disable loggingDisable logging

To disable logging, follow these steps:

aws s3api put-bucket-logging \
    --bucket <bucket_name> \
    --endpoint-url https://storage.yandexcloud.net \
    --bucket-logging-status {}

<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01" />