Using the bucket actions logging mechanism
To track operations with the bucket, enable logging.
Detailed information about requests to the source bucket will be saved in an object in the target bucket. The source and target buckets must be different.
Object Storage does not guarantee complete and timely logging.
By default, logging is disabled. After you enable it, Object Storage will save information about actions with the bucket once an hour.
Enable logging
To log requests to the bucket:
-
Use an existing target bucket or create a new one. This is the bucket your logs will be written to.
How to create a bucket
To create a bucket, you need the minimum
storage.editor
role for the folder.Management consoleYandex Cloud CLIAWS CLITerraformAPI- In the management console
, select the folder you want to create a bucket in. - Select Object Storage.
- Click Create bucket.
- On the bucket creation page:
-
Enter a name for the bucket according to the naming requirements.
By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.
-
Limit the maximum bucket size, if required.
If the value is
0
, the maximum size is not limited and is similar to the enabled No limit option. -
Set the public public access parameters to read objects in the bucket, get a list of objects, and read bucket settings:
Restricted
: Authorized Yandex Cloud users onlyPublic
: All users
Warning
Public access is granted to an unlimited number of anonymous users. Use it only when other access grant mechanisms are not available.
-
Select the default storage class:
Standard
.Cold
.Ice
.
"Cold" classes are designed to store objects that you plan to use less frequently for longer periods of time. The "colder" your storage is, the less you pay for storing data; however, the costs of reading and writing data increase.
-
Add labels, if required:
- Click Add label.
- Enter a label in
key: value
format. - Click Enter.
-
Click Create bucket to complete the operation.
-
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the
--folder-name
or--folder-id
parameter.-
View a description of the CLI command to create a bucket:
yc storage bucket create --help
-
Create a bucket in the default folder:
yc storage bucket create --name <bucket_name>
Where
--name
is the name of the bucket. This is a required parameter. For more information, see Bucket naming rules.By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.
Result:
name: example folder_id: b1gmit33ngp6******** anonymous_access_flags: read: false list: false default_storage_class: STANDARD versioning: VERSIONING_DISABLED max_size: "53687091200" acl: {} created_at: "2022-12-16T14:05:12.196007Z"
Optional parameters
-
--default-storage-class
: Storage class. Possible values:standard
: Standard storage. It is installed by default.cold
: Cold storage.ice
: Ice storage.
"Cold" classes are designed to store objects that you plan to use less frequently for longer periods of time. The "colder" your storage is, the less you pay for storing data; however, the costs of reading and writing data increase.
-
--max-size
: Maximum bucket size, in bytes. Default value:0
(unlimited). -
Parameters for enabling public access to a bucket:
--public-read
: Enables public access to reading bucket objects.--public-list
: Enables public access to viewing the list of bucket objects.--public-config-read
: Enables public access to reading bucket settings.
By default, no public access to buckets is allowed.
Warning
Public access is granted to an unlimited number of anonymous users. Use it only when other access grant mechanisms are not available.
-
Parameters to configure the bucket ACL:
-
--acl
: Predefined ACL. For a list of possible values, see Predefined ACLs. You cannot use this parameter together with the--grants
parameter. -
--grants
: Configures permissions for individual users, service accounts, user groups, and public groups (a group of all internet users or a group of all authenticated Yandex Cloud users). You cannot use this parameter together with the--acl
parameter. The parameter value is specified in the following format:grant-type=<permission_grantee_type>,grantee-id=<grantee_ID>,permission=<permission_type>
, where:grant-type
: Permission grantee type. The possible values are:grant-type-account
: User, service account, or user group.grant-type-all-authenticated-users
: Public group that includes all authenticated Yandex Cloud users.grant-type-all-users
: Public group that includes all internet users.
grantee-id
: ID of the user, service account, or user group you need to grant a permission to. Specified only ifgrant-type=grant-type-account
.permission
: ACL permission type. Possible values:permission-full-control
,permission-write
,permission-read
. For more information about permissions, see Permission types.
To configure multiple permissions, specify the
--grants
parameter multiple times.
By default, an empty ACL is created for each new bucket.
-
For more information about the
yc storage bucket create
command, see the YC CLI reference. -
If you do not have the AWS CLI yet, install and configure it.
To create a bucket, assign the
storage.editor
role to the service account used by the AWS CLI.In the terminal, run this command:
aws s3api create-bucket \ --endpoint-url=https://storage.yandexcloud.net \ --bucket <bucket_name>
Where:
--endpoint-url
: Object Storage endpoint--bucket
: Bucket name
Note
By default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage. For more information, see Bucket naming rules.
Result:
{ "Location": "/<bucket_name>" }
The new bucket will have the following parameters:
- No limitations to the maximum size.
- Limited access to read objects, get a list of objects, and read bucket setting.
- Storage class:
Standard
.
Optional parameters
You can apply a predefined ACL to a bucket or configure permissions for individual users, service accounts, user groups and public groups (e.g., a group of all internet users or a group of all authenticated Yandex Cloud users). These settings are not compatible: a bucket should have either a predefined ACL or a set of individual permissions.
Note
Predefined ACL
aws s3api create-bucket \ --endpoint-url=https://storage.yandexcloud.net \ --bucket <bucket_name> \ --acl <predefined_ACL>
Where
--acl
is a predefined ACL. For the list of values, see Predefined ACLs.Individual permissions
aws s3api create-bucket \ --endpoint-url=https://storage.yandexcloud.net \ --bucket <bucket_name> \ <permission_type> <permission_grantee>
Where:
-
Possible types of ACL permissions:
--grant-read
: Permission to access the list of objects in the bucket, read various bucket settings (lifecycle, CORS, or static hosting), and read all objects in the bucket.--grant-write
: Permission to write, overwrite, and delete objects in the bucket. Can only be used together with--grant-read
.--grant-full-control
: Full access to the bucket and the objects in it.
You can set multiple permissions within the same command.
-
The possible permission grantees are:
id=<grantee_ID>
: ID of the user, service account, or user group you need to grant a permission to.uri=http://acs.amazonaws.com/groups/global/AuthenticatedUsers
: Public group that includes all authenticated Yandex Cloud users.uri=http://acs.amazonaws.com/groups/global/AllUsers
: Public group that includes all internet users.
By default, an empty ACL is created for each new bucket.
For more information about the
aws s3api create-bucket
command, see the AWS documentation .Note
Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g.,
storage.admin
, for the folder where you are going to create resources.Terraform
allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License .For more information about the provider resources, see the documentation on the Terraform
website or the mirror .If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
-
In the configuration file, describe the parameters of the resources you want to create:
terraform { required_providers { yandex = { source = "yandex-cloud/yandex" } } required_version = ">= 0.13" } // Configuring a provider provider "yandex" { token = "<IAM_or_OAuth_token>" cloud_id = "<cloud_ID>" folder_id = "<folder_ID>" zone = "ru-central1-a" } // Creating a service account resource "yandex_iam_service_account" "sa" { name = "<service_account_name>" } // Assigning a role to a service account resource "yandex_resourcemanager_folder_iam_member" "sa-admin" { folder_id = "<folder_ID>" role = "storage.admin" member = "serviceAccount:${yandex_iam_service_account.sa.id}" } // Creating a static access key resource "yandex_iam_service_account_static_access_key" "sa-static-key" { service_account_id = yandex_iam_service_account.sa.id description = "static access key for object storage" } // Creating a bucket using a key resource "yandex_storage_bucket" "test" { access_key = yandex_iam_service_account_static_access_key.sa-static-key.access_key secret_key = yandex_iam_service_account_static_access_key.sa-static-key.secret_key bucket = "<bucket_name>" max_size = <maximum_bucket_size> default_storage_class = "<storage_class>" anonymous_access_flags { read = <true|false> list = <true|false> config_read = <true|false> } tags = { <key_1> = "<value_1>" <key_2> = "<value_2>" ... <key_n> = "<value_n>" } }
Where:
-
yandex_iam_service_account
: Description of the service account to create and use the bucket:-
name
: Service account name -
bucket
: Bucket nameBy default, a bucket with a dot in the name is only available over HTTP. To provide HTTPS support for your bucket, upload your own security certificate to Object Storage.
-
max_size
: Maximum bucket size, in bytes. -
default_storage_class
: Storage class. Possible values:standard
: Standard storage.cold
: Cold storage.ice
: Ice storage.
"Cold" classes are designed to store objects that you plan to use less frequently for longer periods of time. The "colder" your storage is, the less you pay for storing data; however, the costs of reading and writing data increase.
-
anonymous_access_flags
: Access settings:read
: Public access to read bucket objects.list
: Public access to view the list of bucket objects.config_read
: Public access to read bucket settings.
-
tags
: Bucket labels inkey = "value"
format.
-
name
: Required parameter. Other parameters are optional. By default, themax-size
value is0
, public access to the bucket is disabled, and the storage class is set tostandard
.For more information about the
yandex_storage_bucket
resource parameters in Terraform, see the relevant provider documentation . -
-
Create resources:
-
In the terminal, change to the folder where you edited the configuration file.
-
Make sure the configuration file is correct using the command:
terraform validate
If the configuration is correct, the following message is returned:
Success! The configuration is valid.
-
Run the command:
terraform plan
The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.
-
Apply the configuration changes:
terraform apply
-
Confirm the changes: type
yes
in the terminal and press Enter.
-
Terraform will create all the required resources. You can check the new resources and their configuration using the management console
.To create a bucket, use the create REST API method for the Bucket resource, the BucketService/Create gRPC API call, or the create S3 API method.
Note
To write logs to the target bucket with an access policy configured, add there a rule to allow any account to perform the
PutObject
action, and specify the log object key prefix, e.g.,logs/
, as a resource. - In the management console
-
Enable logging in the source bucket that you want to track.
Management consoleAWS CLITerraformAPI- In the management console
, select the folder where the source bucket is located. - Select Object Storage.
- Go to the Logging tab.
- Enable Write logs.
- Select Bucket for log storage.
- In the Prefix field, specify the prefix with which the logs will be saved.
- Click Save.
To enable logging via the AWS CLI:
-
Create a file with logging settings in JSON format. For example:
{ "LoggingEnabled": { "TargetBucket": "<target_bucket_name>", "TargetPrefix": "<key_prefix>" } }
Where:
TargetBucket
: Name of the target bucket for the logs.TargetPrefix
: Prefix of the key used for log objects, e.g.,logs/
.
-
Enable logging in the bucket:
aws s3api put-bucket-logging \ --bucket <source_bucket_name> \ --endpoint-url https://storage.yandexcloud.net \ --bucket-logging-status file://<path_to_settings_file>
Where:
--bucket
: Name of the source bucket to enable action logging for.--bucket-logging-status
: Path to the logging settings file.
Note
Terraform uses a service account to interact with Object Storage. Assign to the service account the required role, e.g.,
storage.admin
, for the folder where you are going to create resources.Terraform
allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License .For more information about the provider resources, see the documentation on the Terraform
website or the mirror .If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
To enable logging for a bucket that you wish to monitor:
-
Open the Terraform configuration file and add the
logging
section to the fragment describing the bucket.resource "yandex_storage_bucket" "log_bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<name_of_bucket_to_store_logs>" } resource "yandex_storage_bucket" "bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<source_bucket_name>" acl = "private" logging { target_bucket = yandex_storage_bucket.log_bucket.id target_prefix = "log/" } }
Where:
access_key
: ID of the static access key.secret_key
: Value of the secret access key.target_bucket
: Reference to the bucket that will store logs.target_prefix
: Prefix of the key used for log objects, e.g.,logs/
.
For more information about the
yandex_storage_bucket
resource parameters in Terraform, see the provider documentation .-
In the terminal, change to the folder where you edited the configuration file.
-
Make sure the configuration file is correct using the command:
terraform validate
If the configuration is correct, the following message is returned:
Success! The configuration is valid.
-
Run the command:
terraform plan
The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.
-
Apply the configuration changes:
terraform apply
-
Confirm the changes: type
yes
in the terminal and press Enter.
All the resources you need will then be created in the specified folder. You can check the new resources and their configuration using the management console
.
To enable logging for your bucket, use the putBucketLogging S3 API method.
Example of the HTTP request body:
<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01"> <LoggingEnabled> <TargetBucket>target bucket name</TargetBucket> <TargetPrefix>key prefix</TargetPrefix> </LoggingEnabled> </BucketLoggingStatus>
Where:
TargetBucket
: Target bucket name.TargetPrefix
: Prefix of the key used for log objects, e.g.,logs/
.
- In the management console
If you want to delete the target bucket, first disable logging or select another target bucket for storing logs. You can delete only an empty bucket.
Get the logging settings
To get the name of the target bucket and the prefix of the key for the log object, follow these steps:
- In the management console
, go to the source bucket. - Go to the Logging tab.
- The Bucket for log storage list contains the name of the target bucket.
- The Prefix field contains the prefix with which the logs are saved.
To retrieve the logging settings via the AWS CLI:
Run this command:
aws s3api get-bucket-logging \
--bucket <bucket_name> \
--output json \
--endpoint-url https://storage.yandexcloud.net
Result:
{
"LoggingEnabled": {
"TargetBucket": "<bucket_name>",
"TargetPrefix": "<key_prefix>"
}
}
To get the logging settings in the bucket you want to track:
-
Open the Terraform configuration file and find the
logging
section in the fragment describing the bucket.resource "yandex_storage_bucket" "log_bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<name_of_bucket_to_store_logs>" } resource "yandex_storage_bucket" "bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<source_bucket_name>" acl = "private" logging { target_bucket = yandex_storage_bucket.log_bucket.id target_prefix = "log/" } }
Where:
access_key
: ID of the static access key.secret_key
: Value of the secret access key.target_bucket
: Reference to the bucket that will store logs.target_prefix
: Prefix of the key used for log objects, e.g.,logs/
.
For more information about the
yandex_storage_bucket
resource parameters in Terraform, see the provider documentation .
Use the getBucketLogging S3 API method.
Example of the HTTP response body:
HTTP/1.1 200
<?xml version="1.0" encoding="UTF-8"?>
<BucketLoggingStatus>
<LoggingEnabled>
<TargetBucket>target bucket name</TargetBucket>
<TargetPrefix>logs/</TargetPrefix>
</LoggingEnabled>
</BucketLoggingStatus>
Get the logs
To get logs, download the object prefixed with logs/
from the target bucket:
- In the management console
, select the folder where the target bucket with logs is located. - Select Object Storage.
- Select the target bucket with the logs.
- Go to the
logs/
folder. - Next to the object with the logs you want to download, click
and select Download.
To get logs using the AWS CLI, download the objects with the logs/
prefix by following this guide.
Use the get S3 API method of the Object service.
Disable logging
To disable logging, follow these steps:
- In the management console
, select the folder where the source bucket is located. - Select Object Storage.
- Go to the Logging tab.
- Enable Write logs.
- Click Save.
To disable logging using the AWS CLI, use the following command:
aws s3api put-bucket-logging \
--bucket <bucket_name> \
--endpoint-url https://storage.yandexcloud.net \
--bucket-logging-status {}
Where --bucket
is the name of the source bucket to disable action logging for.
To disable the logging mechanism:
-
In the Terraform configuration file, delete the
logging
section in the fragment describing the bucket.Example bucket description in a Terraform configuration
... resource "yandex_storage_bucket" "log_bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<name_of_bucket_to_store_logs>" } resource "yandex_storage_bucket" "bucket" { access_key = "<static_key_ID>" secret_key = "<private_key>" bucket = "<source_bucket_name>" acl = "private" logging { target_bucket = yandex_storage_bucket.log_bucket.id target_prefix = "log/" } } ...
-
Apply the configuration changes:
-
In the terminal, change to the folder where you edited the configuration file.
-
Make sure the configuration file is correct using the command:
terraform validate
If the configuration is correct, the following message is returned:
Success! The configuration is valid.
-
Run the command:
terraform plan
The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.
-
Apply the configuration changes:
terraform apply
-
Confirm the changes: type
yes
in the terminal and press Enter.
-
You can check the changes in the management console
Use the putBucketLogging S3 API method. In the request body, send the <BucketLoggingStatus>
parameter with an empty value.
Example of the HTTP request body:
<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01" />