Bucket actions logging mechanism

Object Storage features a bucket actions logging mechanism. For example, you can use logging for an internal security audit or to get more granular information about bucket-related operations.

By default, logging is disabled. Once you enable this option, Object Storage will start writing data on bucket actions in a form of an object once an hour.

To start writing logs, do the following:

• Define the source bucket you want to log the actions with.
• Create a target bucket where you want to save the logs.
• Enable logging using the management console, AWS CLI, Terraform, or API.
• Select the prefix of the object key (optional).

PrerequisitesPrerequisites

• The source and target buckets must be different.

• The source and target buckets must be in the same cloud.

• Encryption must be disabled on the target bucket.

Format of the key for the log objectFormat of the key for the log object

Object Storage uses the following format of the key for the log object:

<prefix>/YYYY-MM-DD-HH-MM-SS-<ID>

Where:

• <prefix>: Prefix of the key for the log object. You can specify the prefix yourself when enabling logging.
• YYYY-MM-DD-HH-MM-SS: Date and time of saving the log object in the target bucket (UTC format).
• <ID>: Unique record ID that prevents the object from being overwritten.

Prefix of the keyPrefix of the key

The key prefix allows you to distinguish:

• Data from different buckets, if the logs for multiple source buckets are saved to the same target bucket.
• Logging actions from other actions with the bucket, if the logs are saved to the source bucket. This is because, in this case, the logging operation is also considered an action with the bucket.
• Log objects from other objects, in order to regularly delete logs. You can set up a lifecycle for the target bucket to automatically delete objects with a specific key prefix.

Log object formatLog object format

Logs are saved to a JSON file. Every action with a bucket is logged to the file by adding the respective record to it.

A full list of logged parameters is provided in the log reference.

Example of a record in the log file:

{
  "bucket": "my-bucket-example",
  "bytes_received": 749,
  "bytes_send": 1251,
  "handler": "REST.GET.OBJECT",
  "http_referer": "https://example.com/page",
  "ip": "84.201.121.46",
  "method": "GET",
  "object_key": "path/logs/2020-11-10-14-42-11-123f57b5-1853-4120-8d7a-5bcc1e9e9b4f",
  "protocol": "HTTP/1.1",
  "range": "-",
  "requester": "-",
  "request_args": "X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Date=20201030T072100Z\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Expires=43200\u0026X-Amz-Credential=ZGB4EY1...\u0026X-Amz-Signature=12f350...",
  "request_id": "1235efda********",
  "request_path": "/my-bucket-example/path/logs/2020-11-10-14-42-11-123f57b5-1853-4120-8d7a-5bcc1e9e9b4f?X-Amz-...",
  "request_time":88,
  "scheme": "https",
  "ssl_protocol": "TLSv1.2",
  "status": 200,
  "storage_class": "STANDARD",
  "timestamp": "2020-11-10T13:21:18Z",
  "user_agent": "docker/19.03.9 go/go1.13.10 git-commit/1d238398e7 kernel/4.4.0-142-generic os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)",
  "version_id": "",
  "vhost": "storage.yandexcloud.net"
}

Logging specificsLogging specifics

There are several points to note about how bucket actions are logged in Object Storage.

Best-effort log deliveryBest-effort log delivery

Most requests to a bucket are written to the log file, if the bucket for logging was set up correctly. Most records are written within a few hours after the request is actually processed.

However, Object Storage does not guarantee that the logs are saved in a complete and timely manner. It may take up to a few hours to record an action with the bucket in a log file. In some cases, a record might fail to appear in the file.

The log file provides an overview of the nature of traffic in the bucket, but is not intended for logging each and every request. In the payment documents, you might find a few requests that do not show up in the log file.

No object lifecycle action loggingNo object lifecycle action logging

In buckets with lifecycle rules configured, actions performed with objects as part of their lifecycle are not logged.

Enabling logging takes timeEnabling logging takes time

It takes around an hour to enable logging and change the settings. The first log object is saved to the bucket in two hours after logging is enabled; sometimes, however, it may take longer.

When you change the target bucket, some logs will still be delivered to the previous target bucket, while others will be delivered to the new one.

All changes to the settings will take effect without additional user actions.

PricingPricing

The logging pricing is based on the regular Object Storage pricing rules.