Uploading cloud audit logs to Cloud Logging
Follow this guide to create a new trail that will upload management event audit logs of all resources and, optionally, data event audit logs of selected services of an individual cloud to a Cloud Logging log group.
Prepare the environment
To export audit logs of a cloud:
-
Create a log group to upload audit logs to.
-
Create a service account.
-
Assign roles to the service account:
CLIIf you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the
--folder-name
or--folder-id
parameter.-
audit-trails.viewer
for the cloud to collect audit logs from.yc resource-manager cloud add-access-binding \ --role audit-trails.viewer \ --id <cloud_ID> \ --service-account-id <service_account_ID>
Where:
--role
: Role being assigned.--id
: ID of the cloud to collect audit logs from.--service-account-id
: Service account ID.
-
logging.writer
for the folder to host the trail:yc resource-manager folder add-access-binding \ --role logging.writer \ --id <folder_ID> \ --service-account-id <service_account_ID>
Where:
--role
: Role you want to assign.--id
: ID of the folder to host the trail.--service-account-id
: Service account ID.
-
-
On the Access permissions
page, make sure you have the following roles:iam.serviceAccounts.user
for the service account.audit-trails.editor
for the folder to host the trail.audit-trails.viewer
for the cloud to collect audit logs from.logging.viewer
for the Cloud Logging log group.
Creating a trail
To create a trail that exports audit logs from the cloud:
-
In the management console
, select the folder to host the trail. -
Select Audit Trails.
-
Click Create trail and specify:
- Name: Name of the new trail.
- Description: Trail description (optional).
-
Under Destination, configure the destination object:
-
Destination:
Cloud Logging
. -
Log group: Select a log group. You can also create a new log group. For this:
-
Click Create new, then specify the parameters of the log group:
- Name: Name of the new group.
- Description: Log group description (optional).
- Log retention.
-
Click Create group.
-
-
-
Under Service account, select the service account that the trail will use to upload audit log files to the log group.
-
Under Collecting management events, configure the collection of management event audit logs:
- Collecting events: Select
Enabled
. - Resource: Select
Cloud
. - Cloud: Automatically populated field showing the name of the cloud to host the trail.
- Folder: Keep the default value,
All
.
- Collecting events: Select
-
(Optional) Under Collecting data events, configure the collection of data event audit logs:
-
Click Create.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
-
View the description of the CLI command for creating a trail:
yc audit-trails trail create --help
-
Run the command to create a trail to collect management event audit logs in the cloud:
yc audit-trails trail create \ --name <trail_name> \ --description "<trail_description>" \ --service-account-id <service_account_ID> \ --destination-log-group-id <log_group_ID> \ --filter-from-cloud-id <cloud_ID> \ --filter-some-folder-ids <cloud_folder_list>
Where:
-
--name
: Name of the trail being created.
The name format requirements are as follows:- The name must be from 3 to 63 characters long.
- It may contain lowercase Latin letters, numbers, and hyphens.
- The first character must be a letter and the last character cannot be a hyphen.
-
--description
: Description of the trail to distinguish it from other trails, e.g.,--description "My very first trail"
. This is an optional parameter. -
--service-account-id
: ID of the service account that the trail will use to upload audit log files to the bucket.Based on the audit log collection scope, the service account must have the
audit-trails.viewer
role for the organization, cloud, or folder whose audit logs the trail will collect.
--destination-log-group-id
: ID of the log group for the trail to save audit logs to.
-
--filter-from-cloud-id
: ID of the cloud the trail being created belongs to and for whose resources audit logs will be collected.When using the
--filter-from-cloud-id
parameter, you also need to set folder IDs in the--filter-some-folder-ids
parameter.You cannot use the
--filter-from-cloud-id
parameter together with the--filter-all-cloud-id
parameter. -
--filter-some-folder-ids
: List of IDs of the folders for whose resources the trail will collect audit logs. Use this parameter only if the---filter-from-cloud-id
parameter is set.The folders specified in the parameter must belong to the cloud set in the
--filter-from-cloud-id
parameter.If you need to collect audit logs from all folders belonging to the cloud, use the
--filter-all-cloud-id
parameter. -
--filter-all-cloud-id
: ID of the cloud the trail being created belongs to and for whose resources audit logs will be collected from all folders.You cannot use the
--filter-all-cloud-id
parameter together with the--filter-from-cloud-id
parameter.
Result:
done (1s) id: cnp0sprpubl0******** folder_id: b1g9d2k0itu4******** created_at: "2024-04-01T09:56:07.858Z" updated_at: "2024-04-01T09:56:07.858Z" name: sample-trail description: My very first trail destination: cloud_logging: log_group_id: e23fvhkglsac******** service_account_id: ajeee339l4m5******** status: ACTIVE filter: path_filter: root: some_filter: resource: id: b1glti4eser3******** type: resource-manager.cloud filters: - any_filter: resource: id: b1g9d2k0itu4******** type: resource-manager.folder - any_filter: resource: id: b1gp73knqj0u******** type: resource-manager.folder event_filter: {} cloud_id: b1glti4eser3********
-
-
To create a trail for collecting management event and data event audit logs, use the trail YAML specification file.
-
Create a trail configuration file, e.g.,
specification.yaml
:specification.yaml
folder_id: b1g9d2k0itu4******** name: sample-trail description: My very first trail labels: key: value destination: cloud_logging: log_group_id: e232pqboot3f******** service_account_id: ajeee339l4m5******** filter: path_filter: root: some_filter: resource: id: b1glti4eser3******** type: resource-manager.cloud filters: - any_filter: resource: id: b1g9d2k0itu4******** type: resource-manager.folder - any_filter: resource: id: b1gp73knqj0u******** type: resource-manager.folder event_filter: filters: - service: mdb.postgresql categories: - plane: DATA_PLANE type: WRITE path_filter: root: some_filter: resource: id: b1glti4eser3******** type: resource-manager.cloud filters: - any_filter: resource: id: b1g9d2k0itu4******** type: resource-manager.folder - any_filter: resource: id: b1gp73knqj0u******** type: resource-manager.folder - service: dns categories: - plane: DATA_PLANE type: READ path_filter: root: any_filter: resource: id: b1glti4eser3******** type: resource-manager.cloud
Specification description
-
folder_id
: ID of the folder to create a trail in. -
name
: Name of the trail being created. The name format requirements are as follows:- The name must be from 3 to 63 characters long.
- It may contain lowercase Latin letters, numbers, and hyphens.
- The first character must be a letter and the last character cannot be a hyphen.
-
description
: Description of the trail to distinguish it from other trails, e.g.,My very first trail
. This is an optional parameter. -
labels
: List of labels inkey=value
format. This is an optional parameter. -
service_account_id
: ID of the service account that the trail will use to upload audit log files to the bucket.Based on the audit log collection scope, the service account must have the audit-trails.viewer
role
for the organization, cloud, or folder whose audit logs the trail will collect.
destination.cloud_logging.log_group_id
: ID of the log group for the trail to save audit logs to.
filter
: Structure describing how the trail processes events. It contains thepath_filter
andevent_filter
objects.-
path_filter
: Structure describing how the trail processes management (control plane) events. It may contain a singleroot.some_filter
object or a singleroot.any_filter
object. If thepath_filter
object is missing in the configuration, the trail will not process control plane events.-
root.some_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.-
resource.id
: ID of the parent resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command and the cloud ID, using theyc resource-manager clouds list
command. -
resource.type
: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specifyorganization-manager.organization
orresource-manager.cloud
in this parameter. -
filters.any_filter
: Structure describing a single child resource whose audit logs will be collected. To specify multiple child resources, set thefilters.any_filter
parameter as many times as you need. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:resource.id
: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.resource.type
: Child resource type. Based on the audit log collection scope, specifyresource-manager.cloud
orresource-manager.folder
in this parameter.
To set the current folder as the audit log collection scope, use the
root.any_filter
parameter instead ofroot.some_filter
.You cannot use the
root.some_filter
parameter together with theroot.any_filter
parameter. -
-
root.any_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use theroot.any_filter
parameter together with theroot.some_filter
parameter.-
resource.id
: ID of the resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command. -
resource.type
: Yandex Cloud resource type. Based on the audit log collection scope, specifyorganization-manager.organization
,resource-manager.cloud
, orresource-manager.folder
.
-
-
-
event_filter
: Structure describing how the trail processes data (data plane) events. If theevent_filter
object is missing in the configuration, the trail will not process data plane events.-
filters.service
: ID of the service whose events the trail will process. A singlefilters.service
section is used to configure data plane audit log processing for a single Yandex Cloud service. To configure the trail processing of data plane events for multiple services, specify thefilters.service
parameter in the configuration as many times as you need.
The possible values include:dns
kms
lockbox
mdb.mongodb
mdb.mysql
mdb.postgresql
storage
-
filters.categories
: Structure describing the type of events to collect.plane
: Event plane. For data plane events, specifyDATA_PLANE
.type
: Type of resource event action. The possible values areREAD
andWRITE
.
-
filters.path_filter
: Structure describing how the trail processes data (data plane) events. It may contain a singleroot.some_filter
object or a singleroot.any_filter
object.-
root.some_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.-
resource.id
: ID of the parent resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command and the cloud ID, using theyc resource-manager clouds list
command. -
resource.type
: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specifyorganization-manager.organization
orresource-manager.cloud
in this parameter. -
filters.any_filter
: Structure describing a single child resource whose audit logs will be collected. To specify multiple child resources, set thefilters.any_filter
parameter as many times as you need. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:resource.id
: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.resource.type
: Child resource type. Based on the audit log collection scope, specifyresource-manager.cloud
orresource-manager.folder
in this parameter.
To set the current folder as the audit log collection scope, use the
root.any_filter
parameter instead ofroot.some_filter
.You cannot use the
root.some_filter
parameter together with theroot.any_filter
parameter. -
-
root.any_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use theroot.any_filter
parameter together with theroot.some_filter
parameter.-
resource.id
: ID of the resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command. -
resource.type
: Yandex Cloud resource type. Based on the audit log collection scope, specifyorganization-manager.organization
,resource-manager.cloud
, orresource-manager.folder
.
-
-
-
-
-
-
Run the command, specifying the path to the created trail specification file:
yc audit-trails trail create --file <specification_file_path>
Result
done (1s) id: cnp57t5l3hd5******** folder_id: b1g9d2k0itu4******** created_at: "2024-04-02T03:54:37.362Z" updated_at: "2024-04-02T03:54:37.362Z" name: sample-trail description: My very first trail destination: cloud_logging: log_group_id: e232pqboot3f******** service_account_id: ajeee339l4m5******** status: ACTIVE filter: path_filter: root: some_filter: resource: id: b1glti4eser3******** type: resource-manager.cloud filters: - any_filter: resource: id: b1g9d2k0itu4******** type: resource-manager.folder - any_filter: resource: id: b1gp73knqj0u******** type: resource-manager.folder event_filter: filters: - service: mdb.postgresql categories: - plane: DATA_PLANE type: WRITE path_filter: root: some_filter: resource: id: b1glti4eser3******** type: resource-manager.cloud filters: - any_filter: resource: id: b1g9d2k0itu4******** type: resource-manager.folder - any_filter: resource: id: b1gp73knqj0u******** type: resource-manager.folder - service: dns categories: - plane: DATA_PLANE type: READ path_filter: root: any_filter: resource: id: b1glti4eser3******** type: resource-manager.cloud cloud_id: b1glti4eser3********
For more information about the
yc audit-trails trail create
command, see the CLI reference. -
Terraform
For more information about the provider resources, see the documentation on the Terraform
If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
-
In the configuration file, describe the parameters of the trail that will collect audit events of the cloud resources:
resource "yandex_audit_trails_trail" "basic_trail" { name = "<trail_name>" folder_id = "<folder_ID>" description = "<trail_description>" labels = { key = "value" } service_account_id = "<service_account_ID>" logging_destination { log_group_id = "<log_group_ID>" } filter { path_filter { some_filter { resource_id = "<organization_ID>" resource_type = "<parent_resource_type>" any_filters { resource_id = "<ID_of_cloud_1>" resource_type = "<child_resource_type>" } any_filters { resource_id = "<ID_of_cloud_2>" resource_type = "<child_resource_type>" } } } event_filters { service = "<ID_of_service_1>" categories { plane = "DATA_PLANE" type = "<action_type>" } path_filter { any_filter { resource_id = "<organization_ID>" resource_type = "<resource_type>" } } } event_filters { service = "<ID_of_service_2>" categories { plane = "DATA_PLANE" type = "<action_type>" } path_filter { any_filter { resource_id = "<organization_ID>" resource_type = "<resource_type>" } } } } }
Where:
-
name
: Name of the trail being created. The name format requirements are as follows:- The name must be from 3 to 63 characters long.
- It may contain lowercase Latin letters, numbers, and hyphens.
- The first character must be a letter and the last character cannot be a hyphen.
-
folder_id
: ID of the folder to create a trail in. -
description
: Description of the trail to distinguish it from other trails, e.g.,My very first trail
. This is an optional parameter. -
labels
: List of labels inkey=value
format. This is an optional parameter. -
service_account_id
: ID of the service account that the trail will use to upload audit log files to the bucket.Based on the audit log collection scope, the service account must have the audit-trails.viewer
role
for the organization, cloud, or folder whose audit logs the trail will collect.
log_group_id
: ID of the log group for the trail to save audit logs to.
filter
: Structure describing how the trail processes events. It contains thepath_filter
andevent_filters
objects.-
path_filter
: Structure describing how the trail processes management (control plane) events. It may contain a singlesome_filter
object or a singleany_filter
object. If thepath_filter
object is missing in the configuration, the trail will not process control plane events.-
some_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.-
resource_id
: ID of the parent resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command and the cloud ID, using theyc resource-manager clouds list
command. -
resource_type
: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specifyorganization-manager.organization
orresource-manager.cloud
in this parameter. -
any_filters
: Structure describing a single child resource whose audit logs will be collected. To specify multiple child resources, set theany_filters
parameter as many times as you need. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:resource_id
: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.resource_type
: Child resource type. Based on the audit log collection scope, specifyresource-manager.cloud
orresource-manager.folder
in this parameter.
To set the current folder as the audit log collection scope, use the
any_filter
parameter instead ofsome_filter
.You cannot use the
some_filter
parameter together with theany_filter
parameter. -
-
any_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use theany_filter
parameter together with thesome_filter
parameter.-
resource_id
: ID of the resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command. -
resource_type
: Yandex Cloud resource type. Based on the audit log collection scope, specifyorganization-manager.organization
,resource-manager.cloud
, orresource-manager.folder
.
-
-
-
event_filters
: Structure describing how the trail processes data (data plane) evens. If theevent_filters
object is missing in the configuration, the trail will not process data plane events. A singleevent_filters
section is used to configure data plane audit log processing for a single Yandex Cloud service. To configure the trail processing of data plane events for multiple services, specify theevent_filters
parameter in the configuration as many times as you need.-
service
: ID of the service whose events the trail will process.
The possible values include:dns
kms
lockbox
mdb.mongodb
mdb.mysql
mdb.postgresql
storage
-
categories
: Structure describing the type of events to collect.plane
: Event plane. For data plane events, specifyDATA_PLANE
.type
: Type of resource event action. The possible values areREAD
andWRITE
.
-
path_filter
: Structure describing how the trail processes data (data plane) events. It may contain a singlesome_filter
object or a singleany_filter
object.-
some_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.-
resource_id
: ID of the parent resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command and the cloud ID, using theyc resource-manager clouds list
command. -
resource_type
: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specifyorganization-manager.organization
orresource-manager.cloud
in this parameter. -
any_filters
: Structure describing a single child resource whose audit logs will be collected. To specify multiple child resources, set theany_filters
parameter as many times as you need. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:resource_id
: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.resource_type
: Child resource type. Based on the audit log collection scope, specifyresource-manager.cloud
orresource-manager.folder
in this parameter.
To set the current folder as the audit log collection scope, use the
any_filter
parameter instead ofsome_filter
.You cannot use the
some_filter
parameter together with theany_filter
parameter. -
-
any_filter
: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use theany_filter
parameter together with thesome_filter
parameter.-
resource_id
: ID of the resource the trail being created belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.You can get the organization ID using the
yc organization-manager organization list
CLI command. -
resource_type
: Yandex Cloud resource type. Based on the audit log collection scope, specifyorganization-manager.organization
,resource-manager.cloud
, orresource-manager.folder
.
-
-
-
-
For more information about the
yandex_audit_trails_trail
resource parameters in Terraform, see the provider documentation . -
-
Create resources:
-
In the terminal, change to the folder where you edited the configuration file.
-
Make sure the configuration file is correct using the command:
terraform validate
If the configuration is correct, the following message is returned:
Success! The configuration is valid.
-
Run the command:
terraform plan
The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.
-
Apply the configuration changes:
terraform apply
-
Confirm the changes: type
yes
in the terminal and press Enter.
Terraform will create all the required resources. You can check the new resources and their configuration using the management console
or this CLI command:yc audit-trails trail get <trail_name>
-
Use the create REST API method for the Trail resource or the TrailService/Create gRPC API call.
The trail will be created and will start uploading audit logs to the log group.
When uploading to Cloud Logging, you may get duplicate events in a log group. To find out whether an event in a log group is a duplicate, use the unique ID of the json_payload.event_id
record.
What's next
- Learn more about the audit log format.
- Find out about the procedure for uploading audit logs to SIEM.
- Learn more about searching audit logs in buckets.