Creating a trail to upload audit logs

1. Create a log group the audit logs will be uploaded to.

2. Create a service account for the trail.

3. Assign to the service account these roles for the trail to collect and upload logs:

   • logging.writer for the log group.

   • audit-trails.viewer for a resource defining the required log collection scope:

     • Organization: To collect logs in selected clouds of the organization.
     • Cloud: To collect logs in selected folders of the cloud.
     • Folder: To collect logs in this folder.

     Child resources inherit access permissions from their parent resources. For example, if a service account gets a role for a cloud, then the trail that uses this account will be able to collect logs for resources across all folders of this cloud. However, the trail will not be able to collect logs in other clouds owned by the organization: a role for the organization is required for that.

4. Make sure the account you are going to use to create a trail has the required roles:

   • audit-trails.editor for the folder to host the trail.
   • iam.serviceAccounts.user for the trail’s service account.

1. Create a data stream the audit logs will be uploaded to.

   Tip

   We recommend enabling autopartitioning on the target data stream.

   When overloading individual shards or the entire stream, some events may get lost. Autopartitioning automatically adds shards as needed and distributes the workloads to avoid losses. If autopartitioning is disabled, remember to manually check and increase the number of shards.

2. Create a service account for the trail.

3. Assign to the service account these roles for the trail to collect and upload logs:

   • yds.writer for the data stream.

   • audit-trails.viewer for a resource defining the required log collection scope:

     • Organization: To collect logs in selected clouds of the organization.
     • Cloud: To collect logs in selected folders of the cloud.
     • Folder: To collect logs in this folder.

     Child resources inherit access permissions from their parent resources. For example, if a service account gets a role for a cloud, then the trail that uses this account will be able to collect logs for resources across all folders of this cloud. However, the trail will not be able to collect logs in other clouds owned by the organization: a role for the organization is required for that.

4. Make sure the account you are going to use to create a trail has the required roles:

   • audit-trails.editor for the folder to host the trail.
   • iam.serviceAccounts.user for the trail’s service account.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

View the description of the CLI command for creating a trail:

yc audit-trails trail create --help

Use a YAML specification for collecting data events. If you only need management events, provide parameters directly in the command.

yc audit-trails trail create \
  --name <trail_name> \
  --service-account-id <service_account_ID> \
  --destination-bucket <bucket_name> \
  --filter-all-folder-id <folder_ID>

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the guides on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To manage infrastructure using Terraform under a service account or user accounts (a Yandex account, a federated account, or a local user), authenticate using the appropriate method.

1. Specify the trail properties in the configuration file. In this template, logs are uploaded to the bucket:

   resource "yandex_audit_trails_trail" "basic_trail" {
     name        = "<trail_name>"
     folder_id   = "<folder_ID>"
     description = "<trail_description>"
     labels = {
       key = "value"
     }
     service_account_id = "<service_account_ID>"
   
     storage_destination {
       bucket_name   = "<bucket_name>"
       object_prefix = "<prefix>"
     }
   
     filtering_policy {
       management_events_filter {
         resource_scope {
           resource_id   = "<cloud_or_folder_organization_ID>"
           resource_type = "<type>"
         }
       }
   
       data_events_filter {
         service         = "<service_name>"
         included_events = ["<full_event_name>"]
   
         resource_scope {
           resource_id   = "<cloud_or_folder_organization_ID>"
           resource_type = "<type>"
         }
       }
     }
   }
   

   Where:

   • name: Name of the new trail. The naming requirements are as follows:

     • Length: between 3 and 63 characters.
     • It can only contain lowercase Latin letters, numbers, and hyphens.
     • It must start with a letter and cannot end with a hyphen.

   • folder_id: ID of the folder to create the trail in.

   • description: Trail description. This is an optional parameter.

   • labels: List of labels in key=value format. This is an optional parameter.

   • service_account_id: ID of the service account the trail uses to collect and upload audit logs.

     Based on the audit log collection scope, the service account must have the audit-trails.viewer role for the organization, cloud, or folder whose audit logs the trail will be collecting.

   Specify only one destination: storage_destination, logging_destination, or data_stream_destination.

   • logging_destination: Uploading logs to a Yandex Cloud Logging group.

     Replace the storage_destination section in the template:

     logging_destination {
       log_group_id = "<log_group_ID>"
     }
     
     • log_group_id: ID of the log group for the trail to save audit logs to.

   • storage_destination: Uploading logs to a Yandex Object Storage bucket:

     • log_group_id: Name of the bucket for the trail to save audit logs to.

     • object_prefix: Prefix that will be assigned to the objects with audit logs in the bucket. It is an optional parameter used in the full name of the audit log file.

       Note

       Use a prefix to store audit logs and third-party data in the same bucket. Do not use the same prefix for logs and other bucket objects because that may cause logs and third-party objects to overwrite each other.

   • data_stream_destination: Uploading logs to a data stream in Yandex Data Streams:

     Replace the storage_destination section in the template:

     data_stream_destination {
       database_id = "<database_ID>"
       stream_name = "<data_stream_name>"
       codec       = "RAW"
     }
     
     • stream_name: Name of the data stream for the trail to save audit logs to.
     • database_id: ID of the Yandex Managed Service for YDB database used by Data Streams.
     • codec: Event compression method when writing to Data Streams. The possible values are RAW (no compression, default), GZIP, and ZSTD. Enable compression if you expect an event flow greater than 1 MB/s.

   • filtering_policy: Settings of the filtering policy that determines which events to collect and include in the audit logs. The policy consists of filters pertaining to different levels of events. It contains the management_events_filter and data_events_filter sections.

     • management_events_filter: Management event filter.

       • resource_scope: Log collection scope. You can specify multiple resource_scope sections for scopes belonging to the same organization. For example, you can collect logs from one entire cloud and only from particular folders in another cloud. Service account permissions must allow collecting logs from the specified scopes.

         • resource_id: ID of the collection scope (organization, cloud, or folder).

         • resource_type: Scope type according to the specified ID:

           • organization-manager.organization: Organization.
           • resource-manager.cloud: Cloud.
           • resource-manager.folder: Folder.

     • data_events_filter: Data event filter. You can configure several filters of this type, one filter per service. A filter for one service has the following structure:

       • service: Name of the service in which the trail will process events. You can get it from the data event reference.

       • resource_scope: Places to collect data events from. You can configure this section the same way as the management event filter.

       • included_events: Full names of events to collect. You can replace it with excluded_events (full names of events to exclude). These parameters are mutually exclusive. If both are not set, selection is not limited to event names.

         Use the full name from the event_type field of the audit log. For example, yandex.cloud.audit.iam.CreateIamToken corresponds to the CreateIamToken event from the reference.

       • include_rule and exclude_rule: Optional rules with conditions based on field values. These apply together with event selection by names.

   For more on the properties of the yandex_audit_trails_trail resource, see this provider guide.

2. Optionally, supplement the filter with conditions based on field values for the relevant service. To do this, under filtering_policy.data_events_filter, set the include_rule and exclude_rule. Use the Yandex Cloud provider version 0.225.0 or higher. For filter configuration examples, see Creating a trail.

3. Create the resources:

   1. In the terminal, navigate to the configuration file directory.

   2. Make sure the configuration is correct using this command:

      terraform validate
      

      If the configuration is valid, you will get this message:

      Success! The configuration is valid.
      

   3. Run this command:

      terraform plan
      

      You will see a list of resources and their properties. No changes will be made at this step. Terraform will show any errors in the configuration.

   4. Apply the configuration changes:

      terraform apply
      

   5. Type yes and press Enter to confirm the changes.

   Terraform will create all the required resources. You can check the new resources and their settings either in the management console or using this CLI command:

   yc audit-trails trail get <trail_name>
   

1. Authenticate with the API.

2. Prepare the following trail parameters: folder, destination object, service account, and filtering policy with the relevant services and scopes.

   To use the settings of an existing trail as a basis, get them using the get REST API method or TrailService/Get gRPC API call.

3. Optionally, supplement the filter with conditions based on field values for the relevant service. Do it by setting the includeRules and excludeRules rules in the filteringPolicy.dataEventsFilters list item in the REST API request. In the gRPC API, use include_rules and exclude_rules in filtering_policy.data_events_filters. For a filter configuration example, see Creating a trail.

4. To create a trail, use the create REST API method for the Trail resource or the TrailService/Create gRPC API call.

1. Specify the trail properties in the Terraform configuration file:

   resource "yandex_audit_trails_trail" "basic_trail" {
     name               = "sample-trail-all-planes"
     folder_id          = "folder0***"
     service_account_id = "service0***"
   
     storage_destination {
       bucket_name = "sample-logs-bucket"
     }
   
     filtering_policy {
       management_events_filter {
         resource_scope {
           resource_id   = "org1***"
           resource_type = "organization-manager.organization"
         }
       }
   
       data_events_filter {
         service = "mdb.postgresql"
         excluded_events = [
           "yandex.cloud.audit.mdb.postgresql.CreateDatabase",
           "yandex.cloud.audit.mdb.postgresql.UpdateDatabase",
         ]
   
         resource_scope {
           resource_id   = "cloud1***"
           resource_type = "resource-manager.cloud"
         }
   
         resource_scope {
           resource_id   = "folder1***"
           resource_type = "resource-manager.folder"
         }
       }
   
       data_events_filter {
         service         = "iam"
         included_events = ["yandex.cloud.audit.iam.CreateIamToken"]
   
         resource_scope {
           resource_id   = "folder2***"
           resource_type = "resource-manager.folder"
         }
   
         exclude_rule {
           condition {
             field    = "$.details.credential.type"
             operator = "IN"
             values   = ["iam.session"]
           }
   
           condition {
             field    = "$.details.subject_name"
             operator = "IN"
             values   = ["user@example.com"]
           }
         }
       }
     }
   }
   

2. Apply the configuration:

   terraform apply
   

Use the create REST API method for the Trail resource.

To use the examples, install cURL.

The example below is for MacOS and Linux. To run it on Windows, check the details on working with Bash in Microsoft Windows.

1. Get an IAM token for API authentication and write it into an environment variable:

   export IAM_TOKEN=<IAM_token>
   

2. Create a file named body.json with the following request body:

   {
     "folderId": "folder0***",
     "name": "sample-trail-all-planes",
     "destination": {
       "objectStorage": {
         "bucketId": "sample-logs-bucket"
       }
     },
     "serviceAccountId": "service0***",
     "filteringPolicy": {
       "managementEventsFilter": {
         "resourceScopes": [
           {
             "id": "org1***",
             "type": "organization-manager.organization"
           }
         ]
       },
       "dataEventsFilters": [
         {
           "service": "mdb.postgresql",
           "resourceScopes": [
             {
               "id": "cloud1***",
               "type": "resource-manager.cloud"
             },
             {
               "id": "folder1***",
               "type": "resource-manager.folder"
             }
           ],
           "excludedEvents": {
             "eventTypes": [
               "yandex.cloud.audit.mdb.postgresql.CreateDatabase",
               "yandex.cloud.audit.mdb.postgresql.UpdateDatabase"
             ]
           }
         },
         {
           "service": "iam",
           "resourceScopes": [
             {
               "id": "folder2***",
               "type": "resource-manager.folder"
             }
           ],
           "includedEvents": {
             "eventTypes": [
               "yandex.cloud.audit.iam.CreateIamToken"
             ]
           },
           "excludeRules": [
             {
               "conditions": [
                 {
                   "field": "$.details.credential.type",
                   "operator": "IN",
                   "values": ["iam.session"]
                 },
                 {
                   "field": "$.details.subject_name",
                   "operator": "IN",
                   "values": ["user@example.com"]
                 }
               ]
             }
           ]
         }
       ]
     }
   }
   

3. Run this request:

   curl \
     --request POST \
     --header "Authorization: Bearer ${IAM_TOKEN}" \
     --header "Content-Type: application/json" \
     --data "@body.json" \
     https://audittrails.api.cloud.yandex.net/audit-trails/v1/trails