Creating a trail to upload audit logs

1. Create a log group the audit logs will be uploaded to.

2. Create a service account for the trail.

3. Assign roles to the service account for the trail to be able to collect and upload logs:

   • logging.writer for a log group.

   • audit-trails.viewer for one of the resources that defines the required log collection scope:

     • Organization: To collect logs in selected clouds of the organization.
     • Cloud: To collect logs in selected folders of the cloud.
     • Folder: To collect logs in this folder.

     Child resources inherit access permissions from their parent resources. For example, if a service account gets a role for a cloud, then the trail that uses this account will be able to collect logs for resources across all folders of this cloud. However, the trail will not be able to collect logs in other clouds owned by the organization: a role for the organization is required for that.

4. Make sure your account has the roles required for creating a trail:

   • audit-trails.editor for the folder to host the trail.
   • iam.serviceAccounts.user for the trail’s service account.

1. Create a data stream the audit logs will be uploaded to.

2. Create a service account for the trail.

3. Assign roles to the service account for the trail to be able to collect and upload logs:

   • yds.writer for a data stream.

   • audit-trails.viewer for one of the resources that defines the required log collection scope:

     • Organization: To collect logs in selected clouds of the organization.
     • Cloud: To collect logs in selected folders of the cloud.
     • Folder: To collect logs in this folder.

     Child resources inherit access permissions from their parent resources. For example, if a service account gets a role for a cloud, then the trail that uses this account will be able to collect logs for resources across all folders of this cloud. However, the trail will not be able to collect logs in other clouds owned by the organization: a role for the organization is required for that.

4. Make sure your account has the roles required for creating a trail:

   • audit-trails.editor for the folder to host the trail.
   • iam.serviceAccounts.user for the trail’s service account.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder through the --folder-name or --folder-id parameter.

You can create a trail in two ways:

• Create a YAML file containing your trail parameters and provide this file to the command to create the trail.

  This method simplifies working with trail parameters and reduces the probability of error.

• Provide the trail parameters in the command arguments to create the trail.

  See the description of the CLI trail create command for details about the arguments you can use:

  yc audit-trails trail create --help
  

  Tip

  Use this method if your trail configuration is simple and contains few parameters.

To create a trail using a YAML file:

1. Create a YAML file with the trail configuration:

   name: <trail_name>
   folder_id: <folder_ID>
   destination:
     # Only one destination must be specified:
     # object_storage, cloud_logging, data_stream
     # Settings for all destinations are provided for illustration purposes
     object_storage:
       bucket_id: <bucket_name>
       object_prefix: <prefix_for_objects>
     cloud_logging:
       log_group_id: <log_group_ID>
     data_stream:
       stream_name: <YDS_name>
       database_id: <YDS_database_ID>
   service_account_id: <service_account_ID>
   filtering_policy:
     management_events_filter:
       resource_scopes:
         - id: <cloud_or_folder_organization_ID>
           type: <type>
     data_events_filters:
       - service: <service_name>
         resource_scopes:
           - id: <cloud_or_folder_organization_ID>
             type: <type>
         # You can specify included_events, excluded_events, 
         # or neither of the two to collect all the events of the service.
         # Both parameters are provided for illustration purposes.
         included_events:
           event_types:
             - <these_events_will_be_collected>
         excluded_events:
           event_types:
             - <these_events_will_not_be_collected>
   

   Where:

   • name: Trail name. It must be unique within the folder.

   • folder_id: ID of the folder the trail will reside in.

   • destination: Settings of the selected destination the audit logs will be uploaded to.

     Warning

     Destination settings are mutually exclusive. Using some settings makes it impossible to use others.

     • object_storage: Uploading logs to the Object Storage bucket:

       • bucket_id: Name of the bucket you created earlier.

         You can request the name of the bucket with the list of buckets in the folder (the default folder is used):

         yc storage bucket list
         

       • object_prefix: Prefix that will be assigned to the objects with audit logs in the bucket. It is an optional parameter used in the full name of the audit log file.

         Note

         Use a prefix to store audit logs and third-party data in the same bucket. Do not use the same prefix for logs and other bucket objects because that may cause logs and third-party objects to overwrite each other.

     • cloud_logging: Upload logs to a Cloud Logging group.

       In the log_group_id parameter, specify the ID of the log group you created earlier. You can request the ID with the list of log groups in the folder.

     • data_stream: Upload logs to a Data Streams data stream:

       • stream_name: Name of the data stream you created earlier. You can request the name with the list of data streams in the folder.

       • database_id: ID of the YDB database used by Data Streams. You can request the ID with the list of YDB databases in the folder.

   • service_account_id: ID of the service account you created earlier.

   • filtering_policy: The settings of the filtering policy that determines which events to collect and include in the audit logs. The policy consists of filters pertaining to different levels of events.

     Warning

     You must configure at least one filter for the policy; otherwise, you will not be able to create a trail.

     Available filters:

     • management_events_filter: Management event filter.

       Specify the log collection scope in the resource_scopes parameter:

       • id: Organization, cloud, or folder ID.

       • type: Scope type according to the specified ID:

         • organization-manager.organization: Organization
         • resource-manager.cloud: Cloud
         • resource-manager.folder: Folder

       You can combine several scopes belonging to the same organization in one resource_scopes parameter. For example, to collect logs from one entire cloud and only from particular folders in another cloud:

       resource_scopes:
         # Collecting logs from entire Cloud 1
         - id: <ID_of_cloud_1>
           type: resource-manager.cloud
         # Collecting logs from Folder 1 of Cloud 2
         - id: <folder_1_ID>
           type: resource-manager.folder
         # Collecting logs from Folder 2 of Cloud 2
         - id: <folder_2_ID>
           type: resource-manager.folder
       

       Service account permissions must allow collecting logs from the specified scopes.

     • data_events_filters: Data event filters. You can configure several filters of this type: one for each service.

       A filter for one service has the following structure:

       • service: Service name. You can get it from the data event reference.

       • resource_scopes: Places to collect data events from. You can configure this parameter the same way as the management event filter.

       • *_events: Data event filters.

         • included_events.event_types: Collect only specified events.
         • excluded_events.event_types: Collect all events other than the specified ones.

         You can get a list of events from the data event reference.

         Warning

         These filters are mutually exclusive:

         • Either configure included_events
         • or excluded_events

         If you configure neither, all events will be collected.

2. Run this command:

   yc audit-trails trail create --file <path_to_file>
   

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the documentation on the Terraform website or mirror website.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

1. In the configuration file, describe the parameters of the trail to collect audit logs:

   resource "yandex_audit_trails_trail" "basic_trail" {
     name = "<trail_name>"
     folder_id   = "<folder_ID>"
     description = "<trail_description>"
     
     labels = {
       key = "value"
     }
     
     service_account_id = "<service_account_ID>"
     
     logging_destination {
       log_group_id = "<log_group_ID>"
     }
     
     filter {
       path_filter {
         some_filter {
           resource_id   = "<organization_ID>"
           resource_type = "<parent_resource_type>"
           any_filters {
               resource_id   = "<ID_of_cloud_1>"
               resource_type = "<child_resource_type>"
           }
           any_filters {
               resource_id   = "<ID_of_cloud_2>"
               resource_type = "<child_resource_type>"
           }
         }
       }
       event_filters {
         service = "<ID_of_service_1>"
         categories {
           plane = "DATA_PLANE"
           type  = "<action_type>"
         }
         path_filter {
           any_filter {
             resource_id = "<organization_ID>"
             resource_type = "<resource_type>"
           }
         }
       }
       event_filters {
         service = "<ID_of_service_2>"
         categories {
           plane = "DATA_PLANE"
           type  = "<action_type>"
         }
         path_filter {
           any_filter {
             resource_id = "<organization_ID>"
             resource_type = "<resource_type>"
           }
         }
       }
     }
   }
   

   Where:

   • name: Name of the trail being created. The name format requirements are as follows:

     • It must be 2 to 63 characters long.
     • It may contain lowercase Latin letters, numbers, and hyphens.
     • It must start with a letter and cannot end with a hyphen.

   • folder_id: ID of the folder to create a trail in.

   • description: Description of the trail to distinguish it from other trails, e.g., My very first trail. This is an optional parameter.

   • labels: List of labels in key=value format. This is an optional parameter.

   • service_account_id: ID of the service account that the trail will use to upload audit log files to the bucket.

     Based on the audit log collection scope, the service account must have the audit-trails.viewer role for the organization, cloud, or folder whose audit logs the trail will collect.

   • log_group_id: ID of the log group for the trail to save audit logs to.
   • filter: Structure describing how the trail processes events. It contains the path_filter and event_filters objects.

     • path_filter: Structure describing how the trail processes management (control plane) events. It may contain a single some_filter object or a single any_filter object. If the path_filter object is missing in the configuration, the trail will not process control plane events.

       • some_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.

         • resource_id: ID of the parent resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.

         • resource_type: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specify organization-manager.organization or resource-manager.cloud in this parameter.

         • any_filters: Structure describing one child resource audit logs will be collected in. To specify multiple child resources, set the any_filters parameter as many times as needed. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:

           • resource_id: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.
           • resource_type: Child resource type. Based on the audit log collection scope, specify resource-manager.cloud or resource-manager.folder in this parameter.

         To set the current folder as the audit log collection scope, use the any_filter parameter instead of some_filter.

         You cannot use some_filter together with any_filter.

       • any_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use any_filter together with some_filter.

         • resource_id: ID of the resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.
         • resource_type: Yandex Cloud resource type. Based on the audit log collection scope, specify organization-manager.organization, resource-manager.cloud, or resource-manager.folder.

     • event_filters: Structure describing how the trail processes data (data plane) evens. If the event_filters object is missing in the configuration, the trail will not process data plane events. A single event_filters section is used to configure data plane audit log processing for a single Yandex Cloud service. To configure the trail processing of data plane events for multiple services, specify the event_filters parameter in the configuration as many times as you need.

       • service: ID of the service whose events the trail will process.
         The possible values include:

         • dns
         • kms
         • lockbox
         • mdb.mongodb
         • mdb.mysql
         • mdb.postgresql
         • storage

       • categories: Structure describing the type of events to collect.

         • plane: Event plane. For data plane events, specify DATA_PLANE.
         • type: Type of resource event action. The possible values are READ and WRITE.

       • path_filter: Structure describing how the trail processes data (data plane) events. It may contain a single some_filter object or a single any_filter object.

         • some_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.

           • resource_id: ID of the parent resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.

           • resource_type: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specify organization-manager.organization or resource-manager.cloud in this parameter.

           • any_filters: Structure describing one child resource audit logs will be collected in. To specify multiple child resources, set the any_filters parameter as many times as needed. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:

             • resource_id: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.
             • resource_type: Child resource type. Based on the audit log collection scope, specify resource-manager.cloud or resource-manager.folder in this parameter.

           To set the current folder as the audit log collection scope, use the any_filter parameter instead of some_filter.

           You cannot use some_filter together with any_filter.

         • any_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use any_filter together with some_filter.

           • resource_id: ID of the resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.
           • resource_type: Yandex Cloud resource type. Based on the audit log collection scope, specify organization-manager.organization, resource-manager.cloud, or resource-manager.folder.

   For more information about the yandex_audit_trails_trail resource parameters in Terraform, see the provider documentation.

2. Create the resources:

   1. In the terminal, change to the folder where you edited the configuration file.

   2. Make sure the configuration file is correct using the command:

      terraform validate
      

      If the configuration is correct, the following message is returned:

      Success! The configuration is valid.
      

   3. Run the command:

      terraform plan
      

      The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

   4. Apply the configuration changes:

      terraform apply
      

   5. Confirm the changes: type yes in the terminal and press Enter.

   Terraform will create all the required resources. You can check the new resources and their settings using the management console or this CLI command:

   yc audit-trails trail get <trail_name>
   

Use the create REST API method for the Trail resource or the TrailService/Create gRPC API call.