Contact UsGet started

Uploading organization audit logs to Cloud Logging

Written by
Updated at November 14, 2024

Follow this guide to create a new trail that will upload management event audit logs of all resources and, optionally, data event audit logs of selected services of an organization to a Cloud Logging log group.

Prepare the environment

To export organization audit logs:

  1. Create a log group to upload audit logs to.

  2. Create a service account.

  3. Assign roles to the service account:

    If you do not have the Yandex Cloud command line interface yet, install and initialize it.

    The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

    • audit-trails.viewer for the organization whose audit logs you will collect:

      yc organization-manager organization add-access-binding \
  --role audit-trails.viewer \
  --id <organization_ID> \
  --service-account-id <service_account_ID>

      Where:

      • --role: Role you want to assign.
      • --id: ID of the organization from whose resources you will collect audit logs.
      • --service-account-id: Service account ID.

    • logging.writer for the folder to host the trail:

      yc resource-manager folder add-access-binding \
  --role logging.writer \
  --id <folder_ID> \
  --service-account-id <service_account_ID>

      Where:

      • --role: Role you want to assign.
      • --id: ID of the folder to host the trail.
      • --service-account-id: Service account ID.

  4. On the Access permissions page, make sure you have the following roles:

    • iam.serviceAccounts.user for the service account.
    • audit-trails.editor for the folder to host the trail.
    • audit-trails.viewer for the organization whose audit logs you will collect.
    • logging.viewer for the Cloud Logging log group.

Creating a trail

To create a trail that exports organization audit logs:

  1. In the management console, select the folder to host the trail.

  2. Select Audit Trails.

  3. Click Create trail and specify:

    • Name: Name of the new trail.
    • Description: Trail description (optional).

  4. Under Destination, configure the destination object:

    • Destination: Cloud Logging.

    • Log group: Select a log group. You can also create a new log group. For this:

      • Click Create new, then specify the parameters of the log group:

        • Name: Name of the new group.
        • Description: Log group description (optional).
        • Log retention.

      • Click Create group.

  5. Under Service account, select the service account that the trail will use to upload audit log files to the log group.

  6. Under Collecting management events, configure the collection of management event audit logs:

    • Collecting events: Select Enabled.
    • Resource: Select Organization.
    • Organization: Automatically populated field containing the name of the current organization.
    • Cloud: Keep the default value, All.

  7. Optionally, under Collecting data events, configure the collection of data event audit logs:

    • Collecting events: Select Enabled.

    • Select the services to collect audit logs for.

    • For each service you select, specify the audit log collection scope and event filter type:

      • Receive all: To receive all events within the service.
      • Selected: To receive only the selected events. Next, select the events.
      • Exclude: To receive all events except for the selected ones. Next, select the events.

  8. Click Create.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

  1. View the description of the CLI command for creating a trail:

    yc audit-trails trail create --help

  2. Run the command to create a trail to collect management event audit logs in the organization:

    yc audit-trails trail create \
  --name <trail_name> \
  --description "<trail_description>" \
  --service-account-id <service_account_ID> \
  --destination-log-group-id <log_group_ID> \
  --filter-from-organisation-id <organization_ID> \
  --filter-some-cloud-ids <list_of_clouds_in_organization>

    Where:

    • --name: Name of the trail being created.
      The name format requirements are as follows:

      • The name must be from 3 to 63 characters long.
      • It may contain lowercase Latin letters, numbers, and hyphens.
      • The first character must be a letter and the last character cannot be a hyphen.

    • --description: Description of the trail to distinguish it from other trails, e.g., --description "My very first trail". This is an optional parameter.

    • --service-account-id: ID of the service account that the trail will use to upload audit log files to the bucket.

      Based on the audit log collection scope, the service account must have the audit-trails.viewer role for the organization, cloud, or folder whose audit logs the trail will collect.

    • --filter-from-organisation-id: ID of the organization the new trail belongs to and for whose resources audit logs will be collected.

      When using the --filter-from-organisation-id parameter, you also need to set cloud IDs in the --filter-some-cloud-ids parameter.

      You cannot use --filter-from-organisation-id together with --filter-all-organisation-id.

    • --filter-some-cloud-ids: List of IDs of the clouds for whose resources the trail will collect audit logs. Use this parameter only if the --filter-from-organisation-id parameter is set.

      The clouds specified in the parameter must belong to the organization named in the --filter-from-organisation-id parameter.

      If you need to collect audit logs in all clouds belonging to an organization, use the --filter-all-organisation-id parameter.

    • --filter-all-organisation-id: ID of the organization the new trail belongs to and for whose resources audit logs will be collected from all clouds.

      You cannot use --filter-all-organisation-id together with --filter-from-organisation-id.

    Result:
    done (1s)
id: cnpa0r6r5157********
folder_id: b1g9d2k0itu4********
created_at: "2024-04-01T09:48:16.337Z"
updated_at: "2024-04-01T09:48:16.337Z"
name: sample-trail
description: My very first trail
destination:
  cloud_logging:
    log_group_id: e23fvhkglsac********
service_account_id: ajeee339l4m5********
status: ACTIVE
filter:
  path_filter:
    root:
      some_filter:
        resource:
          id: bpfaidqca8vd********
          type: organization-manager.organization
        filters:
          - any_filter:
              resource:
                id: b1glti4eser3********
                type: resource-manager.cloud
          - any_filter:
              resource:
                id: b1gssd27h7ra********
                type: resource-manager.cloud
  event_filter: {}
cloud_id: b1glti4eser3********

  3. To create a trail to collect management event and data event audit logs, use the trail's YAML specification file.

    1. Create a trail configuration file, e.g., specification.yaml:

      specification.yaml
      folder_id: b1g9d2k0itu4********
name: sample-trail
description: My very first trail
labels:
  key: value
destination:
  cloud_logging:
    log_group_id: e232pqboot3f********
service_account_id: ajeee339l4m5********
filter:
  path_filter:
    root:
      some_filter:
        resource:
          id: bpfaidqca8vd********
          type: organization-manager.organization
        filters:
          - any_filter:
              resource:
                id: b1glti4eser3********
                type: resource-manager.cloud
          - any_filter:
              resource:
                id: b1gssd27h7ra********
                type: resource-manager.cloud
  event_filter:
    filters:
      - service: mdb.postgresql
        categories:
          - plane: DATA_PLANE
            type: WRITE
        path_filter:
          root:
            some_filter:
              resource:
                id: bpfaidqca8vd********
                type: organization-manager.organization
              filters:
                - any_filter:
                    resource:
                      id: b1glti4eser3********
                      type: resource-manager.cloud
                - any_filter:
                    resource:
                      id: b1gssd27h7ra********
                      type: resource-manager.cloud
      - service: dns
        categories:
          - plane: DATA_PLANE
            type: READ
        path_filter:
          root:
            any_filter:
              resource:
                id: bpfaidqca8vd********
                type: organization-manager.organization
      Specification description

      • folder_id: ID of the folder to create a trail in.

      • name: Name of the trail being created. The name format requirements are as follows:

        • The name must be from 3 to 63 characters long.
        • It may contain lowercase Latin letters, numbers, and hyphens.
        • The first character must be a letter and the last character cannot be a hyphen.

      • description: Description of the trail to distinguish it from other trails, e.g., My very first trail. This is an optional parameter.

      • labels: List of labels in key=value format. This is an optional parameter.

      • service_account_id: ID of the service account that the trail will use to upload audit log files to the bucket.

        Based on the audit log collection scope, the service account must have the audit-trails.viewer role for the organization, cloud, or folder whose audit logs the trail will collect.

      • destination.cloud_logging.log_group_id: ID of the log group for the trail to save audit logs to.
      • filter: Structure describing how the trail processes events. It contains the path_filter and event_filter objects.

        • path_filter: Structure describing how the trail processes management (control plane) events. It may contain a single root.some_filter object or a single root.any_filter object. If the path_filter object is missing in the configuration, the trail will not process control plane events.

          • root.some_filter: Structure describing the resources to collect audit logs for. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.

            • resource.id: ID of the parent resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.

            • resource.type: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specify organization-manager.organization or resource-manager.cloud in this parameter.

            • filters.any_filter: Structure describing one child resource audit logs will be collected in. To specify multiple child resources, set the filters.any_filter parameter as many times as needed. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:

              • resource.id: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.
              • resource.type: Child resource type. Based on the audit log collection scope, specify resource-manager.cloud or resource-manager.folder in this parameter.

            To set the current folder as the audit log collection scope, use the root.any_filter parameter instead of root.some_filter.

            You cannot use root.some_filter together with root.any_filter.

          • root.any_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use root.any_filter together with root.some_filter.

            • resource.id: ID of the resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.
            • resource.type: Yandex Cloud resource type. Based on the audit log collection scope, specify organization-manager.organization, resource-manager.cloud, or resource-manager.folder.

        • event_filter: Structure describing how the trail processes data (data plane) events. If the event_filter object is missing in the configuration, the trail will not process data plane events.

          • filters.service: ID of the service whose events the trail will process. A single filters.service section is used to configure data plane audit log processing for a single Yandex Cloud service. To configure the trail processing of data plane events for multiple services, specify the filters.service parameter in the configuration as many times as you need.
            The possible values include:

            • dns
            • kms
            • lockbox
            • mdb.mongodb
            • mdb.mysql
            • mdb.postgresql
            • storage

          • filters.categories: Structure describing the type of events to collect.

            • plane: Event plane. For data plane events, specify DATA_PLANE.
            • type: Type of resource event action. The possible values are READ and WRITE.

          • filters.path_filter: Structure describing how the trail processes data (data plane) events. It may contain a single root.some_filter object or a single root.any_filter object.

            • root.some_filter: Structure describing the resources to collect audit logs for. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.

              • resource.id: ID of the parent resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.

              • resource.type: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specify organization-manager.organization or resource-manager.cloud in this parameter.

              • filters.any_filter: Structure describing one child resource audit logs will be collected in. To specify multiple child resources, set the filters.any_filter parameter as many times as needed. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:

                • resource.id: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.
                • resource.type: Child resource type. Based on the audit log collection scope, specify resource-manager.cloud or resource-manager.folder in this parameter.

              To set the current folder as the audit log collection scope, use the root.any_filter parameter instead of root.some_filter.

              You cannot use root.some_filter together with root.any_filter.

            • root.any_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use root.any_filter together with root.some_filter.

              • resource.id: ID of the resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.
              • resource.type: Yandex Cloud resource type. Based on the audit log collection scope, specify organization-manager.organization, resource-manager.cloud, or resource-manager.folder.

    2. Run the command, specifying the path to the created trail specification file:

      yc audit-trails trail create --file <specification_file_path>
      Result
      done (1s)
id: cnp678gn34hd********
folder_id: b1g9d2k0itu4********
created_at: "2024-04-02T03:46:19.563Z"
updated_at: "2024-04-02T03:46:19.563Z"
name: sample-trail
description: My very first trail
destination:
  cloud_logging:
    log_group_id: e232pqboot3f********
service_account_id: ajeee339l4m5********
status: ACTIVE
filter:
  path_filter:
    root:
      some_filter:
        resource:
          id: bpfaidqca8vd********
          type: organization-manager.organization
        filters:
          - any_filter:
              resource:
                id: b1glti4eser3********
                type: resource-manager.cloud
          - any_filter:
              resource:
                id: b1gssd27h7ra********
                type: resource-manager.cloud
  event_filter:
    filters:
      - service: mdb.postgresql
        categories:
          - plane: DATA_PLANE
            type: WRITE
        path_filter:
          root:
            some_filter:
              resource:
                id: bpfaidqca8vd********
                type: organization-manager.organization
              filters:
                - any_filter:
                    resource:
                      id: b1glti4eser3********
                      type: resource-manager.cloud
                - any_filter:
                    resource:
                      id: b1gssd27h7ra********
                      type: resource-manager.cloud
      - service: dns
        categories:
          - plane: DATA_PLANE
            type: READ
        path_filter:
          root:
            any_filter:
              resource:
                id: bpfaidqca8vd********
                type: organization-manager.organization
cloud_id: b1glti4eser3********

    For more information about the yc audit-trails trail create command, see the CLI reference.

Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

For more information about the provider resources, see the documentation on the Terraform website or the mirror.

If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

  1. In the configuration file, describe the parameters of the trail that will collect audit events of the organization's resources:

    resource "yandex_audit_trails_trail" "basic_trail" {
  name = "<trail_name>"
  folder_id   = "<folder_ID>"
  description = "<trail_description>"
  
  labels = {
    key = "value"
  }
  
  service_account_id = "<service_account_ID>"
  
  logging_destination {
    log_group_id = "<log_group_ID>"
  }
  
  filter {
    path_filter {
      some_filter {
        resource_id   = "<organization_ID>"
        resource_type = "<parent_resource_type>"
        any_filters {
            resource_id   = "<ID_of_cloud_1>"
            resource_type = "<child_resource_type>"
        }
        any_filters {
            resource_id   = "<ID_of_cloud_2>"
            resource_type = "<child_resource_type>"
        }
      }
    }
    event_filters {
      service = "<ID_of_service_1>"
      categories {
        plane = "DATA_PLANE"
        type  = "<action_type>"
      }
      path_filter {
        any_filter {
          resource_id = "<organization_ID>"
          resource_type = "<resource_type>"
        }
      }
    }
    event_filters {
      service = "<ID_of_service_2>"
      categories {
        plane = "DATA_PLANE"
        type  = "<action_type>"
      }
      path_filter {
        any_filter {
          resource_id = "<organization_ID>"
          resource_type = "<resource_type>"
        }
      }
    }
  }
}

    Where:

    • name: Name of the trail being created. The name format requirements are as follows:

      • The name must be from 3 to 63 characters long.
      • It may contain lowercase Latin letters, numbers, and hyphens.
      • The first character must be a letter and the last character cannot be a hyphen.

    • folder_id: ID of the folder to create a trail in.

    • description: Description of the trail to distinguish it from other trails, e.g., My very first trail. This is an optional parameter.

    • labels: List of labels in key=value format. This is an optional parameter.

    • service_account_id: ID of the service account that the trail will use to upload audit log files to the bucket.

      Based on the audit log collection scope, the service account must have the audit-trails.viewer role for the organization, cloud, or folder whose audit logs the trail will collect.

    • filter: Structure describing how the trail processes events. It contains the path_filter and event_filters objects.

      • path_filter: Structure describing how the trail processes management (control plane) events. It may contain a single some_filter object or a single any_filter object. If the path_filter object is missing in the configuration, the trail will not process control plane events.

        • some_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.

          • resource_id: ID of the parent resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.

          • resource_type: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specify organization-manager.organization or resource-manager.cloud in this parameter.

          • any_filters: Structure describing one child resource audit logs will be collected in. To specify multiple child resources, set the any_filters parameter as many times as needed. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:

            • resource_id: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.
            • resource_type: Child resource type. Based on the audit log collection scope, specify resource-manager.cloud or resource-manager.folder in this parameter.

          To set the current folder as the audit log collection scope, use the any_filter parameter instead of some_filter.

          You cannot use some_filter together with any_filter.

        • any_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use any_filter together with some_filter.

          • resource_id: ID of the resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.
          • resource_type: Yandex Cloud resource type. Based on the audit log collection scope, specify organization-manager.organization, resource-manager.cloud, or resource-manager.folder.

      • event_filters: Structure describing how the trail processes data (data plane) evens. If the event_filters object is missing in the configuration, the trail will not process data plane events. A single event_filters section is used to configure data plane audit log processing for a single Yandex Cloud service. To configure the trail processing of data plane events for multiple services, specify the event_filters parameter in the configuration as many times as you need.

        • service: ID of the service whose events the trail will process.
          The possible values include:

          • dns
          • kms
          • lockbox
          • mdb.mongodb
          • mdb.mysql
          • mdb.postgresql
          • storage

        • categories: Structure describing the type of events to collect.

          • plane: Event plane. For data plane events, specify DATA_PLANE.
          • type: Type of resource event action. The possible values are READ and WRITE.

        • path_filter: Structure describing how the trail processes data (data plane) events. It may contain a single some_filter object or a single any_filter object.

          • some_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in specific organization clouds or cloud folders associated with the trail.

            • resource_id: ID of the parent resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization ID or the ID of the cloud associated with the trail in this parameter.

            • resource_type: Type of the Yandex Cloud parent resource to create a trail in. Based on the audit log collection scope, specify organization-manager.organization or resource-manager.cloud in this parameter.

            • any_filters: Structure describing one child resource audit logs will be collected in. To specify multiple child resources, set the any_filters parameter as many times as needed. Based on the audit log collection scope, in this parameter, specify information on the clouds or folders whose events the trail will process:

              • resource_id: Child resource ID. Based on the audit log collection scope, specify the ID of the cloud or folder whose audit logs the trail will collect.
              • resource_type: Child resource type. Based on the audit log collection scope, specify resource-manager.cloud or resource-manager.folder in this parameter.

            To set the current folder as the audit log collection scope, use the any_filter parameter instead of some_filter.

            You cannot use some_filter together with any_filter.

          • any_filter: Structure describing the resources whose audit logs will be collected. Based on the audit log collection scope, it allows configuring event processing in all clouds of the organization associated with the trail, in all folders of the cloud associated with the trail, or in the current folder where you want to create the trail. You cannot use any_filter together with some_filter.

            • resource_id: ID of the resource the new trail belongs to and for whose resources audit logs will be collected. Based on the audit log collection scope, specify the organization, cloud, or folder ID in this parameter.
            • resource_type: Yandex Cloud resource type. Based on the audit log collection scope, specify organization-manager.organization, resource-manager.cloud, or resource-manager.folder.

    For more information about the yandex_audit_trails_trail resource parameters in Terraform, see the provider documentation.

  2. Create resources:

    1. In the terminal, change to the folder where you edited the configuration file.

    2. Make sure the configuration file is correct using the command:

      terraform validate

      If the configuration is correct, the following message is returned:

      Success! The configuration is valid.

    3. Run the command:

      terraform plan

      The terminal will display a list of resources with parameters. No changes are made at this step. If the configuration contains errors, Terraform will point them out.

    4. Apply the configuration changes:

      terraform apply

    5. Confirm the changes: type yes in the terminal and press Enter.

    Terraform will create all the required resources. You can check the new resources and their settings using the management console or this CLI command:

    yc audit-trails trail get <trail_name>

Use the create REST API method for the Trail resource or the TrailService/Create gRPC API call.

The trail will be created and will start uploading audit logs to the log group.

When uploading to Cloud Logging, you may get duplicate events in a log group. To find out whether an event in a log group is a duplicate, use the unique ID of the json_payload.event_id record.

What's next

Previous
Handling errors
Next
Cloud