Uploading audit logs to Yandex Managed Service for OpenSearch

Create a trail to upload audit logs for Yandex Cloud resources to a Yandex Data Streams data stream. Once done, configure continuous log delivery to a Yandex Managed Service for OpenSearch cluster using Yandex Data Transfer.

You can export organization, cloud, or folder logs.

To export audit logs:

1. Get your cloud ready.
2. Create a trail to send logs to the stream in Data Streams.
3. Create a Managed Service for OpenSearch cluster.
4. Set up a transfer to deliver logs to the Managed Service for OpenSearch cluster.
5. Check the result.
6. Upload additional content.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

• Managed Service for OpenSearch cluster: Use of computing resources and storage size (see Managed Service for OpenSearch pricing).

• Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

• Data Streams (see Data Streams pricing). The cost depends on the pricing model:

  • Based on allocated resources: You pay a fixed hourly rate for the established throughput limit and message retention period, and additionally for the number of units of actually written data.
  • On-demand: You pay for the performed read/write operations, the amount of read or written data, and the actual storage used for messages that are still within their retention period.

• Managed Service for YDB database, operating in serverless mode: data operations, amount of stored data and backups (see Managed Service for YDB pricing).

Create a trail to send logs to a Data Streams data streamCreate a trail to send logs to a Data Streams data stream

Create a trail to send logs to a data stream named audit‑trails. Using a stream with this name makes it easier to upload the Security Content library objects.

When creating a trail, select the log collection scope.

Create a Managed Service for OpenSearch clusterCreate a Managed Service for OpenSearch cluster

Set up a transfer to deliver logs to the Managed Service for OpenSearch clusterSet up a transfer to deliver logs to the Managed Service for OpenSearch cluster

1. Create a source endpoint:

   • Database type: Yandex Data Streams.

   • Endpoint settings:

     • Connection settings:

       • Database: Select the Managed Service for YDB database from the list.
       • Stream: Specify the name of the stream in Data Streams.
       • Service account: Select or create a service account with the yds.editor role.

     • Advanced settings:

       • Conversion rules: AuditTrails.v1 parser.

2. Create a target endpoint:

   • Database type: OpenSearch.

   • Endpoint parameters:

     • Connection:

       • Connection type: Managed Service for OpenSearch cluster.

         • Managed Service for OpenSearch cluster: Select the source cluster from the list.

       • User and Password: Enter the name and password of the user who has access to the database, e.g., admin user.

3. Create and activate your transfer:

   Manually

   Using Terraform

   1. Create a Replication-type transfer configured to use the new endpoints.
   2. Activate the transfer and wait for its status to change to Replicating.

   1. In the trails-to-opensearch.tf file, specify these variables:

      • source_endpoint_id: Source endpoint ID.
      • target_endpoint_id: Target endpoint ID.
      • transfer_enabled: 1 to create a transfer.

   2. Validate your Terraform configuration files using this command:

      terraform validate
      

      Terraform will display any configuration errors detected in your files.

   3. Create the required infrastructure:

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

   4. The transfer will be activated automatically. Wait for its status to change to Replicating.

Check the resultCheck the result

Make sure the data from Audit Trails is successfully uploaded to OpenSearch:

1. Wait for the transfer status to change to Replicating.

2. Connect to the target cluster using OpenSearch Dashboards.

3. Select the Global tenant.

4. Create a new index template named audit-trails*:

   1. Open the management panel by clicking .
   2. Under Management, select Stack Management.
   3. Go to Index Patterns and click create an index pattern at the bottom of the page.
   4. In the Index pattern name field, specify audit-trails* and click Next step.
   5. In Time field, select application_usage_daily.timestamp and click Create index pattern.

5. Open the management panel by clicking .

6. Under OpenSearch Dashboards, select Discover.

7. The dashboard that opens should contain data from Audit Trails in Elastic Common Schema format.

Upload additional contentUpload additional content

For your convenience, the Yandex Cloud security team created Solution Library with examples and recommendations for building a secure infrastructure in Yandex Cloud. The library is available in this public GitHub repository. It contains the following objects to upload to OpenSearch:

• Dashboard with use cases and statistics.
• Set of ready-to-use queries to search for security events.
• Sample events with preset alerts (the client should specify the alert destination on their own).

All required event fields are converted to Elastic Common Schema (ECS) format; the complete mapping table is provided in the Yandex Cloud Security Solution Library document.

To use Security Content:To use Security Content:

1. Clone the Yandex Cloud Security Solution Library repository:

   git clone https://github.com/yandex-cloud-examples/yc-export-auditlogs-to-opensearch.git
   

2. Connect to the target cluster using OpenSearch Dashboards.

3. Open the management panel by clicking .

4. Under Management, select Stack Management.

5. Go to Saved Objects and import files from the yc-export-auditlogs-to-opensearch/update-opensearch-scheme/content-for-transfer/ folder:

   • dashboard.ndjson
   • filters.ndjson
   • search.ndjson

DashboardDashboard

Use the ready-made Audit-trails-dashboard:

1. Open the management panel by clicking .
2. Under OpenSearch Dashboards, select Dashboard.
3. Select Audit-trails-dashboard in the dashboard list.

Security eventsSecurity events

Run a ready-to-use query to view security events that can be selected using filters.

1. Open the management panel by clicking .
2. Under OpenSearch Dashboards, select Discover.
3. In the Open tab, select Search:Yandexcloud: Yandexcloud: Interesting fields.

Setting up alertsSetting up alerts

Use code examples for the monitor and trigger entities when setting up alerts:

1. Open the management panel by clicking .

2. Under OpenSearch Plugins, select Alerting.

3. Copy the sample file contents and paste them into the creation window:

   • monitor.json
   • trigger_action_example.json

Delete the resources you createdDelete the resources you created

To reduce the consumption of resources you do not need, delete them:

1. Delete the transfer.

2. Delete the source and target endpoints.

3. Delete the Managed Service for YDB database.

4. Delete the created service accounts.

5. Delete the Audit Trails trail.

6. Delete other resources using the same method used for their creation:

   Manually

   Using Terraform

   Delete the Managed Service for OpenSearch cluster.

   1. In the terminal window, go to the directory containing the infrastructure plan.

      Warning

      Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

   2. Delete resources:

      1. Run this command:

         terraform destroy
         

      2. Confirm deleting the resources and wait for the operation to complete.

      All the resources described in the Terraform manifests will be deleted.