Exporting audit logs to Yandex Managed Service for OpenSearch

Create a trail to upload audit logs for Yandex Cloud resources to a Yandex Data Streams data stream. Once done, configure continuous log delivery to a Yandex Managed Service for OpenSearch cluster using Yandex Data Transfer.

You can export organization, cloud, or folder logs.

To export audit logs:

1. Get your cloud ready.
2. Create a trail to send logs to a Data Streams data stream.
3. Create a Managed Service for OpenSearch cluster.
4. Set up a transfer to deliver logs to the Managed Service for OpenSearch cluster.
5. Check the result.
6. Upload additional content.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

The infrastructure support cost includes:

• Fee for the OpenSearch cluster computing resources and storage (see Yandex Managed Service for OpenSearch pricing).
• Using a data stream (see Data Streams pricing).
• Using Yandex Managed Service for YDB in serverless mode (see Managed Service for YDB pricing).

Create a trail to send logs to a Data Streams data streamCreate a trail to send logs to a Data Streams data stream

Create a trail to send logs to a data stream named audit‑trails. Using a stream with this name makes it easier to upload the Security Content library objects.

When creating a trail, select the log collection scope.

Create a Managed Service for OpenSearch clusterCreate a Managed Service for OpenSearch cluster

Set up a transfer to deliver logs to the Managed Service for OpenSearch clusterSet up a transfer to deliver logs to the Managed Service for OpenSearch cluster

1. Create a source endpoint:

   • Database type: Yandex Data Streams.

   • Endpoint settings:

     • Connection settings:

       • Database: Select the Managed Service for YDB database from the list.
       • Stream: Specify the name of the Data Streams-enabled stream.
       • Service account: Select or create a service account with the yds.editor role.

     • Advanced settings:

       • Conversion rules: AuditTrails.v1 parser.

2. Create a target endpoint:

   • Database type: OpenSearch.

   • Endpoint parameters:

     • Connection:

       • Connection type: Managed Service for OpenSearch cluster.

         • Managed Service for OpenSearch cluster: Select the source cluster from the list.

       • User and Password: Enter the name and password of the user who has access to the database, e.g., admin user.

3. Create and activate the transfer:

   Manually

   Using Terraform

   1. Create a transfer of the Replication type that will use the created endpoints.
   2. Activate the transfer and wait until its status switches to Replicating.

   1. In the trails-to-opensearch.tf file, specify these variables:

      • source_endpoint_id: Source endpoint ID.
      • target_endpoint_id: Target endpoint ID.
      • transfer_enabled: 1 to create a transfer.

   2. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   3. Create the required infrastructure:

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

   4. The transfer will be activated automatically. Wait for its status to change to Replicating.

Check the resultCheck the result

Make sure the data from Audit Trails is successfully uploaded to OpenSearch:

1. Wait until the transfer status switches to Replicating.

2. Connect to the target cluster using OpenSearch Dashboards.

3. Select the Global tenant.

4. Create a new index template named audit-trails*:

   1. Open the control panel by clicking .
   2. Under Management, select Stack Management.
   3. Go to Index Patterns and click create an index pattern at the bottom of the page.
   4. In the Index pattern name field, specify audit-trails* and click Next step.
   5. In Time field, select application_usage_daily.timestamp and click Create index pattern.

5. Open the control panel by clicking .

6. Under OpenSearch Dashboards, select Discover.

7. The dashboard that opens should contain data from Audit Trails in Elastic Common Schema format.

Upload additional contentUpload additional content

For your convenience, the Yandex Cloud security team created Solution Library with examples and recommendations for building a secure infrastructure in Yandex Cloud. The library is available in this public GitHub repository. It contains the following objects to upload to OpenSearch:

• Dashboard with use cases and statistics.
• Set of ready-to-use queries to search for security events.
• Sample events with preset alerts (the client should specify the alert destination on their own).

All required event fields are converted to Elastic Common Schema (ECS) format; the complete mapping table is provided in the Yandex Cloud Security Solution Library document.

To use Security Content:To use Security Content:

1. Clone the Yandex Cloud Security Solution Library repository:

   git clone https://github.com/yandex-cloud-examples/yc-export-auditlogs-to-opensearch.git
   

2. Connect to the target cluster using OpenSearch Dashboards.

3. Open the control panel by clicking .

4. Under Management, select Stack Management.

5. Go to Saved Objects and import files from the yc-export-auditlogs-to-opensearch/update-opensearch-scheme/content-for-transfer/ folder:

   • dashboard.ndjson
   • filters.ndjson
   • search.ndjson

dashboarddashboard

Use the ready-made Audit-trails-dashboard:

1. Open the control panel by clicking .
2. Under OpenSearch Dashboards, select Dashboard.
3. Select Audit-trails-dashboard in the dashboard list.

Security eventsSecurity events

Run a ready-to-use query to view security events that can be selected using filters.

1. Open the control panel by clicking .
2. Under OpenSearch Dashboards, select Discover.
3. In the Open tab, select Search:Yandexcloud: Yandexcloud: Interesting fields.

Alert settingsAlert settings

Use code examples for the monitor and trigger entities when setting up alerts:

1. Open the control panel by clicking .

2. Under OpenSearch Plugins, select Alerting.

3. Copy the sample file contents and paste them into the creation window:

   • monitor.json
   • trigger_action_example.json

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the transfer.
2. Delete the endpoints for both the source and target.
3. Delete the Managed Service for YDB database.
4. Delete service accounts you created.
5. Delete the Audit Trails trail.

Delete the other resources depending on how they were created: