Uploading logs from Yandex Audit Trails to OpenSearch

Audit Trails is a service for collecting and exporting audit logs that allows you to use analytical tools and rapidly respond to events occurring to Yandex Cloud resources. OpenSearch is used as a SIEM system for analyzing logs and responding to security events.

You will learn how to set up the export of logs from Audit Trails with just a few steps, while using Yandex Data Streams and Yandex Data Transfer and selecting Yandex Managed Service for OpenSearch as a SIEM system to analyze logs and respond to security events.

Create a trail to upload audit logs for Yandex Cloud resources to a Yandex Data Streams data stream. Once done, configure continuous log delivery to a Yandex Managed Service for OpenSearch cluster using Yandex Data Transfer.

You can export organization, cloud, or folder logs.

To export audit logs:

1. Prepare your cloud.
2. Create a trail to send logs to a Data Streams data stream.
3. Create a Managed Service for OpenSearch cluster.
4. Set up a transfer to deliver logs to the Managed Service for OpenSearch cluster.
5. Check the result.
6. Upload additional content.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The infrastructure support cost includes:

• Fee for the OpenSearch cluster computing resources and storage (see Yandex Managed Service for OpenSearch pricing).
• Using a data stream (see Data Streams pricing).
• Using Yandex Managed Service for YDB in serverless mode (see Managed Service for YDB pricing).

Create a trail to send logs to a Data Streams data streamCreate a trail to send logs to a Data Streams data stream

Prepare the environment and create a trail depending on the Yandex Cloud resources:

• Organization
• Cloud
• Folder

Make sure to give your stream the audit‑trails name to make it easier to upload the Security Content library objects.

Create a Managed Service for OpenSearch clusterCreate a Managed Service for OpenSearch cluster

Set up a transfer to deliver logs to the Managed Service for OpenSearch clusterSet up a transfer to deliver logs to the Managed Service for OpenSearch cluster

1. Create a source endpoint:

   • Database type: Yandex Data Streams.

   • Endpoint parameters:

     • Connection settings:

       • Database: Select the Managed Service for YDB database from the list.
       • Stream: Specify the name of the Data Streams data stream.
       • Service account: Select or create a service account with the yds.editor role.

     • Advanced settings:

       • Conversion rules: AuditTrails.v1 parser.

2. Create a target endpoint:

   • Database type: OpenSearch.

   • Endpoint parameters:

     • Connection:

       • Connection type: Managed Service for OpenSearch cluster.

         • Managed Service for OpenSearch cluster: Select the source cluster from the list.

       • User and Password: Enter the name and password of the user who has access to the database, e.g., admin.

3. Create and activate the transfer:

   Manually

   Using Terraform

   1. Create a transfer of the Replication type that will use the created endpoints.
   2. Activate the transfer and wait for its status to change to Replicating.

   1. In the trails-to-opensearch.tf file, specify the following variables:

      • source_endpoint_id: Source endpoint ID.
      • target_endpoint_id: Target endpoint ID.
      • transfer_enabled: Set to 1 to enable transfer creation.

   2. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   3. Create the required infrastructure:

      1. Run the command to view planned changes:

         terraform plan
         

         If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

         1. Run the command:

            terraform apply
            

         2. Confirm the update of resources.

         3. Wait for the operation to complete.

   4. The transfer will be activated automatically. Wait for its status to change to Replicating.

Check the resultCheck the result

Make sure the data from Audit Trails is successfully uploaded to OpenSearch:

1. Wait for the transfer status to change to Replicating.

2. Connect to the target cluster using OpenSearch Dashboards.

3. Select the Global tenant.

4. Create a new index template named audit-trails*:

   1. Open the control panel by clicking .
   2. Under Management, select Stack Management.
   3. Go to Index Patterns and click create an index pattern at the bottom of the page.
   4. In the Index pattern name field, specify audit-trails* and click Next step.
   5. In Time field, select application_usage_daily.timestamp and click Create index pattern.

5. Open the control panel by clicking .

6. Under OpenSearch Dashboards, select Discover.

7. The dashboard that opens should contain data from Audit Trails in Elastic Common Schema format.

Upload additional contentUpload additional content

For your convenience, the Yandex Cloud security team created Solution Library with examples and recommendations for building a secure infrastructure in Yandex Cloud. The library is available in this public GitHub repository. It contains the following objects to upload to OpenSearch:

• Dashboard with use cases and statistics.
• Set of ready-to-use queries to search for security events.
• Sample events with preset alerts (the client should specify the alert destination on their own).

All required event fields are converted to Elastic Common Schema (ECS) format; the complete mapping table is provided in the Yandex Cloud Security Solution Library document.

To use Security Content:To use Security Content:

1. Clone the Yandex Cloud Security Solution Library repository:

   git clone https://github.com/yandex-cloud-examples/yc-export-auditlogs-to-opensearch.git
   

2. Connect to the target cluster using OpenSearch Dashboards.

3. Open the control panel by clicking .

4. Under Management, select Stack Management.

5. Go to Saved Objects and import files from the yc-export-auditlogs-to-opensearch/update-opensearch-scheme/content-for-transfer/ folder:

   • dashboard.ndjson
   • filters.ndjson
   • search.ndjson

DashboardDashboard

Use the ready-made Audit-trails-dashboard:

1. Open the control panel by clicking .
2. Under OpenSearch Dashboards, select Dashboard.
3. Select Audit-trails-dashboard in the dashboard list.

Security eventsSecurity events

Run a ready-to-use query to view security events that can be selected using filters.

1. Open the control panel by clicking .
2. Under OpenSearch Dashboards, select Discover.
3. In the Open tab, select the Search:Yandexcloud: Yandexcloud: Interesting fields query.

Alert settingsAlert settings

Use code samples for the monitor and trigger entities when setting up alerts:

1. Open the control panel by clicking .

2. Under OpenSearch Plugins, select Alerting.

3. Copy the sample file contents and paste them into the creation window:

   • monitor.json
   • trigger_action_example.json

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the transfer.
2. Delete endpoints for both the source and target.
3. Delete the Managed Service for YDB database.
4. Delete the created service accounts.
5. Delete the Audit Trails trail.

Delete the other resources depending on how they were created: