Uploading Audit Trails to Managed Service for OpenSearch

The scenario we’ll look at next will be useful to IS professionals. We will demonstrate how to configure audit log uploads from Audit Trails in a few simple steps, utilizing Data Streams and Data Transfer services, and selecting Managed Service for OpenSearch as your SIEM system for analyzing logs and responding to security events.

Configuring Data Transfer in OpenSearchConfiguring Data Transfer in OpenSearch

1. Create the first data trail to be uploaded to Data Streams. To make it easier to upload objects from the Security Content library, name the stream “audittrails.”

2. Deploy a cluster using Managed Service for OpenSearch.

3. Configure the Source endpoint of the Data Transfer service with Data Streams as the source. Make sure to select the AuditTrails.v1 parser option in the settings (Advanced settings → Conversion rules).

4. Configure the Receiver endpoint in the Data Transfer service using Managed Service for OpenSearch as the receiver. Before importing data into the OpenSearch cluster, create a user with limited access and specify their details.

5. We’ve set up two endpoints. To enable data transfer between them, click the Activate button.

6. All done! The data transfer is in progress.

Verifying data transferVerifying data transfer

Check that the data was loaded into OpenSearch successfully.

1. In the OpenSearch cluster’s web interface, open the Global tenant. Create an index pattern containing the “audittrails*” string. The index into which the data from Audit Trails is loaded will be named “audittrails” after the data stream’s name in Data Streams.

2. Your Audit Trails data will appear in the Elastic Common Schema format on the Discover tab.

Uploading additional contentUploading additional content

The Yandex Cloud security team has compiled Security Content, a library of objects that can be uploaded to OpenSearch, including:

• a dashboard, including case studies and stats
• a collection of pre-built queries for simple security event searches
• examples of events with configured notifications (the purpose of the notifications must be specified manually)

All required event fields have been converted to Elastic Common Schema (ECS) format. The full mapping table can be downloaded here.

To use Security Content:

1. Download the content and run the command:

git clone https://github.com/yandex-cloud/yc-solution-library-for-security.git.

2. Go to the auditlogs/export-auditlogs-to-Opensearch/update-opensearch-scheme/include/audit-trail folder and run the following command:

cd yc-solution-library-for-security/auditlogs/export-auditlogs-to-Opensearch/update-opensearch-scheme/content-for-transfer/.

3. In the OpenSearch console, go to  Stack management → Saved Objects → Import and import the dashboard.ndjson, filters.ndjson, search.ndjson files.

4. Open the dashboard.

5. In the Discover section, go to the Open tab and enter the query Search: Yandexcloud: Yandexcloud: Interesting fields. The columns contain events that can be filtered.

6. Alerts can be set up in OpenSearch. To save time when parsing the format for writing the monitor entity, we have prepared a sample code that you can simply copy into the monitor creation window. You can also use the example of creating a trigger action by specifying the event fields.

Feel free to modify our service to your needs and contact us if you have any questions or concerns.