Contact UsGet started

Uploading audit logs to ArcSight SIEM

Written by
Updated at November 14, 2024

Create a trail to upload management event audit logs of resources in an individual folder to an Yandex Object Storage bucket with encryption enabled. Then configure continuous log delivery to ArcSight SIEM.

To complete the tutorial successfully, you must have an ArcSight instance installed.

The solution described in the tutorial follows the steps below:

  1. A trail uploads logs to an Object Storage bucket.
  2. A bucket is mounted via a FUSE interface to a folder on an intermediate VM.
  3. SmartConnector collects logs from the folder and delivers them to ArcSight for analysis.

For more information about the scripts for delivering audit logs to ArcSight, see Yandex Cloud Security Solution Library.

Note

Yandex Cloud Security Solution Library is a public repo on GitHub with a set of examples and recommendations on how to build a secure infrastructure in Yandex Cloud.

To configure delivery of audit log files to ArcSight:

  1. Prepare your cloud.
  2. Prepare the environment.
  3. Assign roles to the service accounts.
  4. Create a trail.
  5. Mount the bucket.
  6. Connect ArcSight SmartConnector.

If you no longer need the resources you created, delete them.

Getting started

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

Sign up for Yandex Cloud and create a billing account:

  1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

The infrastructure support cost includes:

Prepare the environment

Prepare an intermediate VM

You can use a VM that has access to an ArcSight instance or create a new one:

  1. Create a VM from a Linux image based on Ubuntu 20.04.
  2. Connect to the VM over SSH.

Create a bucket for audit logs

  1. In the management console, select the folder where you want to create a bucket, e.g., example-folder.
  2. Select Object Storage.
  3. Click Create bucket.
  4. On the bucket creation page:

    1. Enter a name for the bucket according to the naming requirements.

    2. Limit the maximum bucket size, if required.

      If the value is 0, the maximum size is not limited and is similar to the enabled No limit option.

    3. Select the Restricted access type.

    4. Select the default storage class.

    5. Click Create bucket.

Create an encryption key in Key Management Service

  1. In the management console, go to example-folder.
  2. Select Key Management Service.
  3. Click Create key and specify:
    • Name: arcsight-kms.
    • Encryption algorithm: AES-256.
    • Leave the other parameters at their default settings.
  4. Click Create.

Enable bucket encryption

  1. In the management console, go to the previously created bucket.
  2. In the left-hand panel, select Security.
  3. Open the Encryption tab.
  4. In the KMS Key field, select arcsight-kms.
  5. Click Save.

Create service accounts

You need to create two accounts: one for a trail and one for a bucket.

Create a service account named sa-arcsight:

  1. In the management console, go to example-folder.

  2. In the list of services, select Identity and Access Management.

  3. Click Create service account.

  4. Enter a name for the service account according to the naming requirements:

    • The name must be from 3 to 63 characters long.
    • It may contain lowercase Latin letters, numbers, and hyphens.
    • The first character must be a letter and the last character cannot be a hyphen.

    Example: sa-arcsight.

  5. Click Create.

Similarly, create a service account named sa-arcsight-bucket.

Create a static key

You will need the key ID and secret key when mounting the bucket.

  1. In the management console, go to example-folder.

  2. In the list of services, select Identity and Access Management.

  3. In the left-hand panel, select Service accounts.

  4. Select the sa-arcsight-bucket service account from the list that opens.

  5. Click Create new key in the top panel.

  6. Select Create static access key.

  7. Enter a description for the key and click Create.

  8. Save the ID and private key.

    Alert

    After you close the dialog, the private key value will become unavailable.

  1. Create an access key for the sa-arcsight-bucket service account:

    yc iam access-key create --service-account-name sa-arcsight-bucket

    Result:

    access_key:
  id: aje*******k2u
  service_account_id: aje*******usm
  created_at: "2022-09-22T14:37:51Z"
  key_id: 0n8*******0YQ
secret: JyT*******zMP1

  2. Save the ID (key_id) and secret key (secret). You will not be able to get the key value again.

Assign roles to the service accounts

Assign the audit-trails.viewer, storage.uploader, and kms.keys.encrypterDecrypter roles to the sa-arcsight service account:

  1. audit-trails.viewer role for a folder:

    yc resource-manager folder add-access-binding \
--role audit-trails.viewer \
--id <folder_ID> \
--service-account-id <service_account_ID>

    Where:

    • --role: Role being assigned.
    • --id: example-folder ID.
    • --service-account-id: sa-arcsight service account ID.

    For more information about the yc resource-manager folder add-access-binding command, see the CLI reference.

  2. Assign the storage.uploader role for the folder the bucket is in:

    yc resource-manager folder add-access-binding \
--role storage.uploader \
--id <folder_ID> \
--service-account-id <service_account_ID>

    Where:

    • --role: Role being assigned.
    • --id: example-folder ID.
    • --service-account-id: sa-arcsight service account ID.

  3. kms.keys.encrypterDecrypter role for the arcsight-kms encryption key:

    yc kms symmetric-key add-access-binding \
--role kms.keys.encrypterDecrypter \
--id <key_ID> \
--service-account-id <service_account_ID>

    Where:

    • --role: Role being assigned.
    • --id: ID of the arcsight-kms KMS key.
    • --service-account-id: sa-arcsight service account ID.

Assign the storage.viewer and kms.keys.encrypterDecrypter roles to the sa-arcsight-bucket service account:

  1. storage.viewer role for a folder:

    yc resource-manager folder add-access-binding \
--id <folder_ID> \
--role storage.viewer \
--service-account-id <service_account_ID>

    Where:

    • --id: example-folder ID.
    • --role: Role being assigned.
    • --service-account-id: sa-arcsight-bucket service account ID.

  2. kms.keys.encrypterDecrypter role for the arcsight-kms encryption key:

    yc kms symmetric-key add-access-binding \
--role kms.keys.encrypterDecrypter \
--id <key_ID> \
--service-account-id <service_account_ID>

    Where:

    • --role: Role being assigned.
    • --id: ID of the arcsight-kms KMS key.
    • --service-account-id: sa-arcsight-bucket service account ID.

Create a trail

  1. In the management console, go to example-folder.

  2. Select Audit Trails.

  3. Click Create trail and specify:

    • Name: Name of the trail you want to create, e.g., arcsight-trail.
    • Description: Trail description (optional).

  4. Under Destination, configure the destination object:

    • Destination: Object Storage.
    • Bucket: Bucket name.
    • Object prefix: Optional parameter used in the full name of the audit log file.

    Note

    Use a prefix to store audit logs and third-party data in the same bucket. Do not use the same prefix for logs and other bucket objects because that may cause logs and third-party objects to overwrite each other.

    • Encryption key: Specify the arcsight-kms encryption key the bucket is encrypted with.

  5. Under Service account, select sa-arcsight.

  6. Under Collecting management events, configure the collection of management event audit logs:

    • Collecting events: Select Enabled.
    • Resource: Select Folder.
    • Folder: Automatically populated field containing the name of the current folder.

  7. Under Collecting data events, select Disabled in the Collecting events field.

  8. Click Create.

Warning

The solution will delete the logs from the bucket after they are exported to ArcSight. If you need to keep the logs in the bucket, create a separate bucket and trail.

Mount a bucket

A bucket is mounted on an intermediate VM where ArcSight SmartConnector is installed.
To mount the bucket, create a file with the sa-arcsight-bucket service account static access key.

  1. On the intermediate VM, create a file with the static access key:

    echo <access_key_ID>:<secret_access_key> > ${HOME}/.passwd-s3fs
chmod 600 ${HOME}/.passwd-s3fs

  2. Install s3fs:

    sudo apt install s3fs

  3. Create a directory to mount the bucket to, e.g., mybucket in your home directory:

    sudo mkdir ${HOME}/mybucket

  4. Mount the bucket:

    s3fs <bucket_name> ${HOME}/mybucket -o passwd_file=${HOME}/.passwd-s3fs -o url=https://storage.yandexcloud.net -o use_path_request_style

  5. Check that the bucket was mounted:

    ls ${HOME}/mybucket

Install and configure ArcSight SmartConnector

Note

To complete this stage of the tutorial, you need an ArcSight SmartConnector distribution and access to an ArcSight instance.

  1. On the intermediate VM, install ArcSight SmartConnector:

    1. When installing it, select ArcSight FlexConnector JSON Folder Follower and specify the path to the mybucket folder.
    2. Specify JSON configuration filename prefix: yc.

  2. Download the arcsight_content files.

  3. Copy the yc.jsonparser.properties file from the flex folder to the <agent_installation_folder>/current/user/agent/flexagent folder.

  4. Copy the map.0.properties file from the flex folder to the <agent_installation_folder>/current/user/agent/map folder.

  5. Edit the <agent_installation_folder>/current/user/agent.properties file:

    agents[0].mode=DeleteFile
agents[0].proccessfoldersrecursively=true

  6. Start the connector and make sure that events are received by ArcSight:

    image

How to delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  1. Delete the Object Storage bucket.
  2. Delete the Key Management Service key.
  3. Delete the intermediate VM if you created it in Compute Cloud.
Previous
Exporting audit logs to SIEM Splunk systems
Next
Overview