Uploading Yandex Audit Trails audit logs to KUMA SIEM through the management console, CLI, or API
To configure delivery of audit log files to KUMA
- Get your cloud ready.
- Set up your environment.
- Create a bucket.
- Create a trail.
- Create a server.
- Mount the bucket on a server.
- Configure the KUMA collector.
If you no longer need the resources you created, delete them.
Get your cloud ready
Sign up for Yandex Cloud and create a billing account:
- Navigate to the management console
and log in to Yandex Cloud or create a new account. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVEorTRIAL_ACTIVEstatus. If you do not have a billing account, create one and link a cloud to it.
If you have an active billing account, you can navigate to the cloud page
Learn more about clouds and folders here.
Required paid resources
The cost of support for a new Yandex Cloud infrastructure includes:
- Fee for data storage, operations with data, and outgoing traffic (see Yandex Object Storage pricing).
- Fee for a symmetric encryption key and cryptographic operations (see Yandex Key Management Service pricing).
- (Optional) Fee for a continuously running VM (see Yandex Compute Cloud pricing).
- (Optional) Fee for using a dynamic or static external IP address (see Yandex Virtual Private Cloud pricing).
In addition, to complete the tutorial, you will need a KUMA user license
Set up your environment
Create service accounts
For your new infrastructure to run properly, create two service accounts as follows:
kuma-bucket-sa: For the Object Storage bucket.kuma-trail-sa: For the Audit Trails trail.
- In the management console
, go to the folder you want to create an infrastructure in. - In the list of services, select Identity and Access Management.
- Click Create service account.
- Enter a name of the service account for the bucket:
kuma-bucket-sa. - Click Create.
- Repeat steps 3-5 to create the
kuma-trail-saservice account for the trail.
If you do not have the Yandex Cloud CLI installed yet, install and initialize it.
By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.
-
Run the following commands:
yc iam service-account create --name kuma-bucket-sa yc iam service-account create --name kuma-trail-saWhere
--namerepresents the names of the service accounts.Result:
id: ajecikmc374i******** folder_id: b1g681qpemb4******** created_at: "2024-11-28T14:11:42.593107676Z" name: kuma-bucket-sa id: ajedc6uq5o7m******** folder_id: b1g681qpemb4******** created_at: "2024-11-28T14:11:45.856807266Z" name: kuma-trail-sa -
Save the new service accounts' IDs (
id): you will need them in the next steps.
For more information about the yc iam service-account create command, see the CLI reference.
To create a service account, use the create REST API method for the ServiceAccount resource or the ServiceAccountService/Create gRPC API call.
Create a static access key
To mount a bucket on a server with a KUMA collector installed, create a static access key for the kuma-bucket-sa service account.
-
In the management console
, go to the folder you want to create an infrastructure in. -
In the list of services, select Identity and Access Management.
-
In the left-hand panel, select
Service accounts. -
Select the
kuma-bucket-saservice account. -
In the top panel, click
Create new key and select Create static access key. -
Enter a description for the key and click Create.
-
Save the ID and secret key: you will need them later when mounting the bucket on the server.
Alert
After you close this dialog, the key value will not be shown again.
-
Run this command:
yc iam access-key create --service-account-name kuma-bucket-saWhere
--service-account-nameis the name of the service account you are creating the key for.Result:
access_key: id: aje726ab18go******** service_account_id: ajecikmc374i******** created_at: "2024-11-28T14:16:44.936656476Z" key_id: YCAJEOmgIxyYa54LY******** secret: YCMiEYFqczmjJQ2XCHMOenrp1s1-yva1******** -
Save the ID (
key_id) and secret key (secret): you will need them later when mounting the bucket on the server.
For more information about the yc iam access-key create command, see the CLI reference.
To create a static access key, use the create REST API method for the AccessKey resource or the AccessKeyService/Create gRPC API call.
Create an encryption key
Create a symmetric encryption key for encryption of audit logs in the bucket.
-
In the management console
, go to the folder you want to create an infrastructure in. -
In the list of services, select Key Management Service.
-
In the left-hand panel, select
Symmetric keys. -
Click Create key and specify its attributes:
- Name:
kuma-key. - Encryption algorithm:
AES-256.
- Name:
-
Click Create.
-
Run this command:
yc kms symmetric-key create \ --name kuma-key \ --default-algorithm aes-256Where:
--name: Key name.--default-algorithm: Encryption algorithm.
Result:
id: abje8mf3ala0******** folder_id: b1g681qpemb4******** created_at: "2024-11-28T14:22:06Z" name: kuma-key status: ACTIVE primary_version: id: abjuqbth02kf******** key_id: abje8mf3ala0******** status: ACTIVE algorithm: AES_256 created_at: "2024-11-28T14:22:06Z" primary: true default_algorithm: AES_256 -
Save the symmetric key ID (
id): you will need it later when creating the bucket.
For more information about the yc kms symmetric-key create command, see the CLI reference.
To create a symmetric encryption key, use the create REST API method for the SymmetricKey resource or the SymmetricKeyService/Create gRPC API call.
Assign roles to the service accounts
Assign to the service accounts the following roles for the folder and the encryption key created earlier:
-
To
kuma-trail-sa:audit-trails.viewerfor the folder.storage.uploaderfor the folder.kms.keys.encrypterDecrypterfor the encryption key.
-
To
kuma-bucket-sa:storage.viewerfor the folder.kms.keys.encrypterDecrypterfor the encryption key.
-
Assign roles for the folder:
-
In the management console
, go to the folder you want to create an infrastructure in. -
Navigate to the Access bindings tab.
-
Click Configure access.
-
In the window that opens, select Service accounts.
-
Select the
kuma-trail-saservice account from the list, use the search if required. -
Click
Add role; in the window that opens, select theaudit-trails.viewerrole.Repeat this step and add the
storage.uploaderrole. -
Click Save.
In the same way, assign the
storage.viewerrole for the folder to thekuma-bucket-saservice account. -
-
Assign roles for the encryption key:
- In the list of services, select Key Management Service.
- In the left-hand panel, select
Symmetric keys and click on the line withkuma-key. - Go to
Access bindings and click Assign roles. - Select the
kuma-trail-saservice account. - Click
Add role and select thekms.keys.encrypterDecrypterrole. - Click Save.
In the same way, assign the
kms.keys.encrypterDecrypterrole for the encryption key to thekuma-bucket-saservice account.
-
Assign the
storage.viewerrole for the folder tokuma-bucket-sa:yc resource-manager folder add-access-binding <folder_name_or_ID> \ --role storage.viewer \ --subject serviceAccount:<kuma-bucket-sa_ID>Where:
<folder_name_or_ID>: Name or ID of the folder the role is assigned for.--role: Role ID.--subject: Subject type and ID of the service account you are assigning the role to.
Result:
effective_deltas: - action: ADD access_binding: role_id: storage.viewer subject: id: ajecikmc374i******** type: serviceAccountIn the same way, assign the
audit-trails.viewerandstorage.uploaderroles for the folder tokuma-trail-sa.For more information about the
yc resource-manager folder add-access-bindingcommand, see the CLI reference. -
Assign the
kms.keys.encrypterDecrypterrole for the encryption key tokuma-bucket-sa.yc kms symmetric-key add-access-binding kuma-key \ --role kms.keys.encrypterDecrypter \ --subject serviceAccount:<kuma-bucket-sa_ID>Where:
--role: Role ID.--subject: Subject type and ID of the service account you are assigning the role to.
Result:
...1s...done (4s)In the same way, assign the
kms.keys.encrypterDecrypterrole for the encryption key tokuma-trail-sa.For more information about the
yc kms symmetric-key add-access-bindingcommand, see the CLI reference.
To assign a role to a service account, use the setAccessBindings REST API method for the ServiceAccount resource or the ServiceAccountService/SetAccessBindings gRPC API call.
Create a bucket
Create a bucket for the trail to save audit logs to and enable encryption.
-
In the management console
, go to the folder you want to create an infrastructure in. -
In the list of services, select Object Storage.
-
At the top right, click Create bucket.
-
In the ** Name** field, enter a name for the bucket, e.g.,
my-audit-logs-for-kuma.Note
The bucket name must be unique across Object Storage. You cannot create two buckets with the same name – even in different folders of different clouds.
-
In the Max size field, set the size of the bucket you are creating or enable No limit.
-
Leave all other parameters as they are and click Create bucket.
-
On the page with a list of buckets that opens, select the new bucket.
-
In the left-hand menu, select
Securityand go to the Encryption tab. -
In the KMS Key field, select the previously created
kuma-key. -
Click Save.
-
Create a bucket:
yc storage bucket create --name <bucket_name>Where
--nameis the bucket name, e.g.,my-audit-logs-for-kuma.Note
The bucket name must be unique across Object Storage. You cannot create two buckets with the same name – even in different folders of different clouds.
Result:
name: my-audit-logs-for-kuma folder_id: b1g681qpemb4******** anonymous_access_flags: read: false list: false default_storage_class: STANDARD versioning: VERSIONING_DISABLED acl: {} created_at: "2024-11-28T15:01:20.816656Z"For more information about the
yc storage bucket createcommand, see the CLI reference. -
Enable encryption for the new bucket:
yc storage bucket update \ --name <bucket_name> \ --encryption key-id=<symmetric_key_ID>Where:
--name: Bucket name.--encryption: Symmetric key ID you got when creating the key.
Result:
name: my-audit-logs-for-kuma folder_id: b1g681qpemb4******** default_storage_class: STANDARD versioning: VERSIONING_DISABLED acl: {} created_at: "2024-11-28T15:01:20.816656Z"For more information about the
yc storage bucket updatecommand, see the CLI reference.
To create a bucket, use the create REST API method for the Bucket resource, the BucketService/Create gRPC API call, or the create S3 API method.
Create a trail
Create a trail to collect and deliver audit logs.
-
In the management console
, go to the folder you want to create an infrastructure in. -
Select Audit Trails.
-
Click Create trail and do the following in the window that opens:
-
In the Name field, enter a name for the trail:
kuma-trail. -
Under Destination, configure the destination object:
- Destination:
Object Storage. - Bucket: Bucket you created earlier, e.g.,
my-audit-logs-for-kuma. - Object prefix: Optional parameter used in the full name of the audit log file.
Note
Use a prefix to store audit logs and third-party data in the same bucket. Do not use the same prefix for logs and other bucket objects because that may cause logs and third-party objects to overwrite each other.
- Destination:
-
Make sure the Encryption key field contains the encryption key named
kuma-key. If the encryption key is not set, click Add and select this key. -
Under Collecting management events, configure the collection of management event audit logs:
- Collecting events: Select
Enabled. - Resource: Select
Folder. - Folder: Automatically populated field containing the name of the current folder.
- Collecting events: Select
-
Under Service account above, select the
kuma-trail-saservice account. -
Under Collecting data events, keep the
Disabledvalue. -
Click Create.
-
Run this command:
yc audit-trails trail create \
--name kuma-trail \
--destination-bucket <bucket_name> \
--destination-bucket-object-prefix <prefix> \
--service-account-id <kuma-trail-sa_ID> \
--filter-from-cloud-id <cloud_ID> \
--filter-some-folder-ids <folder_ID>
Where:
--name: Trail name.--destination-bucket: Name of the bucket you created earlier to upload audit logs to.--destination-bucket-object-prefix: Prefix that will be added to the names of the audit log objects in the bucket. It is an optional parameter used in the full name of the audit log file.--service-account-id: Thekuma-trail-saservice account's ID you got earlier. Your trail will use this account to upload audit log files to the bucket.--filter-from-cloud-id: ID of the cloud whose resources the trail will collect audit logs for.--filter-some-folder-ids: ID of the folder whose resources the trail will collect audit logs for.
Result:
id: cnpabi372eer********
folder_id: b1g681qpemb4********
created_at: "2024-11-28T15:33:28.057Z"
updated_at: "2024-11-28T15:33:28.057Z"
name: kuma-trail
destination:
object_storage:
bucket_id: my-audit-logs-for-kuma
object_prefix: kuma
service_account_id: ajedc6uq5o7m********
status: ACTIVE
cloud_id: b1gia87mbaom********
filtering_policy:
management_events_filter:
resource_scopes:
- id: b1g681qpemb4********
type: resource-manager.folder
For more information about the yc audit-trails trail create command, see the CLI reference.
To create a trail, use the create REST API method for the Trail resource or the TrailService/Create gRPC API call.
Creating a server
As a server to install the KUMA collector on, you can use a Compute Cloud VM or your own hardware. In this tutorial, we are using a Compute Cloud VM residing in a Yandex Virtual Private Cloud cloud network.
Create a network with a subnet
- In the management console
, go to the folder you want to create an infrastructure in. - In the list of services, select Virtual Private Cloud.
- Click Create network.
- Specify the network name, e.g.,
kuma-network. - Make sure the Create subnets option is enabled.
- Click Create network.
-
Create a cloud network:
yc vpc network create --name kuma-networkWhere
--nameis the network name.Result:
id: enpnmb4jvubr******** folder_id: b1g681qpemb4******** created_at: "2024-11-27T22:55:55Z" name: kuma-network default_security_group_id: enpjgspepn8k********For more information about the
yc vpc network createcommand, see the CLI reference. -
Create a subnet:
yc vpc subnet create \ --name kuma-network-ru-central1-b \ --network-name kuma-network \ --zone ru-central1-b \ --range 10.1.0.0/24Where:
--name: Subnet name.--network-name: Name of the network the subnet is created in.--zone: The subnet's availability zone.--range: Subnet CIDR .
Result:
id: e2l7b3gpnhqn******** folder_id: b1g681qpemb4******** created_at: "2024-11-27T22:57:48Z" name: kuma-network-ru-central1-b network_id: enpnmb4jvubr******** zone_id: ru-central1-b v4_cidr_blocks: - 10.1.0.0/24For more information about the
yc vpc subnet createcommand, see the CLI reference.
-
To create a cloud network, use the create REST API method for the Network resource or the NetworkService/Create gRPC API call.
-
To create a subnet, use the create REST API method for the Subnet resource or the SubnetService/Create gRPC API call.
Create a VM
-
In the management console
, select the folder to create the infrastructure in. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Virtual machines. -
Click Create virtual machine.
-
Select Advanced setup.
-
Under Boot disk image, select the Ubuntu 22.04 LTS image.
-
Under Location, select the
ru-central1-bavailability zone. -
Under Network settings:
- In the Subnet field, select
kuma-network-ru-central1-b. - In the Public IP address field, select
Autoto give the VM a random external IP address from the Yandex Cloud pool or select a static IP address from the list if you reserved one in advance.
- In the Subnet field, select
-
Under Access, select SSH key and specify the VM access credentials:
-
In the Login field, enter a username for the user you are going to create on the VM, e.g.,
yc-user.- The name must be from 3 to 63 characters long.
- It may contain uppercase and lowercase Latin and Cyrillic letters, numbers, hyphens, underscores, and spaces.
- The first character must be a letter. The last character cannot be a hyphen, underscore, or space.
Alert
Do not use
rootor other reserved usernames. To perform operations requiring root privileges, use thesudocommand. -
In the SSH key field, select the SSH key saved in your organization user profile.
If there are no SSH keys in your profile or you want to add a new key:
-
Click Add key.
-
Enter a name for the SSH key.
-
Select one of the following:
-
Enter manually: Paste the contents of the public SSH key. You need to create an SSH key pair on your own. -
Load from file: Upload the public part of the SSH key. You need to create an SSH key pair on your own. -
Generate key: Automatically create an SSH key pair.When adding a new SSH key, an archive containing the key pair will be created and downloaded. In Linux or macOS-based operating systems, unpack the archive to the
/home/<user_name>/.sshdirectory. In Windows, unpack the archive to theC:\Users\<user_name>/.sshdirectory. You do not need additionally enter the public key in the management console.
-
-
Click Add.
The system will add the SSH key to your organization user profile. If the organization has disabled the ability for users to add SSH keys to their profiles, the added public SSH key will only be saved in the user profile inside the newly created resource.
-
-
-
Under General information, specify the VM name:
kuma-server. -
Click Create VM.
Run this command:
yc compute instance create \
--name kuma-server \
--zone ru-central1-b \
--network-interface subnet-name=kuma-network-ru-central1-b,nat-ip-version=ipv4 \
--create-boot-disk image-folder-id=standard-images,image-id=fd8ulbhv5dpakf3io1mf \
--ssh-key <SSH_key>
Where:
-
--name: VM name. -
--zone: Availability zone corresponding to thekuma-network-ru-central1-bsubnet. -
--network-interface: Network settings:subnet-name: Subnet name.nat-ip-version: Public IP address.
-
--create-boot-disk: Boot disk settings, whereimage-idis the Ubuntu 22.04 LTS public image ID. -
--ssh-key: Path to the public SSH key file and its name, e.g.,~/.ssh/id_ed25519.pub. You need to create a key pair for the SSH connection to a VM on your own.When the VM is created, a user named
yc-userwill be created in its operating system; use this username to connect to the VM over SSH.
Result:
id: epd4vr5ra728********
folder_id: b1g681qpemb4********
created_at: "2024-11-27T23:00:38Z"
name: kuma-server
zone_id: ru-central1-b
platform_id: standard-v2
resources:
memory: "2147483648"
cores: "2"
core_fraction: "100"
status: RUNNING
metadata_options:
gce_http_endpoint: ENABLED
aws_v1_http_endpoint: ENABLED
gce_http_token: ENABLED
aws_v1_http_token: DISABLED
boot_disk:
mode: READ_WRITE
device_name: epdk5emph7a4********
auto_delete: true
disk_id: epdk5emph7a4********
network_interfaces:
- index: "0"
mac_address: d0:0d:4f:ec:bb:51
subnet_id: e2l7b3gpnhqn********
primary_v4_address:
address: 10.1.0.4
one_to_one_nat:
address: 84.2**.***.***
ip_version: IPV4
serial_port_settings:
ssh_authorization: OS_LOGIN
gpu_settings: {}
fqdn: epd4vr5ra728********.auto.internal
scheduling_policy: {}
network_settings:
type: STANDARD
placement_policy: {}
hardware_generation:
legacy_features:
pci_topology: PCI_TOPOLOGY_V1
For more information about the yc compute instance create command, see the CLI reference.
To create a VM, use the create REST API method for the Instance resource or the InstanceService/Create gRPC API call.
Mount the bucket on a server
-
Connect to the server over SSH.
-
Create a new user named
kuma:sudo useradd kuma -
Create the
kumauser's home directory:sudo mkdir /home/kuma -
Create a file with a static access key and grant permissions for it to the
kumauser:sudo bash -c 'echo <access_key_ID>:<secret_access_key> > /home/kuma/.passwd-s3fs' sudo chmod 600 /home/kuma/.passwd-s3fs sudo chown -R kuma:kuma /home/kumaWhere
<access_key_ID>and<secret_access_key>are the previously saved values of the static access key of thekuma-bucket-saservice account. -
Install the s3fs
package:sudo apt install s3fs -
Create a directory that will serve as a mount point for the bucket and grant permissions for it to the
kumauser:sudo mkdir /var/log/yandex-cloud/ sudo chown kuma:kuma /var/log/yandex-cloud/ -
Mount the bucket you created earlier by specifying its name:
sudo s3fs <bucket_name> /var/log/yandex-cloud \ -o passwd_file=/home/kuma/.passwd-s3fs \ -o url=https://storage.yandexcloud.net \ -o use_path_request_style \ -o uid=$(id -u kuma) \ -o gid=$(id -g kuma)You can configure automatic mounting of the bucket at operating system start-up by opening the
/etc/fstabfile (sudo nano /etc/fstabcommand) and adding the following line to it:s3fs#<bucket_name> /var/log/yandex-cloud fuse _netdev,uid=<kuma_uid>,gid=<kuma_gid>,use_path_request_style,url=https://storage.yandexcloud.net,passwd_file=/home/kuma/.passwd-s3fs 0 0Where:
-
<bucket_name>: Name of the bucket you created earlier, e.g.,my-audit-logs-for-kuma. -
<kuma_uid>:kumauser ID in the VM operating system. -
<kuma_gid>:kumauser group ID in the VM operating system.To learn
<kuma_uid>and<kuma_gid>, run theid kumacommand in the terminal.
-
-
Make certain that the bucket is mounted:
sudo ls /var/log/yandex-cloud/If everything is configured correctly, the command will return the current contents of the audit event bucket.
The Yandex Cloud event transfer setup is complete. The events will reside in JSON
/var/log/yandex-cloud/{audit_trail_id}/{year}/{month}/{day}/*.json
Configure the KUMA collector
For this step, you will need the distribution and license files included with KUMA. Use them to install and configure the collector in the KUMA network infrastructure. For more information, see this guide
Once the setup is successfully completed, audit events will start being delivered to KUMA. The KUMA web interface allows you to search for related events
How to delete the resources you created
To stop paying for the resources you created:
- Delete the VM.
- Delete the static public IP address if you reserved one.
- Delete the subnet.
- Delete the network.
- Delete the trail.
- Delete all objects in the bucket, then delete the bucket itself.
- Delete the KMS encryption key.