Transferring logs through Unified Agent HTTP input to Cloud Logging
Yandex Unified Agent allows you to receive and send user application logs to Yandex Cloud Logging.
In this tutorial, you will configure log transfer from a test Python application. The application will send logs to Unified Agent http input. Unified Agent will send the received logs via the yc_logs output to the Cloud Logging default
log group.
To set up log transfer:
- Prepare your cloud.
- Install and configure Yandex Unified Agent.
- Create and run a log-generating application.
- View the logs.
If you no longer need the resources you created, delete them.
Prepare your cloud
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Required paid resources
-
Fee for continuously running VMs (see Yandex Compute Cloud pricing).
-
Fee for logging operations and log storage in a log group (see Yandex Cloud Logging pricing).
Prepare the infrastructure
Create a service account
- Create a service account named
sa-logger
in the folder you want to write logs to. - Assign the
logging.writer
role to the service account.
Create a VM
-
Create a VM from a public Ubuntu 24.04 image.
Under Access, select the
sa-logger
service account. -
Connect to the VM over SSH.
Install and configure Unified Agent
-
Download the latest deb package:
ubuntu_name="ubuntu-22.04-jammy" ua_version=$(curl --silent https://storage.yandexcloud.net/yc-unified-agent/latest-version) bash -c 'curl --silent --remote-name https://storage.yandexcloud.net/yc-unified-agent/releases/${ua_version}/deb/${ubuntu_name}/yandex-unified-agent_${ua_version}_amd64.deb'
-
Check the deb package version using the
ls
command. -
Install Unified Agent from the deb package by specifying its version:
sudo dpkg -i yandex-unified-agent_24.09.03_amd64.deb
Other installation methods are described in Installing and running Yandex Unified Agent.
-
Check that Unified Agent is running:
sudo systemctl status unified-agent.service
-
Open the Unified Agent configuration file:
sudo nano /etc/yandex/unified_agent/config.yml
-
Add configuration to the file to receive and send logs:
status: port: "16241" routes: - input: plugin: http config: port: 22132 channel: output: plugin: yc_logs config: folder_id: "b1grj7grr1kn********" iam: cloud_meta: {} import: - /etc/yandex/unified_agent/conf.d/*.yml
Where
folder_id
is the ID of the folder you want to write logs to. -
Make sure the configuration file is correct (the command should output the contents of the file):
unified_agent check-config -c /etc/yandex/unified_agent/config.yml
-
Restart Unified Agent:
sudo systemctl restart unified-agent.service
-
Check the Unified Agent status:
sudo systemctl status unified-agent.service
Create and run a log-generating application
-
Create a file named
logtest.py
:import time import requests import random # Possible URLs for requests urls = [ '/', '/admin', '/hello', '/docs', '/api/resource1', '/api/resource2', '/api/resource3' ] # Unified Agent HTTP input configuration unified_agent_url = 'http://51.250.98.18:22132/write' # Possible response codes and their probabilities response_codes = [200, 201, 400, 404, 500] response_weights = [0.7, 0.1, 0.1, 0.05, 0.05] # Generating and sending logs to HTTP input def generate_and_send_logs(): while True: url = random.choice(urls) status_code = random.choices(response_codes, response_weights)[0] log_message = f"Requested {url}, received status code {status_code}" print(log_message) # Sending log to Unified Agent HTTP input send_logs_to_http(log_message) # Sending logs every 5 seconds time.sleep(5) # Sending logs to HTTP input def send_logs_to_http(log_message): headers = {"Content-Type": "text/plain"} response = requests.post(unified_agent_url, headers=headers, data=log_message) if response.status_code == 200: print("Log sent successfully") else: print(f"Failed to send log: {response.status_code}") if __name__ == "__main__": generate_and_send_logs()
Where
unified_agent_url
is the VM's public IP address with Unified Agent.By default Unified Agent accepts data on all interfaces. Therefore, you can specify a public IP address even if the log source is on the same VM. If there is no public address, put
localhost
. -
Upgrade the versions of installed packages:
sudo apt-get update
-
Install Python:
sudo apt install python3
-
Run the application:
python3 logtest.py
View the logs
- In the management console
, go to the folder you specified in the Yandex Unified Agent settings. - Select Cloud Logging.
- Select the
default
log group. - Go to the Logs tab.
- The page that opens will show the log group records.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To view records in the log group, run this command:
yc logging read --folder-id=<folder_ID>
Where --folder-id
is the folder ID specified in the Yandex Unified Agent settings.
To view log group records, use the LogReadingService/Read gRPC API call.
Delete the resources you created
If you no longer need the resources you created, delete them: