Transferring logs through Unified Agent HTTP input to Cloud Logging

Yandex Unified Agent allows you to receive and send user application logs to Yandex Cloud Logging.

In this tutorial, you will configure log transfer from a test Python application. The application will send logs to Unified Agent http input. Unified Agent will send the received logs via the yc_logs output to the Cloud Logging default log group.

To set up log transfer:

1. Prepare your cloud.
2. Install and configure Yandex Unified Agent.
3. Create and run a log-generating application.
4. View the logs.

If you no longer need the resources you created, delete them.

Prepare your cloudPrepare your cloud

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

1. Fee for continuously running VMs (see Yandex Compute Cloud pricing).

2. Fee for logging operations and log storage in a log group (see Yandex Cloud Logging pricing).

Prepare the infrastructurePrepare the infrastructure

Create a service accountCreate a service account

1. Create a service account named sa-logger in the folder you want to write logs to.
2. Assign the logging.writer role to the service account.

Create a VMCreate a VM

1. Create a VM from a public Ubuntu 24.04 image.

   Under Access, select the sa-logger service account.

2. Connect to the VM over SSH.

Install and configure Unified AgentInstall and configure Unified Agent

1. Download the latest deb package:

   ubuntu_name="ubuntu-22.04-jammy" ua_version=$(curl --silent https://storage.yandexcloud.net/yc-unified-agent/latest-version) bash -c 'curl --silent --remote-name https://storage.yandexcloud.net/yc-unified-agent/releases/${ua_version}/deb/${ubuntu_name}/yandex-unified-agent_${ua_version}_amd64.deb'
   

2. Check the deb package version using the ls command.

3. Install Unified Agent from the deb package by specifying its version:

   sudo dpkg -i yandex-unified-agent_24.09.03_amd64.deb
   

   Other installation methods are described in Installing and running Yandex Unified Agent.

4. Check that Unified Agent is running:

   sudo systemctl status unified-agent.service
   

5. Open the Unified Agent configuration file:

   sudo nano /etc/yandex/unified_agent/config.yml
   

6. Add configuration to the file to receive and send logs:

   status:
     port: "16241"
   
   routes:
      - input:
         plugin: http
         config:
            port: 22132
        channel:
         output:
            plugin: yc_logs
            config:
               folder_id: "b1grj7grr1kn********"
               iam:
                  cloud_meta: {}
   
   import:
   - /etc/yandex/unified_agent/conf.d/*.yml
   

   Where folder_id is the ID of the folder you want to write logs to.

7. Make sure the configuration file is correct (the command should output the contents of the file):

   unified_agent check-config -c /etc/yandex/unified_agent/config.yml
   

8. Restart Unified Agent:

   sudo systemctl restart unified-agent.service
   

9. Check the Unified Agent status:

   sudo systemctl status unified-agent.service
   

Create and run a log-generating applicationCreate and run a log-generating application

1. Create a file named logtest.py:

   import time
   import requests
   import random
   
   # Possible URLs for requests
   urls = [
      '/',
      '/admin',
      '/hello',
      '/docs',
      '/api/resource1',
      '/api/resource2',
      '/api/resource3'
   ]
   
   # Unified Agent HTTP input configuration
   unified_agent_url = 'http://51.250.98.18:22132/write'
   
   # Possible response codes and their probabilities
   response_codes = [200, 201, 400, 404, 500]
   response_weights = [0.7, 0.1, 0.1, 0.05, 0.05]
   
   # Generating and sending logs to HTTP input
   def generate_and_send_logs():
      while True:
         url = random.choice(urls)
         status_code = random.choices(response_codes, response_weights)[0]
         log_message = f"Requested {url}, received status code {status_code}"
         print(log_message)
         
         # Sending log to Unified Agent HTTP input
         send_logs_to_http(log_message)
         
         # Sending logs every 5 seconds
         time.sleep(5)
   
   # Sending logs to HTTP input
   def send_logs_to_http(log_message):
      headers = {"Content-Type": "text/plain"}
      response = requests.post(unified_agent_url, headers=headers, data=log_message)
      if response.status_code == 200:
         print("Log sent successfully")
      else:
         print(f"Failed to send log: {response.status_code}")
   
   if __name__ == "__main__":
      generate_and_send_logs()
   

   Where unified_agent_url is the VM's public IP address with Unified Agent.

   By default, Unified Agent accepts data on all interfaces. Therefore, you can specify a public IP address even if the log source is on the same VM. If there is no public address, put localhost.

2. Upgrade the versions of installed packages:

   sudo apt-get update
   

3. Install Python:

   sudo apt install python3
   

4. Run the application:

   python3 logtest.py
   

View the logsView the logs

yc logging read --folder-id=<folder_ID>

Delete the resources you createdDelete the resources you created

If you no longer need the resources you created, delete them:

1. Delete the VM.
2. Delete the log group.