Sending logs via Unified Agent

The data collection agent enables you to separate log generation (collection) from log delivery to the storage system. This approach makes the logging system more reliable and easier to manage. For example, a properly configured agent ensures logs are not lost if your application crashes or shuts down, or if the log sink is temporarily offline: the agent keeps the logs and delivers them once the connection is restored.

Monium.Logs currently only accepts OpenTelemetry data. To send your app's logs in this format, use the SDK for the programming language in question. We recommend following the standard naming conventions also provided in the SDK.

Unified Agent is a versatile telemetry collection agent by Yandex that works with metrics, traces, and logs. It supports authentication via the TVM protocol and integrates with various log sources, including the file system, logrotate-compatible sources, and HTTP/GRPC interfaces. You can deploy Unified Agent directly to the host system, including a real-time container (RTC) environment.

This section provides common examples of how to configure Unified Agent to send logs to Monium.Logs. For more information on setting up an agent, see this Unified Agent guide.

Sending logs from an applicationSending logs from an application

routes:
    - input:
          plugin: otel_logs
          config:
              port: 8889

      channel:
          output:
              plugin: otel_logs
              config:
                  project: folder__<folder_ID>
                  cluster: doc

                  format: proto
                  url: "ingest.monium.yandex.cloud:443"

                  tvm:
                      client_id: 2025442
                      destination_id: 2045111
                      secret:
                          file: /my/secret/token

In this configuration, the agent expects logs in OpenTelemetry format at localhost:8889. The collected logs then go to the collector to be saved in the folder__<folder_ID> project.

A TVM is used for authentication. The secret section specifies the path to the file with the API key for obtaining the TVM token, and the client_id section specifies the ID of the service account with the yc.logging.write role.

Collecting logs from a fileCollecting logs from a file

routes:
  - input:
      plugin: file_input
      config:
        path: /logs/fetcher.log
        state_directory: <path_to_directory>

    channel:
      pipe:
        - filter:
            plugin: subprocess
            config:
              command: <path_to_parser>
              arguments:
                - "--project=examples"
                - "--cluster=doc"
                - "--service=example"
      output:
        plugin: otel_logs
        config:
          project: folder__<folder_ID>
          cluster: doc
          service: example

          format: json
          url: "ingest.monium.yandex.cloud:443"

          tvm:
            client_id: 2025442
            destination_id: 2045111
            secret:
              file: <path_to_secret_file>

Where:

• <path_to_folder>: Folder with agent state files, e.g., /var/lib/ua/state.
• <path_to_parser>: Full path to the parser executable file. The parser accepts log strings as input, converts them to the required format, e.g., protobuf or JSON, and outputs the result. For example, /usr/local/bin/cpp_processes.
• <path_to_secret_file>: File with the TVM secret, e.g., /etc/ua/tvm_secret.

In this configuration, the agent reads entries from the /logs/fetcher.log file. The state_directory parameter defines the folder for storing service status files to ensure logs are read correctly after the agent restarts.

The read logs are processed with an external parser, which converts the logs into OpenTelemetry format.

The final data goes to the folder__<folder_ID> project in Monium at the address specified under url.

A TVM is used for authentication. The secret section specifies the path to the file with the API key for obtaining the TVM token, and the client_id section specifies the ID of the service account with the yc.logging.write role.