Calculating log-based metrics

Log-based metrics means continuous calculation of statistics based on log data to create metrics in Monium.

Here are some examples of metrics you can calculate based on logs:

• Number of logs with an error message in the message field.
• Number of unique error recipients, grouped using the API method.
• Maximum query execution time.

Getting started with log-based metricsGetting started with log-based metrics

Address for logging via the API: ingest.monium.yandex.cloud:443.

Examples of log-based metricsExamples of log-based metrics

Below are some examples of metrics you can set up based on logs: The project, service, and cluster values are for illustration only. Use your environment values for your metrics.

Number of eventsNumber of events

Calculation of the API query frequency:

• Log selector: {project = "logging", service = "query", cluster = "production", message = "*started"}
• Aggregate function: count
• Aggregation attribute: —
• Group by: —
• Evaluation window: 1 minute
• Output metric: {project=logging, cluster=production, service=logging_aggregates, name=query_got_reqs_1m}

Amount of logs read in bytesAmount of logs read in bytes

Bytes read by the service in response to user queries in five minutes:

• Log selector: {project='logging', service='query', cluster='production', meta.ch.profile.bytes='*'}
• Aggregate function: sum
• Aggregation attribute: meta.ch.profile.bytes
• Group by: —
• Evaluation window: 5 minutes
• Output metric: {project=logging, cluster=production, service=logging_aggregates, name=query_bytes_read_5m}

Number of unique users with a breakdown by the API methodNumber of unique users with a breakdown by the API method

How many unique users have used specific API methods in one minute:

• Log selector: {project='logging', service='query', cluster='production', origin.login='*', func='AutocompleteKeys|AutocompleteValues|SearchLogs'}
• Aggregate function: unique
• Aggregation attribute: origin.login
• Group by: func
• Evaluation window: 1 minute
• Output metric: {project=logging, cluster=production, service=logging_aggregates, name=query_unique_users_by_func_1m}

How it worksHow it works

Log-based metric parametersLog-based metric parameters

• Log selector: Logs to select.
• Aggregate function: Function that calculates the metric value: count, min, max, avg, sum, unique, or uniqueNewfound.
• Aggregation attribute for min, max, avg, sum, unique, and uniqueNewfound: Name of the attribute whose value is used as a function argument.
• Group by: Attribute used for creating separate aggregates for each value.
• Evaluation window: Time range for the calculation of the aggregation function. Supported ranges: 1 minute and 5 minutes.
• Output metric: Selector to write the metric to Monium with.

Log selector (query)Log selector (query)

A log selector is a filter in query language which determines which log entries to use to calculate the metrics. To create metrics, you add to the selector environment labels (project, cluster, and service) and log entry labels (logging level, message text, meta.* value, and other attributes).

Environment labels:

• project = <project_ID>: Select the project specified in x-monium-project of the application's telemetry transmission configuration.

  This may be a cloud (cloud__<cloud_ID>) or folder (folder__<folder_ID>) project, or another project.

• cluster = <cluster_name>: Select the name of the installation that runs your application. If no cluster is specified, cluster = default is used.

• service = <service_name>: Name of your application or service. You can use the OTEL_SERVICE_NAME environment variable to provide it.

  If you do not see the required labels in the suggestions, you can enter them manually. However, the system most likely has not received data with these labels. For more on troubleshooting possible issues, see Data delivery troubleshooting.

Selector examples:

{project='my_project', service='metabase', level >= 'ERROR', message='*LEAK*', logger='io.netty.util.ResourceLeakDetector'}
{project='my_project', service='journald', cluster='production', meta.systemd_unit='alerting.service', message=*'terminating on uncaught exception;'}
{project='ci', service='api', cluster='stable', message=*'vtail.api.query.Query/SearchLogsStreaming', message=*'DEADLINE_EXCEEDED'}
{project='my_agent', service='telemetry'}

Aggregation functionAggregation function

Once logs are selected using the selector, they are used to calculate a numeric value:

For the min, max, avg, and sum functions, the attribute value is automatically converted to a floating-point number. The maximum supported accuracy for integer values is 2^53.

GroupingGrouping

You can additionally specify a grouping attribute. Its values will be used to create separate metrics. For example, if you choose to specify api_method as the grouping attribute, its logs containing the Export, Update, and Delete values, three metrics will be created instead of one:

• api_method=Export
• api_method=Update
• api_method=Delete

The grouping attribute is automatically added to the metric selector. There is no need to specify it separately in the final metric.

Calculation (aggregation) windowCalculation (aggregation) window

Logs for the calculation are selected for a fixed time period: 1 minute or 5 minutes.

For example, when counting the number of entries per minute:

Final metric (metric selector)Final metric (metric selector)

The selector must have the required labels, i.e., project, cluster, service, and the metric ID (the last one denoting the metric name in the project or specified shard). Usually, it is name, but you can also use sensor, signal, or any other label name. For more information, see Query language in Monium. Here is an example:

{project='shop', cluster='production', service='logging_aggregates', name='unique_users_5m'}

If a grouping attribute is specified, it gets added to the metric automatically. For example, for the {project='logging', cluster='production', service='logging_aggregates', name='api_errors'} metric selector and the api_method grouping attribute, the following metrics will be created:

Grouping attribute values can be used in substitutions. For example, if the grouping uses api_method, you can specify the following in the metric selector:

{project='logging', cluster='production', service='logging_aggregates', endpoint=grpc.{{api_method}}, name='api_errors'}

As a result, the api_method label is not added automatically and the value is substituted to the endpoint label:

Grouping and substitution scenariosGrouping and substitution scenarios

Automatic substitution of grouping valuesAutomatic substitution of grouping values

If grouping by the origin.login attribute and this metric selector:

{project=logging, service=logs_to_metrics, cluster=production, name=reqs_by_login}

the following label will be added to the metric selector automatically:

{project=logging, service=logs_to_metrics, cluster=production, name=reqs_by_login, origin.login={{origin.login}}

If a log entry contains origin.login=ivan_petrov, the metric will be written based on these labels:

{project=logging, service=logs_to_metrics, cluster=production, name=reqs_by_login, origin.login=ivan_petrov}

Automatic substitution of several valuesAutomatic substitution of several values

If grouping by several attributes, e.g., host and component, they will be added to the metric automatically:

{project=logging, service=logs_to_metrics, cluster=production, name=cnt, host='{{host}}', component='{{component}}'}

Manual substitution setupManual substitution setup

In addition to automatic substitution, you can set up substitution manually. The following metric selectors yield the same result:

• Log selector: {project=logging, service=query, cluster=production, host=*}
• Aggregate function: count
• Group by: host
• Output metric 1: {project=logging, service=logs_to_metrics, cluster=production, name=cnt}
• Output metric 2: {project=logging, service=logs_to_metrics, cluster=production, name=cnt, host='{{host}}'}

In such substitutions, you can replace both the key and the value. Here is an example:

{project=logging, service=logs_to_metrics, cluster=production, name=errors, host='query-{{host}}', context='query_{{host}}_{{component}}_production'}

Grouping by clusterGrouping by cluster

There are several supported options for grouping by cluster:

Option 1. Cluster as the cluster label value:

• Log selector: {project=logging, service=query}
• Aggregate function: count
• Group by: cluster
• Output metric: {project=logging, service=logs_to_metrics, cluster='{{cluster}}', name=cnt}

Option 2. Cluster as the custom user_cluster label value:

• Log selector: {project=logging, service=query}
• Aggregate function: count
• Group by: cluster
• Output metric: {project=logging, service=logs_to_metrics, cluster=production, name=cnt, user_cluster='{{cluster}}'}

Grouping by serviceGrouping by service

Grouping by service is only supported when the values are listed explicitly:

• Log selector: {project=logging, service=query|collector}
• Aggregate function: count
• Group by: service
• Output metric: {project=logging, service='{{service}}', cluster=production, name=cnt}

Quotas and limitsQuotas and limits

Each project is allocated a certain amount of CPU and RAM resources for calculation of log-based metrics. The maximum number of metrics per project is 1,000.

To increase the quota, contact support.

How to reduce resource consumptionHow to reduce resource consumption

• Use exact values in selectors. Avoid using regular and wildcard expressions as they increase the CPU load.

  Not recommended:

  {..., api_endpoint="*Export*", ...}
  

  Recommended:

  {..., api_endpoint="/api/v1/Export", ...}
  

• Move costly comparisons to the end of the selector. If you cannot avoid using wildcard expressions, put them after exact conditions.

  Not recommended:

  {..., message = "*panic*", error_code="500", ...}
  

  Recommended:

  {..., error_code="500", message = "*panic*"}
  

  This way, most of the logs will get filtered out when checking the error_code attribute.

• To reduce RAM usage, set the 5 minutes aggregation window.

Calculation specificsCalculation specifics

• The log-based metric value appears with a delay which depends on the aggregation window duration. The maximum delay is 3 minutes after the end of the window. For example, for the 1m window, the value for the 17:07:00–17:08:00 interval will be available no later than 17:11:00.
• You can get different values from raw logs and log-based metrics. For example, log search may return X entries for a selector, while the count metric may return Y for the same selector.
• The unique function returns an approximate number of unique items. Read more in HyperLogLog.
• Resent log entries are counted again; there is no deduplication.
• Numeric values of attributes are converted to floating-point numbers. The maximum supported accuracy for integer values is 2^53.

Quota exhaustionQuota exhaustion

When the quota for log-based metrics is exceeded:

• Logging continues.
• Metric values may be incomplete.
• New metrics may be created if grouping is configured for a log-based metric.

Writing to the pastWriting to the past

A metric value may be recalculated for logs written up to 30 minutes into the past.

Let’s look into the following example:

• We have set up a log-based metric for per-minute count of {project, cluster, service, message=*'failed'}.
• The current time is 17:29:17.

If Monium receives these two log entries, the following operations take place:

Metric display specificsMetric display specifics

• All metrics are written with the DGAUGE type.
• Each point on the chart shows a value over the aggregation window, not per second. For example, in a metric counting the number of API requests per minute, the point shows the number of requests per minute.
• If during the aggregation window no logs that match the selection were received, the value for that period is not sent resulting in a gap on the chart.
• When decimating metrics, specify an interval equal to the aggregation window: 1 minute or 5 minutes.
• Log-based metrics with the unique, uniqueNewfound, and avg functions are incompatible with decimation, so you should disable it when viewing large time ranges.
• Metric points for the last 30 minutes can be recalculated and resent to Monium. If metric-based aggregation is set up in the shard, this may cause values to be displayed incorrectly.

Decimation and compatibility with aggregation functionsDecimation and compatibility with aggregation functions

When you view metrics over a large time interval, Monium turns on decimation: out of several points, one is calculated using the aggregation function. For example, if a log-based metric is calculated once per minute, and the chart covers 24 hours, each point will represent an average of a set of points.

Recommendations on configuring decimation for different functions:

The avg, unique, and uniqueNewfound functions are incompatible with aggregation in Monium metrics, e.g., for unique:

• The API is alternately accessed by two users, user1 and user2, once a minute.
• The log-based metric with the unique function writes 1 once a minute.
• If viewed on a 24-hour scale, none of the decimation functions will return a correct value:
  • sum will count 60.
  • min and max will count 1.
  • avg will count 1.
  • The correct value is 2 (two unique users during the hour).