Log metrics

To ensure log delivery is controllable and measurable, Monium. Logs publishes a set of standard metrics for each project. These metrics are available in all projects receiving logs, regardless of the delivery method.

Using these metrics, you can estimate the volume of incoming logs (by service and cluster), delivery lag, and the number of ERROR log entries (both across the entire project and per service).

The metrics are aggregated into a service dashboard named Logging — Logs overview:

• Dashboard is created automatically in a project once logs start being written to the system.
• Dashboard is available at level of the folder storing the logs.
• Metrics do not have the host or user_cluster labels.
• user_service label indicates the log group the metrics are calculated for.

Metric descriptionsMetric descriptions

ingest_processed_logsingest_processed_logs

Metric tracking logs per second successfully authorized, validated, and about to be written to the database (logs may also remain unprocessed if the user-sent volume exceeds the log quota).

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).

ingest_processed_bytesingest_processed_bytes

Metric tracking bytes in logs per second successfully authorized, validated, and about to be written to the database (logs may also remain unprocessed if the user-sent bytes exceed the byte quota).

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).

logsvalidator_invalidated_logslogsvalidator_invalidated_logs

Metric tracking logs per second that failed validation.

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).
• reason: Reason why the log failed validation.

Validation errorsValidation errors

• res.attrs.project: Project not specified.
• res.attrs.service: Service not specified.
• res.attrs.invalid: Resource-level attributes failed validation.
• scope.name: Scope name is 0 or over 200 characters long or contains non-unicode characters.
• scope.attrs.invalid: Attributes failed validation.
• log.attrs.invalid: Log line attributes failed validation.
• log.body.not.string: Body type is not text.
• severity.not.match.text: severity does not match the severity text name (severityText).
• severity.0.text.not.empty: severity = 0 with severityText specified.
• severity.not.in.range: Log severity is less than 1 (Trace) or greater than 24 (FATAL 4).
• ts.invalid: Log timestamp is more than 5 minutes ahead of the current time.
• observed.ts.invalid: Log observed timestamp is more than 5 minutes ahead of current time.
• log.component.both.levels: Component specified at both scope and log levels.

severity_logs_user_tsseverity_logs_user_ts

Metric tracking log volume broken down by log severity (Level). This metric’s timestamp derives from the log line’s timestamp and not the data write or send times.

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).
• severity: Log level (Error, Warn, or Info).

receiver_auth_processed_logsreceiver_auth_processed_logs

Metric tracking user-sent logs per second prior to all system checks (authorization, validation, or quota checks).

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).

receiver_auth_processed_logs_bytesreceiver_auth_processed_logs_bytes

Metric tracking user-sent logs in bytes per second prior to authorization, validation, and quota checks.

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).

receiver_auth_authorizationsreceiver_auth_authorizations

Metric tracking failed user authorization requests per second.

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).
• user_auth_result: Authorization result (currently, failure only).
• user_auth_kind: Authorization type.

lag_from_app_seclag_from_app_sec

Histogram tracking time between sending logs by the application and writing them.

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).
• bin: Buckets in seconds (1, 30, 50, 60, 90, 120, 300, 900, or inf).

lag_from_agent_seclag_from_agent_sec

Histogram tracking time between sending logs by the agent and writing them.

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).
• bin: Buckets in seconds (1, 5, 60, 90, 300, or inf).

project_logs_quotaproject_logs_quota

Metric tracking your project’s log line quota.

Labels:

• host: Set to quota.

project_bytes_logs_quotaproject_bytes_logs_quota

Metric tracking your project’s byte-per-second quota.

Labels:

• host: Set to quota.

ingest_dropped_labels_countingest_dropped_labels_count

Metric tracking log lines per second in which the attribute prefixed with labels. was either written to metadata or discarded.

Labels:

• user_cluster: Cluster the logs came from (use total to view the total across all clusters).
• user_service: Service the logs came from (use total to view the total across all services).
• host: Host the logs came from (use cluster to calculate the total across all data centers or select a specific data center).