Query language

This section describes the query language for log selection and filtering. The Monium interface and API use a unified query language to obtain data. This language allows applying filter expressions to every log line. Same as in SQL, you can use conditional expressions in the WHERE part.

Data typesData types

The data type is determined by the literal form of the expression. The query language supports primitive data types:

• String: Any literal expression in single or double quotes will be represented as a string.

• Scalar: An IEEE 754 standard-compliant real double-precision floating point number, not including the special NaN and infinity values.

  The number type supports scientific notation with the fraction and power of ten and the following suffixes:

  • k: 10³
  • M: 10⁶
  • G: 10⁹
  • T: 10¹²
  • P: 10¹⁵
  • E: 10¹⁸

  Here is an example:

  101
  75.3
  20G
  1E-3
  

Log selection and filtering languageLog selection and filtering language

The query language is used to describe log filters and is similar to the monitoring selector language. You can select multiple logs using a set of selectors that filter label values. In this context, top-level fields, e.g., message and level, are treated as labels, same as in labels and meta.
Selector consists of a label name, a statement, and an expression that describes a set of label values. Conditions inside a selector are joined with AND.

Here is an example:

{project="folder__b1g86q4m5vej********", service="fetcher", cluster="production", duration > 500}

Operators:Operators:

For the >, >=, <, and <= operators, the right-hand operand is always expected to be a numeric literal (except for the level label), and the comparison is performed by casting the label value to a number. If the label value cannot be cast, the result of that part of the expression is considered false.

To search by labels stored in the meta metadata, you must specify a fully qualified name in meta.label_name format. You can use all the same operators as are used for top-level labels and labels.

Below are some examples of selectors based on the following data:

....
project="folder__b1g86q4m5vej********"
service="fetcher"
cluser="production"
level=ERROR
labels={hook_type="LIVENESS", object_type="WORKLOAD", start_time=1708348144767787, fail_reason=2000, state="EHttpGetState_SUCCESS"}
meta={user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 YaBrowser/24.1.0.0 Safari/537.36"}

Selector that filters log lines with the INFO or DEBUG logging level:

{project="folder__b1g86q4m5vej********", service="fetcher", cluser="production", level="INFO|DEBUG"}

Selector that filters log lines whose fail_reason label (one of the keys in the labels field) contains an integer larger than 1,000:

{project="folder__b1g86q4m5vej********", service="fetcher", cluser="production", fail_reason>1000}

Searching for all errors that occurred when queried by a particular agent:

{project="folder__b1g86q4m5vej********", service="fetcher", cluser="production", level>="ERROR", fail_reason=200, meta.user_agent=*"Chrome"}

Label names can contain the . characters; they are treated as part of a single label’s name. Such characters are not escaped in the selector:

{exception.type="java.lang.RuntimeException"}

Or like this for meta:

{meta.exception.message="cannot replace metrics*"}

Minimum required selectorMinimum required selector

For a selector, there is a minimum set of required labels that must be there in each query. This effectively represents the maximum search area for a single selector. The required labels are either project + service or log_group_id.

Minimum selector limitations:

Label limitationsLabel limitations

Some labels are subject to limitations related to operators and their values:

Working with trace.idWorking with trace.id

The query language allows omitting any number of leading zeroes in the trace.id value.
Therefore, the following selectors are equivalent:

• {trace.id="000000000000000036a91077c9806b4c"}
• {trace.id="000036a91077c9806b4c"}
• {trace.id="36a91077c9806b4c"}

Recommendations for optimizing selector performanceRecommendations for optimizing selector performance

For sparse data (learn more), optimizations make it possible to search only through a portion of the storage, so the more accurately your query narrows down the selection, the faster it is. Here is an example:

• {project = "...", service = "...", meta.order.id = "*abc*123*"}
  The query does not specify exact attribute values, so the system will have to search everywhere.

• {project, service, meta.order.id = "bf090941d4f44867"}
  With the exact value, we can search fast (as long as the ID is rare).

• {project, service, meta.order.id = "*abc*123*", meta.request.id = "c7438471-fc0f46b9bc68"}
  Here, we cannot search fast by order.id, but we can by request.id!

• {project, service, geo.city = "very_small_city"}
  This also works for labels.

• {project, service, geo.city = "very_small_city", meta.purchase_token = "f97b68f2450e43e4"}
  This query is even faster than the previous one because first we screened out some results by geo.city, and then some more by purchase_token.

• {project, service, trace.id="00008dsf7d8df7d"}
  It also works for trace.id and span.id.

It is recommended to write high-cardinality values (including IDs) into metadata.