Viewing your applications' traces in Monium Traces

The tracing interface enables you to:

• View traces by ID.
• Search for spans and traces by various criteria.
• Build aggregated charts based on span attributes.

Searching and viewing tracesSearching and viewing traces

Query string operationsQuery string operations

• Strings string

  • = and !=: Equality and inequality. The value supports glob expressions with * ? |. [ ] characters are not supported.

  • =~ and !~: Regular expression in re2 format.

  • == and !==: Strict equality and strict inequality. Glob expressions and regular expressions will be interpreted as plain characters.

• duration: Applies to the values of trace.duration and span.duration.

  • Comparison operations > < >= <= are supported.
  • Follow 1s, 10ms, 150us, 2m, and 6h format, without quotes.

Searching for tracesSearching for traces

Searching for traces by parametersSearching for traces by parameters

By default, the search covers the last 24 hours, but you can change the interval as needed.

The results will include traces that have at least one span matching the search criteria.

Supported keys:

• project="value": value project spans allowing the = operator. This is a required key.
• service="value": value service spans supporting all string operators.
• cluster="value": value cluster spans supporting all string operators.
• span.name="value": Spans with the specified name, using all string operators.
• span.id="value": Spans with the specified ID, using the = and != operators.
• span.status="(OK)|(ERROR)|(UNKNOWN)": Spans with the specified status, using the = and != operators.
• span.duration>duration_string: Spans with the specified duration, using the duration operators.

All other keys will be interpreted as labels.

Query examplesQuery examples

Search for traces containing spans from the foo project:

{ project="foo" }

Search for traces containing spans from the foo project and from the bar service:

{ project="foo", service="bar" }

Search for traces containing spans with the some_process name from the foo project:

{ project="foo", span.name="some_process" }

Search for traces containing spans from the foo project and from the bar service, each lasting more than one second:

{ project="foo", service="bar", span.duration>=1s }

Search for traces containing spans from the foo project and from the bar service, each having the fizz label set to buzz:

{ project="foo", service="bar", "fizz"="buzz" }

Search for traces containing spans from the foo project and from the bar service that have the mark and tag labels set to test and best, respectively:

{ project="foo", service="bar", "fizz"="test", "tag"="best" }

Searching for spans within a traceSearching for spans within a trace

The search focuses on a single trace you select. The results will include spans meeting all search criteria.

Supported keys:

• project="value": value project spans allowing the = operator. This is a required key.
• service="value": value service spans supporting all string operators.
• span.name="value": Spans with the specified name, using all string operators.
• span.id="value": Spans with the specified ID, using the = and != operators.
• span.status="(OK)|(ERROR)|(UNKNOWN)": Spans with the specified status, using the = and != operators.
• span.critical_path="(PRESENT)|(ABSENT)": Spans, both on and off the critical path, using the = operator.
• span.duration>duration_string: Spans with the specified duration, using the duration operators.

All other keys will be interpreted as labels.

Query examplesQuery examples

Search for all spans with an error:

span.status="ERROR"

Search for spans with an error in the foo service:

span.status="ERROR", service="foo"

Searching by labelsSearching by labels

Searching for spansSearching for spans

By default, the search covers the last 24 hours, but you can change the interval as needed.

The results will include spans meeting all search criteria.

The <key>="<value>" query will return spans containing the <key> label set to <value>. All string operators are supported.

You can specify * and - for the <value>.

• The some_label="*" selector will find all spans that have the some_label label, regardless of its value.
• The some_label="-" selector will find all spans without the some_label label.

If you specify multiple labels, the search will find spans that include every listed label.

Incomplete tracesIncomplete traces

Some traces show an incomplete data warning. This means that certain spans are missing in the system or in the interface, which distorts the view of the request execution path.

A trace can be marked as incomplete:

• When the root span is missing.
• When more than one root span appears because the parent span is missing.

Reasons for missing spans in the tracing system:

• The application fails to send spans, e.g., during a crash, such as an OOM or segfault.
• Spans contain validation errors.
• The project exceeds its quotas.

Reasons for missing spans in the interface:

• Duplicated span.id: Manual ID generation causes the interface to show only the first span with that ID.
• Your trace exceeds the limit of 100,000 spans: The system only shows the spans within this limit.

Span linksSpan links

A span can be linked to one or more other spans defined in SpanContext. A link may reference a span within the same trace or from a different one. Links can be used to represent batch operations where multiple parent spans initiate a single span.

For more information on how to use span links via the API, see this OTEL guide.

The interface shows span links alongside the span name.

Span eventsSpan events

A span can be labeled with one or more events. Each event contains key:value pairs and a timestamp.

You can use events to send information related to the span, such as error messages.

Events are not searchable. If you need to search for events, move them to logs and use log search.

Span events are marked with a triangle icon and can be viewed in the span details.

Critical path of a requestCritical path of a request

In a distributed system, a request goes through a chain of operations (spans) across different services. Tracing renders this path as a graph of spans, where each operation adds to the total execution time.

The critical path is the span sequence that drives the total request execution time. It consists of operations that directly affect the request completion time. Optimizing spans on this path shortens the total execution time and makes it a key focus for performance improvements.

Critical path segments are highlighted on spans with black lines. When hovering over a span, you can see the operation duration and the total time the span spent on the critical path. You can use the span search selector to filter spans.

Critical path calculation algorithmCritical path calculation algorithm

To determine the critical path, the algorithm traverses the span tree starting from the root. Child spans are sorted by end time in descending order. The algorithm picks the last finishing child (LFC). This span lies on the critical path, as the parent span waited for it to complete before proceeding. The algorithm then recurses on the LFC. After recursion, the algorithm selects the next LFC that finished before the previous LFC started, ensuring spans do not overlap.

LimitationsLimitations

The critical path is calculated for traces with a single root span. If a trace lacks a root span or has multiple root spans, the critical path will not be calculated.