Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Writing metrics via the Remote API

Written by
Updated at October 31, 2024

To set up metric writes via the Remote API with Prometheus used as an agent for collecting metrics:

  1. In the management console, select the folder where you want to store data.
  2. Create a service account with the monitoring.editor role for the selected folder.
  3. Create an API key for the service account.
  4. Add the endpoint (url) to the remote_write section of the Prometheus configuration: 
    ...
remote_write:
  ...
  - url: 'https://monitoring.api.cloud.yandex.net/prometheus/workspaces/<workspace_ID>/api/v1/write'
    bearer_token_file: '<name_of_file_with_API_key>'
    # Or the key itself (not recommended):
    # bearer_token: '<API_key>'
    queue_config:
        max_samples_per_send: 2000 # 10,000 if using vmagent
        min_backoff: 100ms
        max_backoff: 15s
        # for Prometheus version 2.26 or higher
        # retry_on_http_429: true
    metadata_config:
        send: false
  5. Restart Prometheus or reload the configuration.
  6. Make sure the Prometheus logs contain no errors related to the specified endpoint.

Error examples

Error code Description Log example
404 Endpoint not found. Make sure the configuration specifies the correct url. ts=2024-04-04T09:59:15.973Z caller=dedupe.go:112 component=remote level=error remote_name=366d8d
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=105 exemplarCount=0 err="server returned HTTP status 404 Not Found:
{\"status\":\"error\",\"errorType\":\"not_found\",\"error\":\"workspace not found: monb1piptmdo********\"}
401 Service account not found. Make sure the configuration specifies the correct API key. ts=2024-04-12T12:55:10.124Z caller=dedupe.go:112 component=remote level=error remote_name=6b764c
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=1 exemplarCount=0 err="server returned HTTP status 401 Unauthorized:
{\"error\":\"cannot authenticate by either token or api-key, cause: UNAUTHENTICATED: The token is invalid\",\"errorType\":\"\",\"status\":\"error\"}"
403 No write permissions. Make sure the service account has the monitoring.editor role for the selected folder. ts=2024-04-05T17:10:47.706Z caller=dedupe.go:112 component=remote level=error remote_name=a9c26f
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=205 exemplarCount=0 err="server returned HTTP status 403 Forbidden:
{\"error\":\"PERMISSION_DENIED: Permission denied\",\"errorType\":\"\",\"status\":\"error\"}"
413 Number of unique metrics quota exceeded. ts=2024-04-05T16:48:16.002Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=205 exemplarCount=0 err="server returned HTTP status 413 Request Entity Too Large:
{\"status\":\"error\",\"errorType\":\"execution\",\"error\":\"more than 1000000 metrics in shard\"}
413 Number of metrics per write request quota exceeded. ts=2024-04-05T17:39:29.102Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=10001 exemplarCount=0 err="server returned HTTP status 413 Request Entity Too Large:
{\"status\":\"error\",\"errorType\":\"execution\",\"error\":\"more than 10000 metrics from one URL\"}
429 Maximum data write speed in Remote Write format quota exceeded. If the retry_on_http_429 option is enabled, the data will be resent automatically and the load will be distributed over time. If the error persists, contact support. ts=2024-04-06T14:31:01.113Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="Failed to send batch, retrying" count=205 exemplarCount=0 err="server returned HTTP status 429 Too Many Requests:
{\"status\":\"error\",\"errorType\":\"execution\",\"error\":\"too many requests (bytes/sec)\"}
400 Invalid Remote Write data format. ts=2024-04-07T13:42:10.543Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=499 exemplarCount=0 err="server returned HTTP status 400 Bad Request:
{\"status\":\"error\",\"errorType\":\"bad_data\",\"error\":\"unrecognized remote write format (some headers are missing?): monb1piptmdo********\"}

Prometheus metrics

Metric name Units Comment
prometheus_remote_storage_sent_batch_duration_seconds Seconds Write request execution time histogram.
prometheus_remote_storage_bytes_total Bytes Total number of bytes of data (not metadata) sent to remote storage after compression.
prometheus_remote_storage_enqueue_retries_total Number Total number of failed sample enqueue retries.
prometheus_remote_storage_max_samples_per_send Number Maximum number of samples sent per write request.
prometheus_remote_storage_samples_dropped_total Number Total number of samples read from the WAL but not sent to remote storage.
prometheus_remote_storage_samples_failed_total Number Total number of samples that failed to be sent to remote storage due to non-recoverable errors.
prometheus_remote_storage_samples_pending Number Number of pending samples to be sent to remote storage.
prometheus_remote_storage_samples_retried_total Number Total number of samples that failed to be sent to remote storage with their sending retried afterwards as the error was recoverable.
prometheus_remote_storage_samples_total Number Total number of samples sent to remote storage.

© 2024 Linux Foundation. All rights reserved. The Linux Foundation owns and uses registered trademarks. For a list of Linux Foundation trademarks, see Trademark Usage.

Previous
Overview
Next
Overview