Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Writing metrics via the Remote API

Written by
Updated at April 28, 2025

To set up metric writing via the Remote API with Prometheus used as an agent for collecting metrics:

  1. In the management console, select the folder where you want to store data.
  2. Create a service account with the monitoring.editor role for the selected folder.
  3. Create an API key for the service account.
  4. Add the endpoint (url) to the remote_write section of the Prometheus configuration: 
    ...
remote_write:
  ...
  - url: 'https://monitoring.api.cloud.yandex.net/prometheus/workspaces/<workspace_ID>/api/v1/write'
    bearer_token_file: '<name_of_file_with_API_key>'
    # Alternatively, you can add the key itself (not recommended):
    # bearer_token: '<API_key>'
    queue_config:
        max_samples_per_send: 2000 # 10,000 if using `vmagent`
        min_backoff: 100ms
        max_backoff: 15s
        # For Prometheus 2.26 or higher
        # retry_on_http_429: true
    metadata_config:
        send: false
  5. Restart Prometheus or reload the configuration.
  6. Make sure the Prometheus logs contain no errors related to the specified endpoint.

Error examples

Error code Description Log example
404 Endpoint not found. Make sure the configuration specifies the correct url. ts=2024-04-04T09:59:15.973Z caller=dedupe.go:112 component=remote level=error remote_name=366d8d
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=105 exemplarCount=0 err="server returned HTTP status 404 Not Found:
{\"status\":\"error\",\"errorType\":\"not_found\",\"error\":\"workspace not found: monb1piptmdo********\"}
401 Service account not found. Make sure the configuration specifies the correct API key. ts=2024-04-12T12:55:10.124Z caller=dedupe.go:112 component=remote level=error remote_name=6b764c
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=1 exemplarCount=0 err="server returned HTTP status 401 Unauthorized:
{\"error\":\"cannot authenticate by either token or api-key, cause: UNAUTHENTICATED: The token is invalid\",\"errorType\":\"\",\"status\":\"error\"}"
403 No write permissions. Make sure the service account has the monitoring.editor role for the selected folder. ts=2024-04-05T17:10:47.706Z caller=dedupe.go:112 component=remote level=error remote_name=a9c26f
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=205 exemplarCount=0 err="server returned HTTP status 403 Forbidden:
{\"error\":\"PERMISSION_DENIED: Permission denied\",\"errorType\":\"\",\"status\":\"error\"}"
413 Number of unique metrics quota exceeded. ts=2024-04-05T16:48:16.002Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=205 exemplarCount=0 err="server returned HTTP status 413 Request Entity Too Large:
{\"status\":\"error\",\"errorType\":\"execution\",\"error\":\"more than 1000000 metrics in shard\"}
413 Number of metrics per write request quota exceeded. ts=2024-04-05T17:39:29.102Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=10001 exemplarCount=0 err="server returned HTTP status 413 Request Entity Too Large:
{\"status\":\"error\",\"errorType\":\"execution\",\"error\":\"more than 10000 metrics from one URL\"}
429 Maximum data write speed in Remote Write format quota exceeded. If retry_on_http_429 is enabled, the data will be resent automatically and the load will be distributed over time. If the error persists, contact support. ts=2024-04-06T14:31:01.113Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="Failed to send batch, retrying" count=205 exemplarCount=0 err="server returned HTTP status 429 Too Many Requests:
{\"status\":\"error\",\"errorType\":\"execution\",\"error\":\"too many requests (bytes/sec)\"}
400 Invalid Remote Write data format. ts=2024-04-07T13:42:10.543Z caller=dedupe.go:112 component=remote level=error remote_name=37b6df
url=https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/write
msg="non-recoverable error" count=499 exemplarCount=0 err="server returned HTTP status 400 Bad Request:
{\"status\":\"error\",\"errorType\":\"bad_data\",\"error\":\"unrecognized remote write format (some headers are missing?): monb1piptmdo********\"}

Prometheus metrics

Metric name Units Comment
prometheus_remote_storage_sent_batch_duration_seconds Seconds Write request execution time histogram
prometheus_remote_storage_highest_timestamp_in_seconds Seconds Most recent timestamp written to the WAL
prometheus_remote_storage_queue_highest_sent_timestamp_seconds Seconds Most recent timestamp sent to remote storage
prometheus_remote_storage_bytes_total Bytes Total data bytes (non-metadata) sent to remote storage after compression
prometheus_remote_storage_enqueue_retries_total Count Total number of failed sample enqueue retries
prometheus_remote_storage_max_samples_per_send Count Maximum number of samples sent per write request
prometheus_remote_storage_samples_dropped_total Count Total samples read from the WAL but not sent to remote storage
prometheus_remote_storage_samples_failed_total Count Total samples that could not be sent to remote storage due to non-recoverable errors.
prometheus_remote_storage_samples_pending Count Number of pending samples to send to remote storage
prometheus_remote_storage_samples_retried_total Count Total samples that could not be sent to remote storage but were retried as the send error was recoverable.
prometheus_remote_storage_samples_total Count Total samples sent to remote storage
prometheus_remote_storage_shards Count Total shards used for sending samples to remote storage
prometheus_remote_storage_shards_min Count Minimum number of shards available
prometheus_remote_storage_shards_max Count Maximum number of shards available
prometheus_remote_storage_shards_desired Count Result of calculating the optimal number of shards to process the total incoming samples.
prometheus_wal_watcher_current_segment Number Current WAL read segment
prometheus_tsdb_wal_segment_current Number Current WAL write segment

© 2025 Linux Foundation. All rights reserved. The Linux Foundation owns and uses registered trademarks. For a list of Linux Foundation trademarks, see Trademark Usage.

Previous
Metric collection agent
Next
Diagnostics and troubleshooting