Contact UsGet started

Reading metrics via the Remote API

Written by
Updated at December 4, 2024
  1. In the management console, select the folder where the data is stored.
  2. Create a service account with the monitoring.viewer role for the selected folder.
  3. Create an API key for the service account.
  4. Add the endpoint (url) to the remote_read section of the Prometheus configuration: 
    ...
remote_read:
  ...
  - url: '<URL>' # Available on request
    bearer_token: '<API_key>'
    # Or via a file (recommended):
    # bearer_token_file: '<name_of_file_with_API_key>'

    # We recommend adding the X-Lookback-Delta custom header.
    # If you did not change the lookback-delta parameter value in
    # your Prometheus instance configuration, specify the default value, 5m.
    # This information is required for optimized decimation.
    headers:
      X-Lookback-Delta: 5m

    # By default, data is only requested for the time ranges
    # not featured in the Prometheus local storage (older than the Storage retention).
    # If you want to be able to request data always, which makes sense if read and write operations
    # are made from different Prometheus instances, enable the following option:
    # [ read_recent: true ]

    # If you want to read only some metrics via the Remote API, specify their labels:
    # required_matchers:
    # [ <label_name>: <label_value> ... ]
  5. Restart Prometheus or reload the configuration.

Error examples

Error code Error message Description
404 Endpoint not found. Make sure the configuration specifies the correct url. remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returned
HTTP status 404 Not Found: {"status":"error","errorType":"not_found","error":"workspace not found: monb1piptmdo********"}
401 Service account not found. Make sure the configuration specifies the correct API key. remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returned
HTTP status 401 Unauthorized: {"status":"error","errorType":"","error":"cannot authenticate by either token or api-key, cause: UNAUTHENTICATED: The token is invalid"}
403 No read permissions. Make sure the service account has the monitoring.viewer role for the selected folder. remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returned
HTTP status 403 Forbidden: {"status":"error","errorType":"","error":"PERMISSION_DENIED: Permission denied"}
429 Number of Remote Read API read requests per second quota exceeded remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returned
HTTP status 429 Too Many Requests: {"status":"error","errorType":"execution","error":"too many read requests: monb1piptmdo********"}
400 Too many lines returned from the request. Try refining your request. remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returned
HTTP status 400 Bad Request: {"status":"error","errorType":"bad_data","error":"Too many metrics are loaded by selectors {job=='grafana'}, expected not more than: 10000"}

Prometheus metrics

Metric name Units Comment
prometheus_remote_storage_read_queries_total Invocations Total number of read requests
prometheus_remote_storage_read_request_duration_seconds Seconds Read request execution time histogram

Current limitations

  • The maximum number of time series you can get metadata for using the /api/v1/series query is 10,000.
  • The maximum number of time series you can read per one /api/v1/query or /api/v1/query_range query is 10,000.

© 2024 Linux Foundation. All rights reserved. The Linux Foundation owns and uses registered trademarks. For a list of Linux Foundation trademarks, see Trademark Usage.

Previous
Overview
Next
Grafana