Reading metrics via the Remote API
Written by
Updated at May 14, 2024
- In the management console, select the folder where data is stored.
- Create a service account with the
monitoring.viewerrole for the selected folder. - Create an API key for the service account.
- Add the endpoint (
url) to theremote_readsection of the Prometheus configuration:... remote_read: ... - url: '<URL>' # provided upon request bearer_token: '<API_key>' # Or via a file (recommended): # bearer_token_file: '<name_of_the_file_with_the_API_key>' # We recommend adding a user-defined X-Lookback-Delta header. # If you did not change the lookback-delta parameter value in the configuration # of your Prometheus instance, specify the default value, 5m. # This info is required for optimized decimation. headers: X-Lookback-Delta: 5m # By default, data is only requested for the time ranges # not featured in the Prometheus local storage (older than the Storage retention). # If you want to request data at any time, which makes sense if read and write operations # are performed from different Prometheus instances, enable the following option: # [ read_recent: true ] # If you want to read only some of the metrics via the Remote API, specify their labels: # required_matchers: # [ <label_name>: <label_value> ... ] - Restart Prometheus or reload the configuration.
Error examples
| Error code | Error message | Description |
|---|---|---|
404 |
Endpoint not found. Make sure the configuration specifies the correct url. |
remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returnedHTTP status 404 Not Found: {"status":"error","errorType":"not_found","error":"workspace not found: monb1piptmdo********"} |
401 |
Service account not found. Make sure the configuration specifies the correct API key. | remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returnedHTTP status 401 Unauthorized: {"status":"error","errorType":"","error":"cannot authenticate by either token or api-key, cause: UNAUTHENTICATED: The token is invalid"} |
403 |
No read permissions. Make sure the service account has the monitoring.viewer role for the selected folder. |
remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returnedHTTP status 403 Forbidden: {"status":"error","errorType":"","error":"PERMISSION_DENIED: Permission denied"} |
429 |
Number of Remote Read API read requests per second quota exceeded | remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returnedHTTP status 429 Too Many Requests: {"status":"error","errorType":"execution","error":"too many read requests: monb1piptmdo********"} |
400 |
Too many lines returned from the request. Try refining your request. | remote_read: remote server https://monitoring.api.cloud.yandex.net/prometheus/workspaces/monb1piptmdo********/api/v1/read returnedHTTP status 400 Bad Request: {"status":"error","errorType":"bad_data","error":"Too many metrics are loaded by selectors {job=='grafana'}, expected not more than: 10000"} |
Prometheus metrics
| Metric name | Units | Explanations |
|---|---|---|
prometheus_remote_storage_read_queries_total |
Invocations | Total number of read requests |
prometheus_remote_storage_read_request_duration_seconds |
Seconds | Read request execution time histogram |
© 2024 Linux Foundation. All rights reserved. The Linux Foundation owns and uses registered trademarks. For a list of Linux Foundation trademarks, see Trademark Usage.