Monitoring the state of a Valkey™ cluster and its hosts
Data on the cluster and host state is available in the management console
Diagnostic information about cluster states is presented as graphs.
Chart update rate:
- Standard hosts and hosts with an increased RAM to vCPU ratio (
memory-optimized): 15 seconds. - Hosts with a guaranteed vCPU share under 100% (
burstable): 150 seconds.
Note
The most appropriate multiple units (MB, GB, and more) are automatically used in charts.
You can configure alerts in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: Warning and Alarm. If the specified threshold is exceeded, you will receive alerts via the configured notification channels.
Monitoring the cluster state
To view detailed information on the state of a Yandex Managed Service for Valkey™ cluster:
-
In the management console
, select the folder with the cluster you need. -
Go to Yandex Managed Service for Valkey™.
-
Click the cluster name and select the Monitoring tab.
-
To get started with Yandex Monitoring metrics, dashboards, or alerts, click Open in Monitoring in the top panel.
You will see the following charts:
-
Can Read: Displays clusters available for reading.
-
Can Write: Displays clusters available for writing.
-
Cache Hit Rate: Percentage of cache hits on each host.
Values close to 1 show efficient use of the cluster as a caching server. If the hit rate approaches 0, you may need to change your application logic, key lifetime, or the RAM management policy when low on memory.
-
Client recent max input buffer size: Amount of memory used to handle client write connections, in bytes.
-
Client recent max output buffer size: Amount of memory used to handle client read connections:
- Soft Limit: Soft memory usage limit.
- Hard Limit: Hard memory usage limit.
If the Soft Limit value is exceeded, the cluster will wait a few seconds for it to decrease. If the value does not decrease, the connection will be closed.
If the Hard Limit value is exceeded, the connection will be closed immediately. -
Commands Processed: Average number of commands processed by each cluster host.
-
Connected Clients: Number of open connections for each cluster host.
If the cluster is sharded or uses replication, some of the connections will be used for inter-host communication within the cluster.
If you encounter errors when connecting to the cluster, inactive applications may be keeping connections open for too long. If this is the case, update the Valkey™ settings by editing the Timeout value. -
Copy-on-write allocation: Memory consumption by Valkey™ processes when using COW (copy-on-write)
, in bytes.The chart shows the latest measured Valkey™ parameter values:
- module_fork_last_cow_size: Amount of data copied during a
fork()call using COW. - aof_last_cow_size: Size of data copied when creating an AOF file.
- rdb_last_cow_size: Size of data copied when creating an RDB file.
For more information, see Backups.
- module_fork_last_cow_size: Amount of data copied during a
-
DB keys: Number of keys stored in all cluster databases.
-
Evicted keys: Number of keys deleted from memory when inserting new data.
By default, Valkey™ applies the noeviction memory policy, which prevents key deletion and returns an error if memory is too low to insert new data. To use a different memory management policy, adjust the Maxmemory policy value in the Valkey™ settings.
-
Inner memory limit: Amount of RAM available for Valkey™ processes, in bytes:
- maxmemory: Maximum amount of memory allocated for user data.
- used_memory: Actual host memory usage.
If used_memory reaches maxmemory when trying to insert new records, Valkey™ will apply the memory management mode defined by the Maxmemory policy setting.
Note
The value of maxmemory for Valkey™ hosts is set to 75% of the available memory. For more information, see Memory management.
-
Outer memory limit: Total amount of RAM available for use on hosts, in bytes:
- memory_limit: Amount of memory allocated to each host.
- used_memory_rss: Memory used by Valkey™ processes.
If used_memory_rss approaches memory_limit, the operating system may forcibly terminate the relevant Valkey™ process. To avoid this:
- Change the application logic to reduce the amount of data stored in Valkey™.
- Change the Maxmemory policy value defining the RAM management policy when low on memory in the Valkey™ settings.
- Upgrade the host to a higher class.
-
Redis Used Memory on Masters: RAM usage on master hosts, in bytes:
- db_hashtable_overhead: For storing hash tables of all databases.
- used_memory_scripts: For storing and running scripts
. - mem_aof_buffer: For the AOF buffer.
- mem_clients_normal: For handling external client connections.
- mem_clients_slaves: For handling replication connections.
- mem_replication_backlog: For the replication backlog.
- used_memory_startup: For Valkey™ processes at startup (e.g., after a cluster restart).
- used_memory_dataset: For storing data.
- mem_cluster_links: For network connections in cluster mode.
-
Redis Used Memory on Replicas: RAM usage on replica hosts, in bytes:
- db_hashtable_overhead: For storing hash tables of all databases.
- used_memory_scripts: For storing and running scripts
. - mem_aof_buffer: For the AOF buffer.
- mem_clients_normal: For handling external client connections.
- mem_clients_slaves: For handling replication connections.
- mem_replication_backlog: For the replication backlog.
- used_memory_startup: For Valkey™ processes at startup (e.g., after a cluster restart).
- used_memory_dataset: For storing data.
- mem_cluster_links: For network connections in cluster mode.
-
Redis-server OOM kills (for last hour): Number of Valkey™ processes terminated due to OOM (out-of-memory) for the last hour.
To reduce the number of terminated processes:
- Change the application logic to reduce the amount of data stored in Valkey™.
- Change the Maxmemory policy value defining the RAM management policy when low on RAM in the Valkey™ settings.
- Upgrade the host to a higher class.
-
Replication buffer size: Replication backlog size, in bytes:
- repl_backlog_size: Maximum size of memory available for the replication backlog.
- repl_backlog_histlen: Size of memory currently used by the replication backlog.
When the replication backlog runs out of memory, full replication starts. This will reduce cluster performance because full replication significantly increases RAM usage as well as CPU and network workload.
-
Replication Lag: Replica's lag behind the master, in seconds.
A non-zero value indicates either slow command execution on the replica or replica overload.
For more information, see Replication and fault tolerance.
-
Slowlog top operations on Master: List of the 5 slowest commands executed on the master host within one minute.
-
Slowlog top operations on Replicas: List of the 5 slowest commands executed on each replica host within one minute.
-
Slowlog new records: Displays new entries in the slow log.
A slow command is a command whose runtime has exceeded the Slowlog log slower than setting. The chart only shows the first part of the command and its call count per minute.
Monitoring the state of hosts
To view detailed information on the state of individual Yandex Managed Service for Valkey™ hosts:
- In the management console
, select the folder with the cluster you need. - Go to Yandex Managed Service for Valkey™.
- Click the name of your cluster and select Hosts → Monitoring.
- Select the host from the drop-down list.
This page displays the charts showing workloads of individual cluster hosts:
- CPU usage: Processor core workload. As the workload goes up, the
idlevalue goes down. - Disk read/write bytes: Speed of disk operations, in bytes per second.
- Disk IOPS: Number of disk operations per second.
- Disk space usage: Amount of used and total disk space.
- Memory usage: Amount of RAM used, in bytes. At high workloads, the
Freevalue goes down, while the other values go up. - Network bytes: Network data exchange rate, in bytes per second.
- Network packets: Number of network packets exchanged per second.
The Disk read/write bytes and the Disk IOPS charts show the increase of the Read value during database read activity, and in Write, during database write activity..
For Replica hosts, the Received value is normally greater than Sent on the Network bytes and Network packets charts.
Setting up alerts in Yandex Monitoring
To configure state indicator alerts for a cluster and hosts:
- In the management console
, select the folder containing the cluster for which you want to set up alerts. - Go to
Monitoring. - Under Service dashboards, select:
- Yandex Managed Service for Valkey™ — Cluster Overview to set up cluster alerts.
- Yandex Managed Service for Valkey™ — Host Overview to set up host alerts.
- In the chart you need, click
and select Create alert. - If the chart shows multiple metrics, select the data query to generate a metric and click Continue. You can learn more about the query language in this Yandex Monitoring article.
- Set the
AlarmandWarningthresholds to trigger the alert. - Click Create alert.
To have other cluster health indicators monitored automatically:
- Create an alert.
- Add a status metric.
- In the alert parameters, set the alert thresholds.
Below are the recommended thresholds for some metrics:
| Metric | Internal metric name | Alarm |
Warning |
|---|---|---|---|
| Database write availability | can_write |
Equal to 0 |
— |
| Number of out-of-memory errors, per hour | redis_oom_count |
More than 2 |
More than 0 |
| RAM utilization (only for the noeviction policy) | redis_used_memory |
90% of RAM | 75% of RAM |
| Storage space used | disk.used_bytes |
90% of the storage size | 80% of the storage size |
For the disk.used_bytes metric, the Alarm and Warning thresholds are only set in bytes. For example, the recommended values for a 100 GB disk are as follows:
Alarm:96636764160bytes (90%).Warning:85899345920bytes (80%).
You can view the current RAM of the hosts in the detailed information about the cluster. For a complete list of supported metrics, see this Monitoring guide.
Cluster state and status
The State of a cluster shows the health of its hosts, while the Status shows whether the cluster is started, stopped, or is at an intermediate stage.
To check the cluster’s state and status:
- In the management console
, select the folder with the cluster you need. - Go to Yandex Managed Service for Valkey™.
- In the cluster row, hover over the indicator in the Availability column.
Cluster states
| State | Description | Suggested actions |
|---|---|---|
| ALIVE | Cluster is operating normally. | No action is required. |
| DEGRADED | Cluster is not running at its full capacity: the state of at least one of the hosts is other than ALIVE. |
Run the diagnostics:
|
| DEAD | The cluster is down: none of its hosts are running. | Make a support request
|
| UNKNOWN | Cluster state is unknown. | Make a support request
|
Cluster statuses
| Status | Description | Suggested actions |
|---|---|---|
| CREATING | Preparing for the first start | Wait a while and get started. The time it takes to create a cluster depends on the host class. |
| RUNNING | The cluster is operating normally | No action is required. |
| STOPPING | The cluster is stopping | After a while, the cluster status will switch to STOPPED and the cluster will be disabled. No action is required. |
| STOPPED | The cluster is stopped | Start the cluster to get it running again. |
| STARTING | Starting the cluster that was stopped earlier | After a while, the cluster status will switch to RUNNING. Wait a while and get started. |
| UPDATING | Updating the cluster's configuration | Once the update is complete, the cluster will get the status it had prior to the update: RUNNING or STOPPED. |
| ERROR | Error when performing an operation with the cluster or during a maintenance window | If the cluster remains in this status for a long time, contact support |
| STATUS_UNKNOWN | The cluster is unable to determine its status | If the cluster remains in this status for a long time, contact support |