Monitoring the state of Apache Kafka® clusters and hosts

1. In the management console, go to the appropriate folder.

2. In the list of services, select Managed Service for Kafka.

3. Click the cluster name and open the Monitoring tab.

4. To get started with Yandex Monitoring metrics, dashboards, or alerts, click Open in Monitoring in the top panel.

The page displays the following charts:

• Alive brokers: Number of functional brokers for each host with the KAFKA role.
• Offline partitions: Displays the parameter values below:
  • OfflineReplicaCount: Number of partitions without a leader. These partitions do not support message writes or reads.
  • Underreplicated partitions: Number of partitions whose replication factor is greater than the number of their in-sync replicas (ISRs).
  • Under min ISR partitions: Number of partitions with a number of in-sync replicas below the minimum allowed value specified in the settings.
• Errors: Number of failed requests by error type.
• Free space: Free disk space for each host with the KAFKA role, in bytes.
• Messages in: Message write rate, in messages per second.
• Bytes In/Out: Write and message read speed for each host with the KAFKA role, in bytes per second.
• Top 5 topics by size: Amount of data for each of the 5 largest topics, in bytes.
• Replicated bytes: Replication data streaming rate for each host with the KAFKA role, in bytes per second.
• Partitions: Total number of partition replicas.
• Replication lag [messages]: Greatest replication lag for each host with the KAFKA role, in messages.
• Request time (0.95 quantile): Request processing time in the 0.95 quantile by request type.
• Requests: Request rate by request type, in requests per second.

1. In the management console, go to the appropriate folder.
2. In the list of services, select Managed Service for Kafka.
3. Click the cluster name and select the Hosts → Monitoring tab.
4. Select the host from the drop-down list.

This page displays charts showing the load on an individual host in the cluster:

• CPU usage: Usage of processor cores. As the load goes up, the Idle value goes down.
• Memory usage: Use of RAM, in bytes. At high loads, the Free value goes down, while the other values go up.
• Disk read/write bytes: Speed of disk operations, in bytes per second.
• Disk IOPS: Number of disk operations per second.
• Network bytes: Speed of data exchange over the network, in bytes per second.
• Network packets: Number of packets exchanged over the network, per second.
• Free space: Free disk space, in bytes.

1. In the management console, select the folder with the cluster you want to configure alerts for.
2. In the list of services, select  Monitoring.
3. Under Service dashboards, select Managed Service for Apache Kafka® — Cluster Overview.
4. In the chart you need, click and select Create alert.
5. If the chart shows multiple metrics, select a data query to generate a metric and click Continue. You can learn more about the query language in the Yandex Monitoring documentation.
6. Set the Alarm and Warning threshold values to trigger the alert.
7. Click Create alert.

1. Create an alert.
2. Add a status metric.
3. In the alert parameters, set up your alert thresholds.