General questions about Managed Service for Apache Kafka®

• How do I ensure my cluster is fault tolerant?

• What should I do if I get this or similar error: "disk size must be at least ... according to topics partitions number and replication factor, but size is ..."?

• How do I get the logs of my activity in the Yandex Cloud services?

• How do I move a cluster to a different network?

• How do I set up an alert that triggers as soon as a certain percentage of disk space is used up?

• Why is my cluster slow even though the computing resources are not fully utilized?

• Can I connect to cluster hosts over SSH or get superuser privileges on hosts?

• What should I do if I get a revocation check error when using PowerShell to obtain an SSL certificate?

• Can I manage a cluster using scripts from an Apache Kafka® distribution?

• What should I do if the kafkacat: command not found error occurs?

• Will Karapace be available during maintenance?

• Is Karapace's endpoint constant or does it vary depending on the broker?

• What is the role of Managed Service for Apache Kafka® in database management and maintenance?

• How do I track message loss in an Apache Kafka® topic?

How do I achieve high cluster availability?How do I achieve high cluster availability?

Make sure it complies with all the high availability conditions.

What should I do if I get this or similar error: "disk size must be at least ... according to topics partitions number and replication factor, but size is ..."?What should I do if I get this or similar error: "disk size must be at least ... according to topics partitions number and replication factor, but size is ..."?

This error occurs because the log segments of your topics use more space than the broker storage can provide. For more information about calculating the required space, see Minimum storage size.

To solve this issue, you can increase the disk storage size for the entire cluster or reduce the topic segment size.

Can I get logs of my operations in Yandex Cloud?Can I get logs of my operations in Yandex Cloud?

Yes, you can request information about operations with your resources from Yandex Cloud logs. Do it by contacting support.

What is the retention period for logs?What is the retention period for logs?

Cluster logs are stored for 30 days.

How do I move a cluster to a different network?How do I move a cluster to a different network?

You cannot move a Managed Service for Apache Kafka® cluster to different network.

Here is an alternative solution:

1. Create a new cluster with the same configuration in the target network.
2. Use MirrorMaker to move the topics of your source cluster to the new cluster.

How do I set up an alert that triggers as soon as a certain percentage of disk space is used up?How do I set up an alert that triggers as soon as a certain percentage of disk space is used up?

Create an alert for the disk.used_bytes metric in Yandex Monitoring. This metric shows the disk space usage in the Managed Service for Apache Kafka® cluster.

For disk.used_bytes, use notification thresholds. Their recommended values are as follows:

• Alarm: 90% of disk space
• Warning: 80% of disk space

Thresholds are set in bytes only. For example, the recommended values for a 100 GB disk are as follows:

• Alarm: 96636764160 bytes (90%)
• Warning: 85899345920 bytes (80%)

Why is my cluster slow even though the computing resources are not fully utilized?Why is my cluster slow even though the computing resources are not fully utilized?

Your storage may have insufficient maximum IOPS and bandwidth to process the current number of requests. In this case, throttling occurs, which degrades the entire cluster performance.

The maximum IOPS and bandwidth values increase by a fixed value when the storage size increases by a certain step. The step and increment values depend on the disk type:

To increase the maximum IOPS and bandwidth values and make throttling less likely, expand the storage.

Can I connect to cluster hosts over SSH or get superuser privileges on hosts?Can I connect to cluster hosts over SSH or get superuser privileges on hosts?

You cannot connect to hosts via SSH. This is done for the sake of security and user cluster fault tolerance because direct changes inside a host can render it completely inoperable.

What should I do if I get a revocation check error when using PowerShell to obtain an SSL certificate?What should I do if I get a revocation check error when using PowerShell to obtain an SSL certificate?

Here is the full text of the error:

curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x80092012)
The revocation function was unable to check revocation for the certificate

This means that when connecting to the website, the service was unable to check whether or not its certificate was listed among revoked ones.

To fix this error:

• Make sure your corporate network policies are not blocking the check.

• Run the command with the --ssl-no-revoke parameter:

  mkdir $HOME\.kafka; curl.exe --ssl-no-revoke -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
  

Can I manage a cluster using scripts from an Apache Kafka® distribution?Can I manage a cluster using scripts from an Apache Kafka® distribution?

Yes, you can, but with some limitations.

For more information about the API types available in a cluster and applicable limitations, see APIs in Managed Service for Apache Kafka®.

For an example of using scripts, see Connecting to an Apache Kafka® cluster from applications.

What should I do if the kafkacat: command not found error occurs?What should I do if the kafkacat: command not found error occurs?

On Ubuntu 24.04 and higher, use kcat instead of kafkacat (this command will not work).

Will Karapace be available during maintenance?Will Karapace be available during maintenance?

Karapace is unavailable during maintenance and automatically restarts after its completion. Meanwhile, all schema data is retained as it is stored in the __schema_registry service topic.

Is Karapace's endpoint constant or does it vary depending on the broker?Is Karapace's endpoint constant or does it vary depending on the broker?

Karapace is deployed on each broker host on a separate port, with its own endpoint available for connection. When you delete a broker, the associated endpoint becomes unavailable.

What is the role of Managed Service for Apache Kafka® in database management and maintenance?What is the role of Managed Service for Apache Kafka® in database management and maintenance?

Be mindful of what is managed by the service and what, by the Yandex Cloud customer. Understanding these areas of management helps you use your cloud resources efficiently and avoid potential database-related issues. For more information, see Zones of control between managed database (MDB) service users and Yandex Cloud.

How do I track message loss in an Apache Kafka® topic?How do I track message loss in an Apache Kafka® topic?

If a topic uses the Delete log cleanup policy with a short segment lifetime, messages may be deleted before the consumer group reads them. You can monitor message loss using Managed Service for Apache Kafka® metrics delivered to Monitoring.

To monitor message loss:

1. Use Yandex Monitoring to plot the kafka_group_topic_partition_offset and kafka_log_Log_LogStartOffset metrics on the same chart:
   • For kafka_group_topic_partition_offset, specify these labels:
     • service = managed-kafka
     • name = kafka_group_topic_partition_offset
     • host = <host_FQDN>
     • topic = <topic_name>
     • partition = <partition_number>
     • group = <consumer_group_name>
   • For kafka_log_Log_LogStartOffset, specify these labels:
     • service = managed-kafka
     • name = kafka_log_Log_LogStartOffset
     • host = <host_FQDN>
     • topic = <topic_name>
     • partition = <partition_number>
2. Wait until the number of messages written to the topic is enough for analysis.
3. Navigate to Yandex Monitoring and review the behavior of the previously created metrics:
   • If the kafka_log_Log_LogStartOffset value is higher than kafka_group_topic_partition_offset within the entire monitoring period, the selected consumer group will have enough time to read all new messages from the specified topic segment.
   • kafka_group_topic_partition_offset falling below kafka_log_Log_LogStartOffset signals a message loss.

For more information, see Monitoring message loss in an Apache Kafka® topic.