General questions about Managed Service for Apache Kafka®

• How do I ensure my cluster is fault tolerant?

• What should do I do if I get this or similar error: "disk size must be at least ... according to topics partitions number and replication factor, but size is ..."?

• How do I get the logs of my actions in the services?

• How do I move a cluster to a different network?

• How do I set up an alert that triggers as soon as a certain percentage of disk space has been used up?

• Why is my cluster slow even though there are still some computing resources to spare?

• Can I connect to cluster hosts over SSH or get superuser privileges on hosts?

• What should I do if I get the revocation check error when using PowerShell to obtain an SSL certificate?

• Can I manage a cluster using scripts from a Apache Kafka® distribution?

• What should I do if the kafkacat: command not found error occurs?

• Will Karapace be available during maintenance?

• Is Karapace's endpoint constant or does it vary depending on the broker?

• What parts of database management and maintenance does Managed Service for Apache Kafka® handle?

• How do I track message loss in an Apache Kafka® topic?

How do I achieve high cluster availability?How do I achieve high cluster availability?

Make sure it complies with all the high availability conditions.

What should do I do if I get this or similar error: "disk size must be at least ... according to topics partitions number and replication factor, but size is ..."?What should do I do if I get this or similar error: "disk size must be at least ... according to topics partitions number and replication factor, but size is ..."?

The error results from the fact that your topics' log segments take up more space than is available in broker storage. For more information about calculating the required space, see Minimum storage size.

To solve the issue, you can increase the disk size or reduce the topic segment or total cluster size.

Can I get logs of my operations in Yandex Cloud?Can I get logs of my operations in Yandex Cloud?

Yes, you can request information about operations with your resources from Yandex Cloud logs. Do it by contacting support.

What is the retention period for logs?What is the retention period for logs?

Cluster logs are stored for 30 days.

How do I move a cluster to a different network?How do I move a cluster to a different network?

You cannot move a Managed Service for Apache Kafka® cluster to another network.

Here is an alternative solution:

1. Create a new cluster with the same configuration in the target network.
2. Use MirrorMaker to move the topics of the source cluster to the new one.

How do I set up an alert that triggers as soon as a certain percentage of disk space has been used up?How do I set up an alert that triggers as soon as a certain percentage of disk space has been used up?

Create an alert with the disk.used_bytes metric in Yandex Monitoring. This metric shows the disk space usage in the Managed Service for Apache Kafka® cluster.

For disk.used_bytes, use notification thresholds. Their recommended values are as follows:

• Alarm: 90% of disk space
• Warning: 80% of disk space

Thresholds are set in bytes only. For example, the recommended values for a 100 GB disk are as follows:

• Alarm: 96636764160 bytes (90%)
• Warning: 85899345920 bytes (80%)

Why is my cluster slow even though the computational resources are not being fully utilized?Why is my cluster slow even though the computational resources are not being fully utilized?

Your storage may have insufficient maximum IOPS and bandwidth to process the current number of requests. In this case, throttling occurs, which degrades the entire cluster performance.

The maximum IOPS and bandwidth values increase by a fixed value when the storage size increases by a certain step. The step and increment values depend on the disk type:

To increase the maximum IOPS and bandwidth values and make throttling less likely, increase the storage size.

Can I connect to cluster hosts over SSH or get superuser privileges on hosts?Can I connect to cluster hosts over SSH or get superuser privileges on hosts?

You cannot connect to hosts via SSH. This is done for the sake of security and user cluster fault tolerance because direct changes inside a host can render it completely inoperable.

What should I do if I get a revocation check error when using PowerShell to obtain an SSL certificate?What should I do if I get a revocation check error when using PowerShell to obtain an SSL certificate?

Here is the full text of the error:

curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x80092012)
The revocation function was unable to check revocation for the certificate

This means, when connecting to the website, the service was unable to check whether or not its certificate was listed among revoked ones.

To fix this error:

• Make sure your corporate network policies are not blocking the verification.

• Run the following command with the --ssl-no-revoke parameter:

  mkdir $HOME\.kafka; curl.exe --ssl-no-revoke -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
  

Can I manage a cluster using scripts from an Apache Kafka® distribution?Can I manage a cluster using scripts from an Apache Kafka® distribution?

Yes, you can, but with some limitations.

For more information about the API types available in a cluster and applicable limitations, see APIs in Managed Service for Apache Kafka®.

For an example of using scripts, see Connecting to a Apache Kafka® cluster from applications.

What should I do if the kafkacat: command not found error occurs?What should I do if the kafkacat: command not found error occurs?

On Ubuntu 24.04 and higher, use kcat instead of kafkacat (this command will not work).

Will Karapace be available during maintenance?Will Karapace be available during maintenance?

Karapace is unavailable during maintenance and automatically restarts after its completion. Meanwhile, all schema data is retained as it is stored in the __schema_registry service topic.

Is Karapace's endpoint constant or does it vary depending on the broker?Is Karapace's endpoint constant or does it vary depending on the broker?

Karapace is deployed on each broker host on a separate port, with its own endpoint available for connection. When you delete a broker, the corresponding endpoint becomes unavailable.

What parts of database management and maintenance does Managed Service for Apache Kafka® handle?What parts of database management and maintenance does Managed Service for Apache Kafka® handle?

Be mindful of what is what is controlled by the service, and what by the Yandex Cloud customer. Understanding these control zones will help you use your cloud resources effectively and avoid potential database-related problems. For more information, see Zones of control between managed database (MDB) service users and Yandex Cloud.

How do I track message loss in an Apache Kafka® topic?How do I track message loss in an Apache Kafka® topic?

If a topic uses the Delete log cleanup policy with a short segment lifetime, messages may be deleted before the consumer group reads them. You can monitor message loss using Managed Service for Apache Kafka® metrics delivered to Monitoring.

To monitor message loss:

1. Use Yandex Monitoring to plot the kafka_group_topic_partition_offset and kafka_log_Log_LogStartOffset metrics on the same chart:
   • For kafka_group_topic_partition_offset, specify these labels:
     • service = managed-kafka
     • name = kafka_group_topic_partition_offset
     • host = <host_FQDN>
     • topic = <topic_name>
     • partition = <partition_number>
     • group = <consumer_group_name>
   • For kafka_log_Log_LogStartOffset, specify these labels:
     • service = managed-kafka
     • name = kafka_log_Log_LogStartOffset
     • host = <host_FQDN>
     • topic = <topic_name>
     • partition = <partition_number>
2. Wait until the number of messages written to the topic is enough for analysis.
3. Navigate to Yandex Monitoring and review the behavior of the previously created metrics:
   • If the kafka_log_Log_LogStartOffset value is higher than kafka_group_topic_partition_offset within the entire monitoring period, the selected consumer group will have enough time to read all new messages from the specified topic segment.
   • kafka_group_topic_partition_offset falling below kafka_log_Log_LogStartOffset signals a message loss.

For more information, see Monitoring message loss in an Apache Kafka® topic.