Apache Kafka® settings

For Managed Service for Apache Kafka® clusters, you can configure Apache Kafka® settings. Some settings are configured at the cluster level, while others, at the topic level.

The label next to the setting name shows which interface can be used to configure this setting: the management console, CLI, API, or Terraform. The All interfaces label means that all of the above interfaces are supported.

Depending on the selected interface, the same setting will be represented differently. For example, Compression type in the management console is the same as:

• --compression-type in the CLI
• compression_type in the gRPC API and Terraform
• compressionType in the REST API

Cluster-level settingsCluster-level settings

You can use the following settings:

• Auto create topics enable All interfaces

  Manages automatic topic creation.

  This option is disabled by default (false).

  For more information, see the Apache Kafka® documentation.

• Compression type All interfaces

  Codecs used for message compression include:

  Management console, Terraform, and API	CLI	Description
  COMPRESSION_TYPE_UNCOMPRESSED	uncompressed	Compression is disabled
  COMPRESSION_TYPE_ZSTD	zstd	zstd codec
  COMPRESSION_TYPE_LZ4	lz4	lz4 codec
  COMPRESSION_TYPE_SNAPPY	snappy	snappy codec
  COMPRESSION_TYPE_GZIP	gzip	gzip codec
  COMPRESSION_TYPE_PRODUCER	producer	The codec is set on the producer side

  By default, the compression codec is set by the producer (COMPRESSION_TYPE_PRODUCER).

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting corresponds to Compression type.

  For more information, see the Apache Kafka® documentation.

• Default replication factor All interfaces

  Number of data copies for a topic within a cluster.

  This setting only applies to automatically created topics.

  The minimum and default value is 1. The maximum value is equal to the number of broker hosts in the cluster.

  See also the Replication factor topic-level setting.

  For more information, see the Apache Kafka® documentation.

• Log flush interval messages All interfaces

  Number of topic messages that can accumulate in memory before these messages are flushed to the disk. For example, if set to 1, each message will be flushed to the disk once received. If set to 5, messages will be flushed to the disk in groups of five.

  The minimum value is 1; the maximum one is 9223372036854775807 (default).

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting corresponds to Flush messages.

  For more information, see the Apache Kafka® documentation.

• Log flush interval ms All interfaces

  Maximum time in milliseconds a message can be stored in memory before being flushed to the disk. If no value is specified, the Log flush scheduler interval ms value is used.

  The maximum value is 9223372036854775807.

  This is a global cluster-level setting. You can override it at the topic level.

  For more information, see the Apache Kafka® documentation.

• Log flush scheduler interval ms All interfaces

  The period (in milliseconds) to check whether there are logs to flush to the disk. This check is performed by the log flusher.

  The maximum value is 9223372036854775807 (default).

  For more information, see the Apache Kafka® documentation.

• Log preallocate All interfaces

  This setting determines whether to pre-allocate space for log file segments.

  By default, the space for the log file segments is allocated as logs become full (false).

  This is a global cluster-level setting. You can override it at the topic level.

  For more information, see the Apache Kafka® documentation.

• Log retention bytes All interfaces

  Maximum size a partition can grow to. When the partition reaches this size, Apache Kafka® starts deleting the old log segments. This setting applies if the Delete log cleanup policy is effective.

  The minimum value is -1 (default; the log size is unlimited); the maximum one is 9223372036854775807.

  Use this setting if you need to manage the log size due to limited disk space.

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting corresponds to Retention, bytes.

  For more information, see the Apache Kafka® documentation.

  See also the Log retention ms setting.

• Log retention hours All interfaces

  Time (in hours) for Apache Kafka® to keep a log segment file. This setting applies if the Delete log cleanup policy is effective; once the specified timeout expires, the segment file is deleted.

  The default value is 168.

  For more information, see the Apache Kafka® documentation.

• Log retention minutes All interfaces

  Time (in minutes) for Apache Kafka® to keep a log segment file. This setting applies if the Delete log cleanup policy is effective; once the specified timeout expires, the segment file is deleted.

  The maximum value is 2147483647. If no value is specified, the Log retention hours value is used.

  For more information, see the Apache Kafka® documentation.

• Log retention ms All interfaces

  Time (in milliseconds) for Apache Kafka® to keep a log segment file. This setting applies if the Delete log cleanup policy is effective; once the specified timeout expires, the segment file is deleted.

  The minimum value is -1 (logs are stored without any time limit); the maximum one is 9223372036854775807. If no value is specified, the Log retention minutes value is used.

  Warning

  If both Log retention bytes and Log retention ms are set to -1, the log grows indefinitely. In this case, the cluster can run out of storage space quickly.

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting corresponds to Retention, ms.

  For more information, see the Apache Kafka® documentation.

  See also the Log retention bytes setting.

• Log segment bytes All interfaces

  This setting determines the segment size for log files, in bytes.

  Segment files are saved and cleared file-by-file. The smaller the specified segment size, the more files will be in the log, while the storage management will be more flexible.

  The minimum value is 14; the maximum one is 2147483647; default: 1073741824 (1 GB).

  This is a global cluster-level setting. You can override it at the topic level.

  For more information, see section Maximum log segment size and the Apache Kafka® documentation.

• Message max bytes All interfaces

  Message maximum size (in bytes) set by the producer.

  The minimum value is 0 (no limit); the default one is 1048588. The setting value relates to the Replica fetch max bytes value as follows:
  replica.fetch.max.bytes >= message.max.bytes + 12

  This is a global cluster-level setting.

  For more information, see the Apache Kafka® documentation.

• Num partitions All interfaces

  The number of log partitions per topic in the cluster.

  This setting only applies to automatically created topics.

  The minimum and default value is 1.

  See also the Num partitions topic-level setting.

  For more information, see the Apache Kafka® documentation.

• Offsets retention minutes All interfaces

  Offsets retention period in minutes after a consumer group loses all its consumers.

  The minimum value is 1; the default one is 10080.

  This is a global cluster-level setting.

  For more information, see the Apache Kafka® documentation.

• Replica fetch max bytes All interfaces

  Size of the buffer for messages exchanged by brokers (in bytes).

  The minimum value is 0 (no limit); the default one is 1048576. The setting value relates to the Message max bytes value as follows:
  message.max.bytes <= replica.fetch.max.bytes - 12

  This is a global cluster-level setting.

  For more information, see the Apache Kafka® documentation.

• Sasl enabled mechanisms All interfaces

  Encryption tools that can be used when connecting to a cluster.

  You can select one or more values:

  Management console, Terraform, and API	CLI
  SASL_MECHANISM_SCRAM_SHA_256	SCRAM-SHA-256
  SASL_MECHANISM_SCRAM_SHA_512	SCRAM-SHA-512

  If the setting is not defined by the user, the only tool used is SCRAM-SHA-512.

  For more information, see the Apache Kafka® documentation.

• Socket receive buffer bytes Management console Terraform API

  Socket receive buffer size (in bytes).

  The minimum value is -1 (default; the OS settings are used); the maximum one is 2147483647.

  For more information, see the Apache Kafka® documentation.

• Socket send buffer bytes Management console Terraform API

  Socket send buffer size (in bytes).

  The minimum value is -1 (default; the OS settings are used); the maximum one is 2147483647.

  For more information, see the Apache Kafka® documentation.

• Ssl cipher suites All interfaces

  List of cluster encryption algorithms.

  There is no default value (all encryption algorithms are supported).

  This is a global cluster-level setting.

  For more information, see the Apache Kafka® documentation.

Settings for individual topicsSettings for individual topics

You can use the following settings:

• Cleanup policy All interfaces

  It determines the retention policy for old log messages:

  • Delete (CLEANUP_POLICY_DELETE for Terraform and API): Delete log segments when either their retention time or log size limit is reached.
  • Compact (CLEANUP_POLICY_COMPACT for Terraform and API): Compress log messages.
  • CompactAndDelete (CLEANUP_POLICY_COMPACT_AND_DELETE for Terraform and API): Both compress and delete log segments.

  See the full description of this setting in the Apache Kafka® documentation. In the management console, this setting corresponds to Cleanup policy.

• Compression type All interfaces

  Codecs used for message compression.

  For more information, see the description of the Compression type cluster-level setting.

  In the management console, this setting corresponds to Compression type.

  For more information, see the Apache Kafka® documentation.

• Delete delay, ms All interfaces

  This is the time to wait before deleting a file from the file system.

  In the management console, this setting corresponds to Delete delay, ms.

  See the full description of this setting in the Apache Kafka® documentation.

• Delete retention All interfaces

  This setting determines the time in milliseconds to retain delete tombstone markers for topics with compressed logs. This setting only applies if the log cleanup policy is set either to Compact or CompactAndDelete.

  See the full description of this setting in the Apache Kafka® documentation.

• Flush messages All interfaces

  This is the number of topic messages that can be kept in memory before these messages are flushed to the disk.

  For more information, see the description of the Log flush interval messages cluster-level setting.

  In the management console, this setting corresponds to Flush messages.

  See the full description of this setting in the Apache Kafka® documentation.

• Flush, ms All interfaces

  Maximum time in milliseconds a message can be stored in memory before being flushed to the disk.

  For more information, see the description of the Log flush interval ms cluster-level setting.

  In the management console, this setting corresponds to Flush, ms.

  See the full description of this setting in the Apache Kafka® documentation.

• Message max bytes All interfaces

  This is maximum size in bytes for the producer's message.

  The minimum value is 0 (no limit); default: 1048588. This setting value must not exceed the Replica fetch max bytes value by more than 12 bytes.

  In the management console, this setting corresponds to Maximum batch size.

  See the full description of this setting in the Apache Kafka® documentation.

• Min compaction lag, ms All interfaces

  It determines the minimum time a message will remain uncompressed in the log.

  In the management console, this setting corresponds to Min compaction lag, ms.

  See the full description of this setting in the Apache Kafka® documentation.

• Minimum number of in-sync replicas All interfaces

  This is the minimum number of replicas to wait for the record confirmation from in order to consider a message successfully written to a topic. Use this setting if the producer has to wait for too long for a successful write acknowledgement from all broker hosts in the cluster.

  The minimum value depends on the number of broker hosts:

  • For clusters with one broker host: 1.
  • For clusters with two or more broker hosts: 2.

  In the management console, this setting corresponds to Minimum number of in-sync replicas.

  See the full description of this setting in the Apache Kafka® documentation.

• Num partitions All interfaces

  It determines the number of log partitions per topic.

  The minimum and default value is 1.

  For more information, see the description of the Num partitions cluster-level setting.

  In the management console, this setting corresponds to Number of partitions.

  See the full description of this setting in the Apache Kafka® documentation.

• Pre-allocation of the segment file All interfaces

  This setting determines whether to pre-allocate space for log file segments. By default, the space for log segment files gets allocated as logs are filling up (false).

  For more information, see the description of the Log preallocate cluster-level setting.

  In the management console, this setting corresponds to Pre-allocation of the segment file.

  See the full description of this setting in the Apache Kafka® documentation.

• Replication factor All interfaces

  It determines the number of data copies for a topic.

  The minimum and default value is 1. The maximum value is equal to the number of broker hosts in the cluster.

  For more information, see the description of the Default replication factor cluster-level setting.

  In the management console, this setting corresponds to Replication factor.

  See the full description of this setting in the Apache Kafka® documentation.

• Retention, bytes All interfaces

  The maximum size a partition can grow to.

  For more information, see the description of the Log retention bytes cluster-level setting.

  In the management console, this setting corresponds to Retention, bytes.

  See the full description of this setting in the Apache Kafka® documentation.

• Retention, ms All interfaces

  Time (in milliseconds) for Apache Kafka® to keep a log segment file.

  For more information, see the description of the Log retention ms cluster-level setting.

  In the management console, this setting corresponds to Retention, ms.

  See the full description of this setting in the Apache Kafka® documentation.

• Segment bytes All interfaces

  This setting determines the segment size for log files, in bytes.

  Segment files are saved and cleared file-by-file. The smaller the specified segment size, the more files will be in the log, while the storage management will be more flexible.

  The minimum value is 14; default: 1073741824 (1 GB).

  For more information, see the description of the Log segment bytes cluster-level setting.

  For more information, see Maximum log segment size and the Apache Kafka® documentation.