Apache Kafka® settings

For Managed Service for Apache Kafka® clusters, you can configure Apache Kafka® settings. Some settings are configured at the cluster level, while others, at the topic level.

The label next to the setting name shows which interface can be used to configure this setting: the management console, CLI, API, or Terraform. The All interfaces label means that all of the above interfaces are supported.

Depending on the selected interface, the same setting will be represented differently. For example, Compression type in the management console is the same as:

• --compression-type in the CLI
• compression_type in the gRPC API and Terraform
• compressionType in the REST API

Cluster-level settingsCluster-level settings

The following settings are available:

• Auto create topics enable All interfaces

  Manages automatic topic creation.

  This option is disabled by default (false).

  For more information, see this Apache Kafka® guide.

• Compression type All interfaces

  Codecs used for message compression:

  Management console, Terraform, and API	CLI	Description
  COMPRESSION_TYPE_UNCOMPRESSED	uncompressed	Compression is disabled.
  COMPRESSION_TYPE_ZSTD	zstd	zstd
  COMPRESSION_TYPE_LZ4	lz4	lz4
  COMPRESSION_TYPE_SNAPPY	snappy	snappy
  COMPRESSION_TYPE_GZIP	gzip	gzip
  COMPRESSION_TYPE_PRODUCER	producer	The codec is set by the producer.

  By default, the compression codec is set by the producer (COMPRESSION_TYPE_PRODUCER).

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting appears as Compression type.

  For more information, see this Apache Kafka® guide.

• Default replication factor All interfaces

  Number of data replicas per topic within a cluster.

  This setting only applies to automatically created topics.

  The minimum and default value is 1. The maximum value is equal to the number of broker hosts in the cluster.

  See also the Replication factor topic-level setting.

  For more information, see this Apache Kafka® guide.

• Log flush interval messages All interfaces

  Number of topic messages that can accumulate in memory before being flushed to disk. For example, if set to 1, each message will be flushed to disk once received. If set to 5, messages will be flushed to disk in groups of five.

  The minimum value is 1, the maximum and default one is 9223372036854775807.

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting appears as Flush messages.

  For more information, see the this Apache Kafka® guide.

• Log flush interval ms All interfaces

  Maximum time in milliseconds a message can be stored in memory before being flushed to disk. If no value is specified, the Log flush scheduler interval ms value is used.

  The maximum value is 9223372036854775807.

  This is a global cluster-level setting. You can override it at the topic level.

  For more information, see this Apache Kafka® guide.

• Log flush scheduler interval ms All interfaces

  Time period (in milliseconds) after which the log flusher checks for logs to flush to disk.

  The maximum and default value is 9223372036854775807.

  For more information, see this Apache Kafka® guide.

• Log preallocate All interfaces

  This setting determines whether to pre-allocate space for log segment files.

  By default, the space for log segment files is allocated as the segments get full (false).

  This is a global cluster-level setting. You can override it at the topic level.

  For more information, see this Apache Kafka® guide.

• Log retention bytes All interfaces

  Maximum size a partition can grow to. When a partition reaches this size, Apache Kafka® starts deleting old log segments. This setting applies if the Delete log cleanup policy is effective.

  The minimum and default value is -1, which means the log size is unlimited. The maximum value is 9223372036854775807.

  Use this setting if you need to manage the log size due to limited disk space.

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting appears as Retention, bytes.

  For more information, see this Apache Kafka® guide.

  See also the Log retention ms setting.

• Log retention hours All interfaces

  Time (in hours) for Apache Kafka® to keep a log segment file. This setting applies if the Delete log cleanup policy is effective; once the specified timeout expires, the segment file is deleted.

  The default value is 168.

  For more information, see this Apache Kafka® guide.

• Log retention minutes All interfaces

  Time (in minutes) for Apache Kafka® to keep a log segment file. This setting applies if the Delete log cleanup policy is effective; once the specified timeout expires, the segment file is deleted.

  The maximum value is 2147483647. If no value is specified, the Log retention hours value is used.

  For more information, see this Apache Kafka® guide.

• Log retention ms All interfaces

  Time (in milliseconds) for Apache Kafka® to keep a log segment file. This setting applies if the Delete log cleanup policy is effective; once the specified timeout expires, the segment file is deleted.

  The minimum value is -1, which means logs are stored without any time limit. The maximum value is 9223372036854775807. If no value is specified, the Log retention minutes value is used.

  Warning

  If both Log retention bytes and Log retention ms are set to -1, the log grows indefinitely. This can quickly exhaust the cluster storage space.

  This is a global cluster-level setting. You can override it at the topic level. In the management console, this setting appears as Retention, ms.

  For more information, see this Apache Kafka® guide.

  See also the Log retention bytes setting.

• Log segment bytes All interfaces

  This setting determines the segment size for log files, in bytes.

  Segment files are saved and cleared file-by-file. A smaller segment size results in more log files but offers more flexible storage management.

  The valid values range from 14 to 2147483647. The default value is 1073741824 (1 GB).

  This is a global cluster-level setting. You can override it at the topic level.

  For more information, see Maximum log segment size and this Apache Kafka® guide.

• Message max bytes All interfaces

  Maximum size of a message sent by the producer.

  The minimum value is 0 (no limit); the default one is 1048588. The setting value relates to the Replica fetch max bytes value as follows:
  replica.fetch.max.bytes >= message.max.bytes + 12

  This is a global cluster-level setting.

  For more information, see this Apache Kafka® guide.

• Num partitions All interfaces

  Number of log partitions per topic in the cluster.

  This setting only applies to automatically created topics.

  The minimum and default value is 1.

  See also the Num partitions topic-level setting.

  For more information, see this Apache Kafka® guide.

• Offsets retention minutes All interfaces

  Offset retention period for inactive consumer groups (in minutes).

  The minimum value is 1 and the default is 10080.

  This is a global cluster-level setting.

  For more information, see this Apache Kafka® guide.

• Replica fetch max bytes All interfaces

  Inter-broker message buffer size (in bytes).

  The minimum value is 0 (no limit); the default one is 1048576. The setting value relates to the Message max bytes value as follows:
  message.max.bytes <= replica.fetch.max.bytes - 12

  This is a global cluster-level setting.

  For more information, see this Apache Kafka® guide.

• Sasl enabled mechanisms All interfaces

  Encryption options for cluster connections.

  You can select one or more values:

  Management console, Terraform, and API	CLI
  SASL_MECHANISM_SCRAM_SHA_256	SCRAM-SHA-256
  SASL_MECHANISM_SCRAM_SHA_512	SCRAM-SHA-512

  If the setting is not defined by the user, SCRAM-SHA-512 is only used.

  For more information, see this Apache Kafka® guide.

• Socket receive buffer bytes Management console Terraform API

  Socket receive buffer size (in bytes).

  The minimum and default value is -1, which means the OS settings apply. The maximum value is 2147483647.

  For more information, see this Apache Kafka® guide.

• Socket send buffer bytes Management console Terraform API

  Socket send buffer size (in bytes).

  The minimum and default value is -1, which means the OS settings apply. The maximum value is 2147483647.

  For more information, see this Apache Kafka® guide.

• Ssl cipher suites All interfaces

  List of cluster encryption algorithms.

  There is no default value, which means all supported encryption algorithms are enabled.

  This is a global cluster-level setting.

  For more information, see this Apache Kafka® guide.

Settings for individual topicsSettings for individual topics

The following settings are available:

• Cleanup policy All interfaces

  Retention policy for old log messages:

  • Delete (CLEANUP_POLICY_DELETE for Terraform and API): Delete log segments when either their retention time or log size limit is reached.
  • Compact (CLEANUP_POLICY_COMPACT for Terraform and API): Compact the log.
  • CompactAndDelete (CLEANUP_POLICY_COMPACT_AND_DELETE for Terraform and API): Both compact the log and delete log segments.

  For a full description of this setting, see this Apache Kafka® guide. In the management console, this setting appears as Cleanup policy.

• Compression type All interfaces

  Codec used for message compression.

  For more information, see the description of the Compression type cluster-level setting.

  In the management console, this setting appears as Compression type.

  For a full description of this setting, see this Apache Kafka® guide.

• Delete delay, ms All interfaces

  Time to wait before deleting a file from the file system.

  In the management console, this setting appears as Delete delay, ms.

  For a full description of this setting, see this Apache Kafka® guide.

• Delete retention All interfaces

  Time (in milliseconds) to retain delete tombstone markers for topics with a compacted log. This setting only applies if the log cleanup policy is set either to Compact or CompactAndDelete.

  For a full description of this setting, see this Apache Kafka® guide.

• Flush messages All interfaces

  Number of topic messages that can accumulate in memory before being flushed to disk.

  For more information, see the description of the Log flush interval messages cluster-level setting.

  In the management console, this setting appears as Flush messages.

  For a full description of this setting, see this Apache Kafka® guide.

• Flush, ms All interfaces

  Maximum time in milliseconds a message can be stored in memory before being flushed to disk.

  For more information, see the description of the Log flush interval ms cluster-level setting.

  In the management console, this setting appears as Flush, ms.

  For a full description of this setting, see this Apache Kafka® guide.

• Message max bytes All interfaces

  Maximum size of a message sent by the producer.

  The minimum value is 0 (no limit); the default one is 1048588. The value of this setting must not exceed the Replica fetch max bytes value by more than 12 bytes.

  In the management console, this setting appears as Maximum batch size.

  For a full description of this setting, see this Apache Kafka® guide.

• Min compaction lag, ms All interfaces

  Minimum time a message remains uncompacted in the log.

  In the management console, this setting appears as Min compaction lag, ms.

  For a full description of this setting, see this Apache Kafka® guide.

• Minimum number of in-sync replicas All interfaces

  Minimum number of replicas that have to acknowledge a write to consider a message successfully written to a topic. Use this setting if the producer waits too long for write acknowledgments from all broker hosts in the cluster.

  The minimum value depends on the number of broker hosts:

  • For clusters with one broker host: 1.
  • For clusters with two or more broker hosts: 2.

  In the management console, this setting appears as Minimum number of in-sync replicas.

  For a full description of this setting, see this Apache Kafka® guide.

• Num partitions All interfaces

  Number of log partitions per topic.

  The minimum and default value is 1.

  For more information, see the description of the Num partitions cluster-level setting.

  In the management console, this setting appears as Number of partitions.

  For a full description of this setting, see this Apache Kafka® guide.

• Pre-allocation of the segment file All interfaces

  This setting determines whether to pre-allocate space for log segment files. By default, the space for log segment files is allocated as the segments get full (false).

  For more information, see the description of the Log preallocate cluster-level setting.

  In the management console, this setting appears as Pre-allocation of the segment file.

  For a full description of this setting, see this Apache Kafka® guide.

• Replication factor All interfaces

  Number of data replicas per topic.

  The minimum and default value is 1. The maximum value is equal to the number of broker hosts in the cluster.

  For more information, see the description of the Default replication factor cluster-level setting.

  In the management console, this setting appears as Replication factor.

  For a full description of this setting, see this Apache Kafka® guide.

• Retention, bytes All interfaces

  Maximum size a partition can grow to.

  For more information, see the description of the Log retention bytes cluster-level setting.

  In the management console, this setting appears as Retention, bytes.

  For a full description of this setting, see this Apache Kafka® guide.

• Retention, ms All interfaces

  Time (in milliseconds) for Apache Kafka® to keep a log segment file.

  For more information, see the description of the Log retention ms cluster-level setting.

  In the management console, this setting appears as Retention, ms.

  For a full description of this setting, see this Apache Kafka® guide.

• Segment bytes All interfaces

  This setting determines the segment size for log files, in bytes.

  Segment files are saved and cleared file-by-file. A smaller segment size results in more log files but offers more flexible storage management.

  The minimum value is 14 and the default is 1073741824 (1 GB).

  For more information, see the description of the Log segment bytes cluster-level setting.

  For more information, see Maximum log segment size and this Apache Kafka® guide.