Managing Apache Kafka® topics

Written by

Updated at October 17, 2024

Managing topics via Yandex Cloud interfaces
Managing topics via the Apache Kafka® Admin API

A Managed Service for Apache Kafka® cluster provides two ways for you to manage topics and partitions (which can be used separately or combined):

Using native Yandex Cloud interfaces, such as the CLI, API, or management console. Choose this method if you want to manage topics using Managed Service for Apache Kafka® features.

You can perform the following actions on Managed Service for Apache Kafka® topics:
Using the Apache Kafka® Admin API. Select this method if you prefer to use your existing solution to manage topics and partitions.

Managing topics via Yandex Cloud interfaces

Creating a topic

Prior to creating a topic, calculate the minimum storage size.

Management console

CLI

Terraform

API

In the management console, go to the appropriate folder.
In the list of services, select Managed Service for Kafka.
Click the cluster name and go to the Topics tab.
Click Create topic.
Under Basic parameters, set the basic parameters of the topic:
- Topic name (must be unique in the Apache Kafka® cluster).
- Number of topic partitions.
- Replication factor. This parameter value should not exceed the number of brokers in the cluster. The minimum value is 1. The maximum value is 3. The default value is:
  - For a cluster with one or two brokers: 1.
  - For a cluster with three or more brokers: 3.
Under Topic settings, specify the topic settings.
Click Create.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a topic:

View a description of the CLI create topic command:
```
yc managed-kafka topic create --help
```

Create a topic:

yc managed-kafka topic create <topic_name> \
  --cluster-name <cluster_name> \
  --partitions <number_of_partitions> \
  --replication-factor <replication_factor>

If necessary, specify the topic settings here.

Open the current Terraform configuration file with an infrastructure plan.

For more information about how to create this file, see Creating clusters.

Add the yandex_mdb_kafka_topic resource and configure the topic under topic_config if required:

resource "yandex_mdb_kafka_topic" "<topic_name>" {
  cluster_id         = "<cluster_ID>"
  name               = "<topic_name>"
  partitions         = <number_of_partitions>
  replication_factor = <replication_factor>
  topic_config {
    compression_type = "<compression_type>"
    flush_messages   = <maximum_number_of_messages_in_memory>
    ...
  }
}

Make sure the settings are correct.
1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
2. Run the command:
```
terraform validate
```
  If there are errors in the configuration files, Terraform will point to them.
Confirm updating the resources.
1. Run the command to view planned changes:
```
terraform plan
```
  If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
2. If you are happy with the planned changes, apply them:
  1. Run the command:
```
terraform apply
```
  2. Confirm the update of resources.
  3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

To create a topic, use the create REST API method for the Topic resource or the TopicService/Create gRPC API call and provide the following in the request:

In the clusterId parameter, the ID of the cluster where you want to create a topic. To find out the cluster ID, get a list of clusters in the folder.
Topic settings in the topicSpec parameter.

Note

While running, Managed Service for Apache Kafka® is able to create service topics. You cannot write user data to such topics.

Updating topic settings

You cannot reduce the number of partitions in Managed Service for Apache Kafka® topics. You cannot create new partitions if there is not enough storage space.

For more information, see Minimum storage size.

Management console

CLI

Terraform

API

In the management console, go to the appropriate folder.
In the list of services, select Managed Service for Kafka.
Click the cluster name and select the Topics tab.
Click for the topic you need and select Edit.
Change the basic parameters of the topic:
- Number of topic partitions.
- Replication factor. This parameter value should not exceed the number of brokers in the cluster. The minimum value is 1. The maximum value is 3. The default value is:
  - For a cluster with one or two brokers: 1.
  - For a cluster with three or more brokers: 3.
Change additional topic settings.
Click Save.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To update topic settings:

View a description of the CLI update topic command:
```
yc managed-kafka topic update --help
```

Change topic settings:

yc managed-kafka topic update <topic_name> \
  --cluster-name <cluster_name> \
  --partitions <number_of_partitions> \
  --replication-factor <replication_factor>

Open the current Terraform configuration file with an infrastructure plan.

For more information about how to create this file, see Creating clusters.

Edit the parameter values in the yandex_mdb_kafka_topic resource description:

resource "yandex_mdb_kafka_topic" "<topic_name>" {
  cluster_id         = "<cluster_ID>"
  name               = "<topic_name>"
  partitions         = <number_of_partitions>
  replication_factor = <replication_factor>
  topic_config {
    compression_type = "<compression_type>"
    flush_messages   = <maximum_number_of_messages_in_memory>
    ...
  }
}

Make sure the settings are correct.
1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
2. Run the command:
```
terraform validate
```
  If there are errors in the configuration files, Terraform will point to them.
Confirm updating the resources.
1. Run the command to view planned changes:
```
terraform plan
```
  If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
2. If you are happy with the planned changes, apply them:
  1. Run the command:
```
terraform apply
```
  2. Confirm the update of resources.
  3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

To change topic settings, use the update REST API method for the Topic resource or the TopicService/Update gRPC API call and provide the following in the request:

ID of the cluster in which the topic is located in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
Topic name in the topicName parameter. To find out the name, get a list of cluster topics.
New values of topic settings in the topicSpec parameter.
List of settings to update, in the updateMask parameter.

Warning

This API method overrides all parameters of the object being modified that were not explicitly passed in the request to the default values. To avoid this, list the settings you want to change in the updateMask parameter (one line separated by commas).

Getting a list of topics in a cluster

Management console

CLI

API

In the management console, go to the appropriate folder.
In the list of services, select Managed Service for Kafka.
Click the cluster name and go to the Topics tab.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of topics, run the following command:

yc managed-kafka topic list --cluster-name <cluster_name>

To get a list of topics in a cluster, use the list REST API method for the Topic resource or the TopicService/List gRPC API call and provide the cluster ID in the clusterId request parameter.

To find out the cluster ID, get a list of clusters in the folder.

Getting detailed information about a topic

Management console

CLI

API

In the management console, go to the appropriate folder.
In the list of services, select Managed Service for Kafka.
Click the cluster name and go to the Topics tab.
Click the topic name.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get detailed information about a topic, run the following command:

yc managed-kafka topic get <topic_name> --cluster-name <cluster_name>

To get topic details, use the get REST API method for the Topic resource or the TopicService/Get gRPC API call and provide the following in the request:

ID of the cluster in which the topic is located in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
Topic name in the topicName parameter. To find out the name, get a list of cluster topics.

Importing topics to Terraform

Using import, you can bring the existing cluster topics under Terraform management.

Terraform

In the Terraform configuration file, specify the topic you want to import:
```
resource "yandex_mdb_kafka_topic" "<topic_name>" {}
```
Run the following command to import the topic:
```
terraform import yandex_mdb_kafka_topic.<topic_name> <cluster_ID>:<topic_name>
```
To learn more about importing topics, see the Terraform provider documentation.

Deleting a topic

Note

Permissions granted to the user for a topic remain even after the topic is deleted. If, after deleting a topic, you don't revoke the permissions, then, when you create a topic with the same name, the user will have access to it even if you don't explicitly assign them new permissions.

Management console

CLI

Terraform

API

In the management console, go to the appropriate folder.
In the list of services, select Managed Service for Kafka.
Click the cluster name and go to the Topics tab.
Click for the topic you need and select Delete topic.
In the window that opens, click Delete.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To delete a topic:

View a description of the CLI update topic command:
```
yc managed-kafka topic delete --help
```

Delete a topic:

yc managed-kafka topic delete <topic_name> --cluster-name <cluster_name>

Open the current Terraform configuration file with an infrastructure plan.

For more information about how to create this file, see Creating clusters.
Delete the yandex_mdb_kafka_topic resource with the relevant topic description.
Make sure the settings are correct.
1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
2. Run the command:
```
terraform validate
```
  If there are errors in the configuration files, Terraform will point to them.
Confirm updating the resources.
1. Run the command to view planned changes:
```
terraform plan
```
  If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
2. If you are happy with the planned changes, apply them:
  1. Run the command:
```
terraform apply
```
  2. Confirm the update of resources.
  3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

To delete a topic, use the delete REST API method for the Topic resource or the TopicService/Delete gRPC API call and provide the following in the request:

ID of the cluster in which the topic is located in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
Topic name in the topicName parameter. To find out the name, get a list of cluster topics.

Managing topics via the Apache Kafka® Admin API

To manage topics via the Apache Kafka® Admin API:

Create an admin user with the ACCESS_ROLE_ADMIN role in the cluster.
Manage topics on behalf of this user by making requests to the Apache Kafka® Admin API. Review your favorite programming language manual for information on working with the Admin API.

For more information about using the Admin API and existing limitations, see Managing topics and partitions and the Apache Kafka® documentation.

Managing Apache Kafka® topics

Managing topics via Yandex Cloud interfacesManaging topics via Yandex Cloud interfaces

Creating a topicCreating a topic

Updating topic settingsUpdating topic settings

Getting a list of topics in a clusterGetting a list of topics in a cluster

Getting detailed information about a topicGetting detailed information about a topic

Importing topics to TerraformImporting topics to Terraform

Deleting a topicDeleting a topic

Managing topics via the Apache Kafka® Admin APIManaging topics via the Apache Kafka® Admin API

Was the article helpful?

Managing topics via Yandex Cloud interfaces

Creating a topic

Updating topic settings

Getting a list of topics in a cluster

Getting detailed information about a topic

Importing topics to Terraform

Deleting a topic

Managing topics via the Apache Kafka® Admin API