Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Managing Apache Kafka® users

Written by
Updated at November 18, 2024

Users in Apache Kafka®:

  • Keep the access permissions of data producers and consumers separate.

    A producer or consumer can only access topics that are allowed for their users. You can use the same user for multiple producers or consumers: the former get the rights to write data to certain topics and the latter get the read rights.

  • Manage topics. For more information, see Topics and partitions.

After creating an Apache Kafka® cluster, you can:

Getting a list of users in a cluster

  1. In the management console, go to the relevant folder.
  2. In the list of services, select Managed Service for Kafka.
  3. Click the cluster name and go to the Users tab.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

  1. To get a list of users, run the following command:

    yc managed-kafka user list --cluster-name <cluster_name>

  2. To get detailed information for a specific user, run this command:

    yc managed-kafka user get <username> --cluster-name <cluster_name>

To find out the cluster name, get a list of clusters in the folder.

To get a list of users, use the list REST API method for the User resource or the UserService/List gRPC API call and provide the ID of the cluster you need in the clusterId request parameter.

To find out the cluster ID, get a list of clusters in the folder.

Creating a user

Note

Use the CLI, API, or Terraform to create an admin user.

To create a user for a producer or consumer in a cluster:

  1. In the management console, go to the relevant folder.

  2. In the list of services, select Managed Service for Kafka.

  3. Click the cluster name and go to the Users tab.

  4. Click Create user.

  5. Enter your username and password.

    Note

    The username may contain Latin letters, numbers, hyphens, and underscores but must begin with a letter or underscore.

    The password must be from 8 to 128 characters long.

  6. Grant access permissions for the relevant topics.

  7. Click Create.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a user:

  1. View a description of the CLI create user command:

    yc managed-kafka user create --help

  2. Create a user with the producer role for the producer or the consumer role for the consumer and grant access permissions for the relevant topics:

    yc managed-kafka user create <username> \
  --cluster-name <cluster_name> \
  --password <password> \
  --permission topic=<topic_name>,role=<user_role>

To create an admin user to manage cluster topics:

  1. See the description of the create user CLI command:

    yc managed-kafka user create --help

  2. Create a user with the admin role valid for all (*) cluster topics:

    yc managed-kafka user create <username> \
  --cluster-name <cluster_name> \
  --password <password> \
  --permission topic=*,role=admin

Note

The username may contain Latin letters, numbers, hyphens, and underscores but must begin with a letter or underscore.

The password must be from 8 to 128 characters long.

To find out the cluster name, get a list of clusters in the folder.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Add the yandex_mdb_kafka_user resource:

    resource "yandex_mdb_kafka_user" "<username>" {
  cluster_id = "<cluster_ID>"
  name       = "<username>"
  password   = "<password>"
  ...
}

    Note

    The username may contain Latin letters, numbers, hyphens, and underscores but must begin with a letter or underscore.

    The password must be from 8 to 128 characters long.

  3. Grant access permissions for the relevant topics.

  4. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  5. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To create a user, use the create REST API method for the User resource or the UserService/Create gRPC API call and provide the following in the request:

  • Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
  • User settings in the userSpec parameter:
    • Username in the name parameter.
    • User password in the password parameter.
    • Topic access permissions (one or more permissions parameters, one for each topic):
      • Topic name in the topicName parameter. To find out the name, get a list of cluster topics.
      • Topic access permissions in the role parameter: ACCESS_ROLE_PRODUCER for the producer or ACCESS_ROLE_CONSUMER for the consumer.

To create an admin user to manage cluster topics, provide the following values under permission in the userSpec parameter when creating the user:

  • topicName: *.
  • role: ACCESS_ROLE_ADMIN.

Note

The username may contain Latin letters, numbers, hyphens, and underscores but must begin with a letter or underscore.

The password must be from 8 to 128 characters long.

Changing user settings

  1. In the management console, go to the relevant folder.

  2. In the list of services, select Managed Service for Kafka.

  3. Click the cluster name and go to the Users tab.

  4. Click for the appropriate user and select:

  5. Click Save.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

Using the CLI, you can change a user's password, grant or revoke topic access permissions.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. In this file, locate the yandex_mdb_kafka_user resource for the user and make the changes.

    Using Terraform, you can change a user's password, grant or revoke topic access permissions.

To update user settings, use the update REST API method for the User resource or the UserService/Update gRPC API call and provide the following in the request:

  • Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
  • Username in the userName parameter. To find out the name, get a list of users in the cluster.
  • A list of settings to update (single line, comma-separated) in the updateMask parameter. If you skip this parameter, the API method will reset any user settings that are not explicitly specified in the request to their default values.
  • New set of permissions to topics (one or more permissions parameters, one for each topic).

Warning

The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

You can also use the update method to change a user's password, and the grantPermission and revokePermission methods to grant or revoke topic access permissions.

Changing a user password

  1. In the management console, go to the relevant folder.
  2. In the list of services, select Managed Service for Kafka.
  3. Click the cluster name and go to the Users tab.
  4. Click for the appropriate user and select Change password.
  5. Set a new password and click Edit.

Note

The password must be between 8 and 128 characters.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the user password, run this command:

yc managed-kafka user update <username> \
  --cluster-name <cluster_name> \
  --password <new_password>

Note

The password must be between 8 and 128 characters.

To find out the cluster name, get a list of clusters in the folder.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Locate the user's yandex_mdb_kafka_user resource in the file.

  3. Change the value of the password field:

    resource "yandex_mdb_kafka_user" "<username>" {
  ...
  password = "<password>"
  ...
}

    Note

    The password must be between 8 and 128 characters.

  4. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  5. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To update a user's password, use the update REST API method for the User resource or the UserService/Update gRPC API call and provide the following in the request:

  • Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.

  • Username in the userName parameter. To find out the name, get a list of users in the cluster.

  • New user password in the password parameter.

  • Name of the password setting in the updateMask parameter.

    Note

    The password must be between 8 and 128 characters.

Warning

The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

Granting user permissions

Note

Permissions granted to the user for a topic remain even after the topic is deleted. If, after deleting a topic, you don't revoke the permissions, then, when you create a topic with the same name, the user will have access to it even if you don't explicitly assign them new permissions.

  1. In the management console, go to the relevant folder.

  2. In the list of services, select Managed Service for Kafka.

  3. Select the cluster.

  4. Go to the Users tab.

  5. Click for the user you need to issue topic permissions to and select Configure.

  6. Click Add topic. If there is no such button, it means that the user got permissions for all existing cluster topics.

    If the user does not need permissions to certain topics, you can revoke them.

  7. Select the appropriate topic from the drop-down list or enter its name:

    1. Specify the following in the Topic field:

      • * to allow access to any topics.
      • Full topic name to allow access to a specific topic.
      • <prefix>* to grant access to topics whose names start with the specified prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you specify topic*, access will be granted for topic_a1 and topic_a2.

    2. Click Add topic.

  8. Click the icon in the Roles column for the topic you need and select a role:

    • ACCESS_ROLE_CONSUMER: Access to the topic will be allowed to consumers logged in as this user.
    • ACCESS_ROLE_PRODUCER: Access to the topic will be allowed to producers logged in as this user.
    • ACCESS_ROLE_ADMIN: Only available if access to all topics is selected.

    You can select the ACCESS_ROLE_CONSUMER and ACCESS_ROLE_PRODUCER roles at the same time to make the user suitable for both producers and consumers.

    The user also gains access to data schema subjects. The availability of subjects depends on the specified roles and topics. For more information, see Managed Schema Registry subjects.

  9. To grant permissions to other topics, repeat the steps.

  10. (Optional) If you granted permissions for a topic accidentally, revoke them.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To grant user permissions:

  1. Retrieve a list of cluster topics:

    yc managed-kafka topic list --cluster-name <cluster_name>

  2. Grant access permissions for required topics by providing the --permission parameters:

    yc managed-kafka user update <username> \
  --cluster-name <cluster_name> \
  --permission topic=<topic_name>,role=<user_role>

    The following --permission parameters are available:

    • topic: Name of the topic to issue permissions for.

      If the user does not need permissions to certain topics, you can revoke them.

    • role: User’s role, producer, consumer, or admin.

      The admin role is only available if all topics are selected topic=*).

    When you update user permissions, existing permissions get revoked and replaced with new ones. This means the command must always include a complete list of permissions to be assigned to the user.

    For example, to grant permissions to a user named test-user in the kafka-cli cluster for the topic2 topic with the producer role, while keeping the existing topic1 permissions, run this command:

    yc managed-kafka user update test-user \
  --cluster-name kafka-cli \
  --permission topic=topic1,role=consumer \
  --permission topic=topic2,role=producer

    Along with access to the topic, users also gain access to data schema subjects. The availability of subjects depends on the specified roles and topics. For more information, see Managed Schema Registry subjects.

To find out the cluster name, get a list of clusters in the folder.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Locate the user's yandex_mdb_kafka_cluster resource in the file.

  3. Add the permission section:

    resource "yandex_mdb_kafka_user" "<username>" {
  ...
  permission {
    topic_name = "<topic>"
    role       = "<user's_role>"
  }
}

    Where:

    • topic_name: Topic name. Specify the following:

      • * to allow access to any topics.
      • Full topic name to allow access to a specific topic.
      • <prefix>* to grant access to topics whose names start with the specified prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you specify topic*, access will be granted for topic_a1 and topic_a2.

    • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topic_name = "*").

    Along with access to the topic, users also gain access to data schema subjects. The availability of subjects depends on the specified roles and topics. For more information, see Managed Schema Registry subjects.

    If the user does not need permissions to certain topics, you can revoke them.

  4. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  5. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To grant user permissions, use the grantPermission REST API method for the User resource or the UserService/GrantPermission gRPC API call and provide the following in the request:

Along with access to the topic, users also gain access to data schema subjects. The availability of subjects depends on the specified roles and topics. For more information, see Managed Schema Registry subjects.

Revoking user permissions

If you revoke the ACCESS_ROLE_ADMIN role from the admin user in a cluster, you will no longer be able to manage topics. Do not revoke this role without first granting it to another user.

  1. In the management console, go to the relevant folder.
  2. In the list of services, select Managed Service for Kafka.
  3. Select the cluster.
  4. Go to the Users tab.
  5. Click for the appropriate user and select Configure.
  6. Find the appropriate topic in the list of topics.
  7. Delete the role you no longer need: click the icon next to its name. To revoke all permissions for a topic, delete it from the list: hover over the topic name and click at the end of the line.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To revoke access permissions for specific topics, provide an updated list of --permission parameters:

yc managed-kafka user update <username> \
  --cluster-name <cluster_name> \
  --permission topic=<topic_name>,role=<user's_role>

When you update user permissions, existing permissions get revoked and replaced with new ones. This means the command must always include a complete list of permissions to be assigned to the user.

The --permission flag must include at least one topic-role pair, where:

  • topic: Topic name.
  • role: User’s role, producer, consumer, or admin.

To find out the cluster name, get a list of clusters in the folder.

To revoke all the permissions granted to the user, use the console or delete the user.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Locate the user's yandex_mdb_kafka_user resource in the file.

  3. Edit or delete the permission section.

  4. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  5. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To revoke user permissions, use the revokePermission REST API method for the User resource or the UserService/RevokePermission gRPC API call and provide the following in the request:

Importing users to Terraform

Using import, you can bring the existing cluster users under Terraform management.

  1. In the Terraform configuration file, specify the user you want to import:

    resource "yandex_mdb_kafka_user" "<username>" {}

  2. Run the following command to import the user:

    terraform import yandex_mdb_kafka_user.<username> <cluster_ID>:<username>

    To learn more about importing users, see the Terraform provider documentation.

Deleting a user

If you delete the admin user with the ACCESS_ROLE_ADMIN role in a cluster, you will no longer be able to manage topics. To avoid this, assign this role to another user before deleting the admin user.

  1. In the management console, go to the relevant folder.
  2. In the list of services, select Managed Service for Kafka.
  3. Click the cluster name and go to the Users tab.
  4. Click for the appropriate user and select Delete.
  5. In the window that opens, click Delete.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To remove a user, run:

yc managed-kafka user delete <username> --cluster-name <cluster_name>

To find out the cluster name, get a list of clusters in the folder.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Delete the user's yandex_mdb_kafka_user resource.

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To delete a user, use the delete REST API method for the User resource or the UserService/Delete gRPC API call and provide the following in the request:

Previous
Managing topics
Next
Managing connectors