Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Deploying the UI for Apache Kafka®

Written by
Updated at May 5, 2025

You can install the UI for Apache Kafka® for your Managed Service for Apache Kafka® cluster. With a web interface, you can track data streams, troubleshoot, manage brokers, cluster, producers, and consumers.

You can deploy the UI for Apache Kafka® in two ways:

  • In a Docker container on a Yandex Cloud virtual machine. This option is cheaper but less reliable, which makes it more suitable for getting started with the UI for Apache Kafka®.
  • In a Yandex Managed Service for Kubernetes cluster. This option is more expensive and more reliable, which makes it suitable for consistent and long-term use of the web interface.

Deploying in Docker containers

To deploy the UI for Apache Kafka® in a Docker container:

  1. Install additional dependencies.
  2. Create a TrustStore.
  3. Set up the UI for Apache Kafka®.

If you no longer need the resources you created, delete them.

The support cost includes:

  • Managed Service for Apache Kafka® cluster fee: using computing resources allocated to hosts (including ZooKeeper hosts) and disk space (see Apache Kafka® pricing).
  • VM fee: using computing resources, operating system, and storage (see Compute Cloud pricing).
  • Fee for public IP addresses for VMs and cluster hosts if public access is enabled for them (see Virtual Private Cloud pricing).

Getting started

Set up your infrastructure:

  1. Configure a security group for your Managed Service for Apache Kafka® cluster and VM so that you can connect to topics from a cloud-based VM.

  2. Create a Managed Service for Apache Kafka® cluster. When creating it, specify the configured security group.

  3. Create an Apache Kafka® user.

  4. In the network hosting the Managed Service for Apache Kafka® cluster, create a VM running Ubuntu 22.04 with a public IP address, and the configured security group.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the kafka-ui-via-docker.tf configuration file to the same working directory.

    This file describes:

    • Network.

    • Subnet.

    • VM running Ubuntu 22.04.

    • Default security group and rules required to connect to the cluster and VM from the internet.

    • Managed Service for Apache Kafka® cluster.

    • Apache Kafka® user.

  6. In the kafka-ui-via-docker.tf file, specify the values of variables.

  7. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Install additional dependencies

  1. Connect to the VM via SSH:

    ssh <username>@<VM_public_IP_address>

    Where <username> is the VM account username. You can find the VM's public IP address on the VM page in the management console.

  2. To check that the Managed Service for Apache Kafka® cluster is available, connect to one of its hosts with the KAFKA role:

    telnet <host_FQDN> 9091

    You can view the FQDN in the management console:

    1. Go to the cluster page.
    2. Navigate to Hosts.
    3. In the row of the host with the KAFKA role, copy the value of the Host FQDN column.

    If the cluster is available, you will get this message:

    Connected to <host_FQDN>

    After this, you can abort the command as it does not complete but awaits data transfer.

  3. Install Docker:

    sudo apt update && sudo apt install docker.io

  4. Install keytool to manage keys and certificates:

    sudo apt install openjdk-19-jre-headless

Create a TrustStore

When deploying Apache Kafka® in a Docker container, TrustStore commands run on a VM.

TrustStore is a trusted certificate store used in JKS files. It serves for authenticating a client when connecting to the server. The server validates the client using certificates stored in TrustStore. However, the client stores the private key and the certificate on their side in KeyStore.

In the example below, TrustStore is used to connect to a Managed Service for Apache Kafka® cluster. With no TrustStore created, the Apache Kafka® web interface will lack information about the cluster.

To use TrustStore:

  1. Create an SSL certificate:

    sudo mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
sudo wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexCA.crt && \
sudo chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexCA.crt

  2. Create a directory named /truststore:

    mkdir /truststore

    It will store the truststore.jks file. You need a separate directory so that the file path is correctly recognized in commands and configuration files.

  3. Upload the YandexCA.crt certificate to the truststore.jks file:

    sudo keytool -import \
             -file /usr/local/share/ca-certificates/Yandex/YandexCA.crt \
             -alias "kafka-ui-cert" \
             -keystore /truststore/truststore.jks

    You will be prompted to create a password. Memorize it: you will need it to deploy the Apache Kafka® web interface.

Set up the UI for Apache Kafka®

  1. On the VM, run the Docker container to deploy your web interface in:

    sudo docker run -it -p 8080:8080 \
   -e DYNAMIC_CONFIG_ENABLED=true \
   -e KAFKA_CLUSTERS_0_NAME=<cluster_name> \
   -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=<host_FQDN>:9091 \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=PLAIN \
   -e KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP=use_all_dns_ips \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="<user_login>" password="<user_password>";' \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION=/truststore/truststore.jks \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD=<password_for_TrustStore> \
   -v /truststore/truststore.jks:/truststore/truststore.jks \
   provectuslabs/kafka-ui

    Specify the following in the environment variables:

    • KAFKA_CLUSTERS_0_NAME: Managed Service for Apache Kafka® cluster name.
    • KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: FQDN of the host with the KAFKA role in the Managed Service for Apache Kafka® cluster.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, username: Apache Kafka® user login.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, password: Apache Kafka® user password.
    • KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: Password you set when creating the truststore.jks file.

    The command does not complete after you run it. While it is running, the UI for Apache Kafka® is available.

  2. On a local machine, go to http://<VM_public_IP_address>:8080 in your browser. The UI for Apache Kafka® with Managed Service for Apache Kafka® cluster data will open.

    You can find the VM's public IP address in the management console, on the VM page.

Deploying in a Managed Service for Kubernetes cluster

To deploy the UI for Apache Kafka® in a Managed Service for Kubernetes cluster:

  1. Install additional dependencies.
  2. Create a TrustStore.
  3. Deploy your application with the UI for Apache Kafka® in the Kubernetes pod.
  4. Check the result.

If you no longer need the resources you created, delete them.

The support cost includes:

  • Managed Service for Apache Kafka® cluster fee: using computing resources allocated to hosts (including ZooKeeper hosts) and disk space (see Apache Kafka® pricing).
  • Fee for the Managed Service for Kubernetes cluster: using the master and outgoing traffic (see Managed Service for Kubernetes pricing).
  • Fee for Managed Service for Kubernetes cluster nodes (VM): using computing resources, operating system, and storage (see Compute Cloud pricing).
  • Fee for public IP addresses for Managed Service for Apache Kafka® cluster hosts and Managed Service for Kubernetes cluster nodes if public access is enabled for them (see Virtual Private Cloud pricing).

Getting started

Set up your infrastructure:

  1. Configure a single security group:

  2. Create a Managed Service for Apache Kafka® cluster. When creating it, specify the configured security group.

  3. Create an Apache Kafka® user.

  4. In the network hosting the Managed Service for Apache Kafka® cluster, create a Managed Service for Kubernetes cluster. When creating it, specify the configured security group and assign a public address to the cluster.

  5. Create a node group in the Managed Service for Kubernetes cluster. When creating it, specify the configured security group.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the kafka-ui-via-kubernetes.tf configuration file to the same working directory.

    This file describes:

    • Network.

    • Subnet.

    • Default security group and rules required to connect to the following from the internet:

      • Managed Service for Apache Kafka® cluster.
      • Managed Service for Kubernetes cluster.
      • Managed Service for Kubernetes node group.

    • Managed Service for Apache Kafka® cluster.

    • Apache Kafka® user.

    • Managed Service for Kubernetes cluster.

    • Managed Service for Kubernetes node group.

  6. Specify the values of variables in the kafka-ui-via-kubernetes.tf file.

  7. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Install additional dependencies

On a local machine:

  1. Install kubect and configure it to work with the new cluster.

  2. To check that the Managed Service for Apache Kafka® cluster is available, connect to one of its hosts with the KAFKA role:

    telnet <host_FQDN> 9091

    You can view the FQDN in the management console:

    1. Go to the cluster page.
    2. Navigate to Hosts.
    3. In the row of the host with the KAFKA role, copy the value of the Host FQDN column.

    If the cluster is available, you will get this message:

    Connected to <host_FQDN>

    After this, you can abort the command as it does not complete but awaits data transfer.

  3. Install keytool to manage keys and certificates:

    sudo apt update && sudo apt install openjdk-19-jre-headless

Create a TrustStore

When deploying Apache Kafka® in a Managed Service for Kubernetes cluster, TrustStore commands run on a local machine.

TrustStore is a trusted certificate store used in JKS files. It serves for authenticating a client when connecting to the server. The server validates the client using certificates stored in TrustStore. However, the client stores the private key and the certificate on their side in KeyStore.

In the example below, TrustStore is used to connect to a Managed Service for Apache Kafka® cluster. With no TrustStore created, the Apache Kafka® web interface will lack information about the cluster.

To use TrustStore:

  1. Create an SSL certificate:

    sudo mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
sudo wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexCA.crt && \
sudo chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexCA.crt

  2. Create a directory named /truststore:

    mkdir /truststore

    It will store the truststore.jks file. You need a separate directory so that the file path is correctly recognized in commands and configuration files.

  3. Upload the YandexCA.crt certificate to the truststore.jks file:

    sudo keytool -import \
             -file /usr/local/share/ca-certificates/Yandex/YandexCA.crt \
             -alias "kafka-ui-cert" \
             -keystore /truststore/truststore.jks

    You will be prompted to create a password. Memorize it: you will need it to deploy the Apache Kafka® web interface.

Deploy your application with the UI for Apache Kafka® in the Kubernetes pod

  1. To deliver the truststore.jks file to the Kubernetes pod, create a secret containing this file:

    kubectl create secret generic truststore --from-file=/truststore/truststore.jks

  2. Create a file named kafka-ui-configMap.yaml with the configMap configuration. It contains information about the Managed Service for Apache Kafka® cluster and TrustStore:

    apiVersion: v1
kind: ConfigMap
metadata:
   name: kafka-ui-values
data:
   KAFKA_CLUSTERS_0_NAME: <Apache Kafka®_cluster_name>
   KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: <host_FQDN>:9091
   KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
   KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
   KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP: use_all_dns_ips
   KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="<user_login>" password="<user_password>";'
   KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION: /truststore/truststore.jks
   KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: <password_for_TrustStore>
   AUTH_TYPE: "DISABLED"
   MANAGEMENT_HEALTH_LDAP_ENABLED: "FALSE"

    Specify the following in the environment variables:

    • KAFKA_CLUSTERS_0_NAME: Managed Service for Apache Kafka® cluster name.
    • KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: FQDN of the host with the KAFKA role in the Managed Service for Apache Kafka® cluster.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, username: Apache Kafka® user login.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, password: Apache Kafka® user password.
    • KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: Password you set when creating the truststore.jks file.

  3. Create a file named kafka-ui-pod.yaml with the configuration of the pod in which to deploy your application with the Apache Kafka® UI:

    apiVersion: v1
kind: Pod
metadata:
   name: kafka-ui-pod
spec:
   containers:
   - name: kafka-ui-pod
     image: provectuslabs/kafka-ui
     envFrom:
        - configMapRef:
             name: kafka-ui-values
     volumeMounts:
     - name: truststore
       mountPath: "/truststore"
       readOnly: true
   volumes:
   - name: truststore
     secret:
        secretName: truststore
   - name: kafka-ui-configmap
     configMap:
        name: kafka-ui-values

  4. Apply the configMap configuration:

    kubectl apply -f kafka-ui-configMap.yaml

  5. Apply the pod configuration:

    kubectl apply -f kafka-ui-pod.yaml

Check the result

  1. View the pod logs to make sure the UI for Apache Kafka® is deployed successfully:

    kubectl logs kafka-ui-pod

    The result contains the following lines:

     _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
 \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
                                 |_|                                             

2024-01-23 12:13:25,648 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 8.0.0.Final
2024-01-23 12:13:25,745 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.6 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2024-01-23 12:13:25,746 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v3.0.6, Spring v6.0.8
2024-01-23 12:13:25,747 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"

  2. Assign the Apache Kafka® UI to port 8080:

    kubectl --namespace default port-forward kafka-ui-pod 8080:8080

  3. In the browser, go to http://127.0.0.1:8080/. The UI for Apache Kafka® with Managed Service for Apache Kafka® cluster data will open.

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Delete:

  1. Managed Service for Apache Kafka® cluster
  2. Virtual machine
  3. Managed Service for Kubernetes node group
  4. Managed Service for Kubernetes cluster

  1. In the terminal window, go to the directory containing the infrastructure plan.

    Warning

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

  2. Delete resources:

    1. Run this command:

      terraform destroy

    2. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.

Previous
All tutorials
Next
Configuring Kafka Connect to work with Managed Service for Apache Kafka®