Contact UsGet started
© 2024 Direct Cursus Technology L.L.C.

Deploying the Apache Kafka® web interface

Written by
Updated at September 6, 2024

You can install the UI for Apache Kafka® for your Managed Service for Apache Kafka® cluster. With a web interface, you can track data streams, troubleshoot, manage brokers, cluster, producers, and consumers.

You can deploy the UI for Apache Kafka® in two ways:

  • In a Docker container on a Yandex Cloud virtual machine. This option is cheaper but less reliable, which makes it more suitable for getting started with the UI for Apache Kafka®.
  • In a Yandex Managed Service for Kubernetes cluster. This option is more expensive and more reliable, which makes it suitable for consistent and long-term use of the web interface.

Deploying in Docker containers

To deploy the UI for Apache Kafka® in a Docker container:

  1. Install additional dependencies.
  2. Create a TrustStore.
  3. Set up the UI for Apache Kafka®.

If you no longer need the resources you created, delete them.

Getting started

Prepare the infrastructure:

  1. Configure a security group for your Managed Service for Apache Kafka® cluster and VM so that you can connect to topics from a cloud-based VM.

  2. Create a Managed Service for Apache Kafka® cluster. When creating it, specify the configured security group.

  3. Create an Apache Kafka® user.

  4. In the network hosting the Managed Service for Apache Kafka® cluster, create a VM running Ubuntu 22.04 with a public IP address, and the configured security group.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the kafka-ui-via-docker.tf configuration file to the same working directory.

    This file describes:

    • Network.

    • Subnet.

    • VM running Ubuntu 22.04.

    • Default security group and rules required to connect to the cluster and VM from the internet.

    • Managed Service for Apache Kafka® cluster.

    • Apache Kafka® user.

  6. Specify the variable values in the kafka-ui-via-docker.tf file.

  7. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Install additional dependencies

  1. Connect to the VM via SSH:

    ssh <username>@<VM_public_IP_address>

    Where <username> is the VM account username. You can find the VM's public IP address in the management console, on the VM page.

  2. To check that the Managed Service for Apache Kafka® cluster is available, connect to one of its hosts with the KAFKA role:

    telnet <host_FQDN> 9091

    You can view the FQDN in the management console:

    1. Go to the cluster page.
    2. Go to Hosts.
    3. Copy the value in the Host FQDN column, in the row of the host with the KAFKA role.

    If the cluster is available, you will get this message:

    Connected to <host_FQDN>

    After this, you can abort the command as it does not complete but awaits data transfer.

  3. Install Docker:

    sudo apt update && sudo apt install docker.io

  4. Install keytool to manage keys and certificates:

    sudo apt install openjdk-19-jre-headless

Create a TrustStore

When deploying Apache Kafka® in a Docker container, TrustStore commands run on a VM.

TrustStore is a trusted certificate store used in JKS files. It serves for authenticating a client when connecting to the server. The server validates the client using certificates stored in TrustStore. However, the client stores the private key and the certificate on their side in KeyStore.

In the example below, TrustStore is used to connect to a Managed Service for Apache Kafka® cluster. With no TrustStore created, the Apache Kafka® web interface will lack information about the cluster.

To use TrustStore:

  1. Create an SSL certificate:

    sudo mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
sudo wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexCA.crt && \
sudo chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexCA.crt

  2. Create a directory named /truststore:

    mkdir /truststore

    It will store the truststore.jks file. You need a separate directory so that the file path is correctly recognized in commands and configuration files.

  3. Upload the YandexCA.crt certificate to the truststore.jks file:

    sudo keytool -import \
             -file /usr/local/share/ca-certificates/Yandex/YandexCA.crt \
             -alias "kafka-ui-cert" \
             -keystore /truststore/truststore.jks

    You will be prompted to create a password. Memorize it: you will need it to deploy the Apache Kafka® web interface.

Set up the UI for Apache Kafka®

  1. On the VM, run the Docker container to deploy your web interface in:

    sudo docker run -it -p 8080:8080 \
   -e DYNAMIC_CONFIG_ENABLED=true \
   -e KAFKA_CLUSTERS_0_NAME=<cluster_name> \
   -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=<host_FQDN>:9091 \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=PLAIN \
   -e KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP=use_all_dns_ips \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="<user_name>" password="<user_password>";' \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION=/truststore/truststore.jks \
   -e KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD=<TrustStore_password> \
   -v /truststore/truststore.jks:/truststore/truststore.jks \
   provectuslabs/kafka-ui

    Specify the following in the environment variables:

    • KAFKA_CLUSTERS_0_NAME: Managed Service for Apache Kafka® cluster name.
    • KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: FQDN of the host with the KAFKA role in the Managed Service for Apache Kafka® cluster.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, username: Apache Kafka® user name.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, password: Apache Kafka® user password.
    • KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: Password you set when creating the truststore.jks file.

    Once started, the command does not terminate. While it is running, the UI for Apache Kafka® is available.

  2. On a local machine, go to http://<VM_public_IP_address>:8080 in your browser. The UI for Apache Kafka® with Managed Service for Apache Kafka® cluster data will open.

    You can find the VM's public IP address in the management console, on the VM page.

Deploying in a Managed Service for Kubernetes cluster

To deploy the UI for Apache Kafka® in a Managed Service for Kubernetes cluster:

  1. Install additional dependencies.
  2. Create a TrustStore.
  3. Deploy your application with the UI for Apache Kafka® in the Kubernetes pod.
  4. Check the result.

If you no longer need the resources you created, delete them.

Getting started

Prepare the infrastructure:

  1. Configure a single security group:

  2. Create a Managed Service for Apache Kafka® cluster. When creating it, specify the configured security group.

  3. Create an Apache Kafka® user.

  4. In the network hosting the Managed Service for Apache Kafka® cluster, create a Managed Service for Kubernetes cluster. When creating it, specify the configured security group and assign a public address to the cluster.

  5. Create a node group in the Managed Service for Kubernetes cluster. When creating it, specify the configured security group.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the kafka-ui-via-kubernetes.tf configuration file to the same working directory.

    This file describes:

    • Network.

    • Subnet.

    • Default security group and rules required to connect to the following from the internet:

      • Managed Service for Apache Kafka® cluster.
      • Managed Service for Kubernetes cluster.
      • Managed Service for Kubernetes node group.

    • Managed Service for Apache Kafka® cluster.

    • Apache Kafka® user.

    • Managed Service for Kubernetes cluster.

    • Managed Service for Kubernetes node group.

  6. Specify the variable values in the kafka-ui-via-kubernetes.tf file.

  7. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Install additional dependencies

On a local machine:

  1. Install kubectl and configure it to work with the created cluster.

  2. To check that the Managed Service for Apache Kafka® cluster is available, connect to one of its hosts with the KAFKA role:

    telnet <host_FQDN> 9091

    You can view the FQDN in the management console:

    1. Go to the cluster page.
    2. Go to Hosts.
    3. Copy the value in the Host FQDN column, in the row of the host with the KAFKA role.

    If the cluster is available, you will get this message:

    Connected to <host_FQDN>

    After this, you can abort the command as it does not complete but awaits data transfer.

  3. Install keytool to manage keys and certificates:

    sudo apt update && sudo apt install openjdk-19-jre-headless

Create a TrustStore

When deploying Apache Kafka® in a Managed Service for Kubernetes cluster, TrustStore commands run on a local machine.

TrustStore is a trusted certificate store used in JKS files. It serves for authenticating a client when connecting to the server. The server validates the client using certificates stored in TrustStore. However, the client stores the private key and the certificate on their side in KeyStore.

In the example below, TrustStore is used to connect to a Managed Service for Apache Kafka® cluster. With no TrustStore created, the Apache Kafka® web interface will lack information about the cluster.

To use TrustStore:

  1. Create an SSL certificate:

    sudo mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
sudo wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexCA.crt && \
sudo chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexCA.crt

  2. Create a directory named /truststore:

    mkdir /truststore

    It will store the truststore.jks file. You need a separate directory so that the file path is correctly recognized in commands and configuration files.

  3. Upload the YandexCA.crt certificate to the truststore.jks file:

    sudo keytool -import \
             -file /usr/local/share/ca-certificates/Yandex/YandexCA.crt \
             -alias "kafka-ui-cert" \
             -keystore /truststore/truststore.jks

    You will be prompted to create a password. Memorize it: you will need it to deploy the Apache Kafka® web interface.

Deploy your application with the UI for Apache Kafka® in the Kubernetes pod

  1. To deliver the truststore.jks file to the Kubernetes pod, create a secret containing this file:

    kubectl create secret generic truststore --from-file=/truststore/truststore.jks

  2. Create a file named kafka-ui-configMap.yaml with the configMap configuration. It contains information about the Managed Service for Apache Kafka® cluster and TrustStore:

    apiVersion: v1
kind: ConfigMap
metadata:
   name: kafka-ui-values
data:
   KAFKA_CLUSTERS_0_NAME: <Apache Kafka®_cluster_name>
   KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: <host_FQDN>:9091
   KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
   KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
   KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP: use_all_dns_ips
   KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="<user_name>" password="<user_password>";'
   KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION: /truststore/truststore.jks
   KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: <TrustStore_password>
   AUTH_TYPE: "DISABLED"
   MANAGEMENT_HEALTH_LDAP_ENABLED: "FALSE"

    Specify the following in the environment variables:

    • KAFKA_CLUSTERS_0_NAME: Managed Service for Apache Kafka® cluster name.
    • KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: FQDN of the host with the KAFKA role in the Managed Service for Apache Kafka® cluster.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, username: Apache Kafka® user name.
    • KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG, password: Apache Kafka® user password.
    • KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD: Password you set when creating the truststore.jks file.

  3. Create a file named kafka-ui-pod.yaml with the configuration of the pod to deploy your application with the UI for Apache Kafka® in:

    apiVersion: v1
kind: Pod
metadata:
   name: kafka-ui-pod
spec:
   containers:
   - name: kafka-ui-pod
     image: provectuslabs/kafka-ui
     envFrom:
        - configMapRef:
             name: kafka-ui-values
     volumeMounts:
     - name: truststore
       mountPath: "/truststore"
       readOnly: true
   volumes:
   - name: truststore
     secret:
        secretName: truststore
   - name: kafka-ui-configmap
     configMap:
        name: kafka-ui-values

  4. Apply the configMap configuration:

    kubectl apply -f kafka-ui-configMap.yaml

  5. Apply the pod configuration:

    kubectl apply -f kafka-ui-pod.yaml

Check the result

  1. View the pod logs to make sure the UI for Apache Kafka® is deployed successfully:

    kubectl logs kafka-ui-pod

    The result contains the following lines:

     _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
 \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
                                 |_|                                             

2024-01-23 12:13:25,648 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 8.0.0.Final
2024-01-23 12:13:25,745 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 17.0.6 with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2024-01-23 12:13:25,746 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v3.0.6, Spring v6.0.8
2024-01-23 12:13:25,747 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to 1 default profile: "default"

  2. Set the UI for Apache Kafka® port to 8080:

    kubectl --namespace default port-forward kafka-ui-pod 8080:8080

  3. In your browser, open http://127.0.0.1:8080/. The UI for Apache Kafka® with Managed Service for Apache Kafka® cluster data will open.

Delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Delete:

  1. Managed Service for Apache Kafka® cluster
  2. Virtual machine
  3. Managed Service for Kubernetes node group
  4. Managed Service for Kubernetes cluster

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the kafka-ui-via-docker.tf or the kafka-ui-via-kubernetes.tf configuration file depending on the deployment method used.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the resources described in the configuration file will be deleted.

Previous
All tutorials
Next
Migrating a database from a third-party Apache Kafka® cluster to Managed Service for Apache Kafka®