Connecting to an Apache Kafka® cluster
In this tutorial, we will analyze the kcat commands for external and internal connections to an Apache Kafka® cluster and send a message from the producer to the consumer.
Getting started
- Install kcat.
- If the project does not exist yet, create it:
kubectl create namespace <project name>. - Create an Apache Kafka® cluster.
- Create a topic.
- Create a user with administrator privileges.
How to connect to an Apache Kafka® cluster
Note
To connect to Apache Kafka® from outside the cluster, you must enable the public access option (listenerType: LoadBalancer) in the Apache Kafka® cluster settings.
-
Get an external link to the Apache Kafka® cluster:
kubectl get kafkaclusters <cluster name> -o jsonpath='{.status.listenersStatus.external.fqdn}' -n <project_name>. -
Get the port to connect to:
kubectl get kafkaclusters <cluster name> -o jsonpath='{.status.listenersStatus.external.port}' -n <project name>. -
Copy the connection certificate from the Apache Kafka® cluster resource field named
status.listenersStatus.external.certificate:kubectl describe kafkaclusters <cluster name> -n <project name>. -
Save the certificate to a file named
<path to certificate>/ca.crt. -
Run the following command to receive messages from the topic:
kcat -C \ -b <link to cluster>:<port> \ -t <topic name> \ -X security.protocol=SASL_SSL \ -X sasl.mechanism=SCRAM-SHA-512 \ -X sasl.username="<username>" \ -X sasl.password="<user password>" \ -X ssl.endpoint.identification.algorithm=none \ -X ssl.ca.location=/<path to certificate>/ca.crt -Z -K: -
Open a new terminal and send the following message to the topic:
echo "test message" | kcat -P \ -b <link to cluster>:<port> \ -t <topic name> \ -k key \ -X security.protocol=SASL_SSL \ -X sasl.mechanism=SCRAM-SHA-512 \ -X sasl.username="<username>" \ -X sasl.password="<user password>" \ -X ssl.endpoint.identification.algorithm=none \ -X ssl.ca.location=/<path to certificate>/ca.crt -Z
The consumer will receive a message: "test message".
-
Get an internal link to the Apache Kafka® cluster:
kubectl get kafkaclusters <cluster name> -o jsonpath='{.status.listenersStatus.internal.fqdn}' -n <project_name>. -
Get the port to connect to:
kubectl get kafkaclusters <cluster name> -o jsonpath='{.status.listenersStatus.internal.port}' -n <project name>. -
Run the following command to receive messages from the topic:
kcat -C \ -b <link to cluster>:<port> \ -t <topic name> \ -X security.protocol=SASL_PLAINTEXT \ -X sasl.mechanism=SCRAM-SHA-512 \ -X sasl.username="<username>" \ -X sasl.password="<user password>" -Z -K: -
Open a new terminal and send the following message to the topic:
echo "test message" | kcat -P \ -b <link to cluster>:<port> \ -t <topic name> \ -k key \ -X security.protocol=SASL_PLAINTEXT \ -X sasl.mechanism=SCRAM-SHA-512 \ -X sasl.username="<username>" \ -X sasl.password="<user password>" -Z
The consumer will receive a message: "test message".