Pre-configuration for connection to an Apache Kafka® cluster
You can connect to Managed Service for Apache Kafka® cluster hosts:
-
Over the internet if you configured public access for the cluster. You can only connect to this type of cluster using an SSL connection.
-
From Yandex Cloud virtual machines located in the same cloud network. If the cluster is not publicly available, you do not need to use an SSL connection to connect to such VMs.
You can connect to the Apache Kafka® cluster both with encryption (SASL_SSL
, port 9091) and without it (SASL_PLAINTEXT
, port 9092).
To connect to an Apache Kafka® cluster:
- Create users for clients (producers and consumers) with access to the required topics.
- Connect the clients to the cluster:
- Producers using the Kafka Producer API
. - Consumers using the Kafka Consumer API
.
- Producers using the Kafka Producer API
There are ready-made Apache Kafka® API implementations for most popular programming languages. To view use examples, see the Code examples section.
Configuring security groups
To connect to a cluster, security groups must include rules allowing traffic from certain ports, IP addresses, or from other security groups.
Rule settings depend on the connection method you select:
Configure all the cluster security groups to allow incoming traffic on port 9091 from any IP address. To do this, create the following rule for incoming traffic:
- Port range:
9091
. - Protocol:
TCP
- Source:
CIDR
- CIDR blocks:
0.0.0.0/0
To allow connections to Managed Schema Registry, add a rule for incoming traffic:
- Port range:
443
- Protocol:
TCP
- Source:
CIDR
- CIDR blocks:
0.0.0.0/0
-
Configure all the cluster security groups to allow incoming traffic from the security group where the VM is located on ports 9091 and 9092. To do this, create the following rule for incoming traffic in these groups:
- Port range:
9091-9092
. - Protocol:
TCP
. - Source:
Security group
. - Security group: If your cluster and VM are in the same security group, select
Current
(Self
). Otherwise, specify the VM security group.
To allow connections to Managed Schema Registry, add a rule for incoming traffic:
- Port range:
443
- Protocol:
TCP
- Source:
CIDR
- CIDR blocks:
0.0.0.0/0
- Port range:
-
Configure the security group where the VM is located to enable connections to the VM and traffic between the VM and the cluster hosts.
For example, you can set the following rules for a VM:
-
For incoming traffic:
- Port range:
22
. - Protocol:
TCP
- Source:
CIDR
- CIDR blocks:
0.0.0.0/0
This rule allows you to connect to the VM over SSH.
- Port range:
-
For outgoing traffic:
- Protocol:
Any
- Port range:
0-65535
. - Destination name:
CIDR
- CIDR blocks:
0.0.0.0/0
This rule allows all outgoing traffic, thus enabling you not only to connect to the cluster but also to install the certificates and utilities your VM needs for the connection.
- Protocol:
-
Note
You can specify more detailed rules for your security groups, e.g., to allow traffic only in specific subnets.
You must configure security groups correctly for all subnets in which the cluster hosts will reside. If security group settings are incomplete or incorrect, you may lose access to the cluster.
For more information about security groups, see Security groups.
Getting an SSL certificate
To use an encrypted connection, get an SSL certificate:
mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
--output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
file.
mkdir $HOME\.kafka; curl.exe -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
The certificate is saved to the $HOME\.kafka\YandexInternalRootCA.crt
file.
The resulting SSL certificate is also used when working with Managed Schema Registry.
What's next
- Get the FQDN of the host you want to connect to.
- Connect to the cluster using command line tools.
- Integrate the cluster connection into your aplication code.