Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Pre-configuring a connection to an Apache Kafka® cluster

Written by
Updated at January 23, 2025

You can connect to Managed Service for Apache Kafka® cluster hosts:

  • Over the internet if you configured public access for the cluster. You can only connect to this type of cluster using an SSL connection.

  • From Yandex Cloud virtual machines located in the same cloud network. If the cluster is not publicly available, you do not need to use an SSL connection to connect to such VMs.

You can connect to an Apache Kafka® cluster both with encryption (SASL_SSL, port 9091) and without it (SASL_PLAINTEXT, port 9092).

To connect to an Apache Kafka® cluster:

  1. Create users for clients (producers and consumers) with access to the required topics.
  2. Connect the clients to the cluster:

There are ready-made Apache Kafka® API implementations for most popular programming languages. To view use examples, see the Code examples section.

Configuring security groups

To connect to a cluster, security groups must include rules allowing traffic from certain ports, IP addresses, or from other security groups.

Rule settings depend on the connection method you select:

Configure all the cluster security groups to allow incoming traffic on port 9091 from any IP address. To do this, create the following rule for incoming traffic:

  • Port range: 9091
  • Protocol: TCP
  • Source: CIDR
  • CIDR blocks: 0.0.0.0/0

To allow the use of Managed Service for Apache Kafka® API, e.g., to work with Managed Schema Registry, add a rule for incoming traffic:

  • Port range: 443
  • Protocol: TCP
  • Source: CIDR
  • CIDR blocks: 0.0.0.0/0

  1. Configure all the cluster security groups to allow incoming traffic from the security group where the VM is located on ports 9091 and 9092. To do this, create the following rule for incoming traffic in these groups:

    • Port range: 9091-9092
    • Protocol: TCP
    • Source: Security group
    • Security group: If your cluster and VM are in the same security group, select Current (Self). Otherwise, specify the VM security group.

    To allow the use of Managed Service for Apache Kafka® API, e.g., to work with Managed Schema Registry, add a rule for incoming traffic:

    • Port range: 443
    • Protocol: TCP
    • Source: CIDR
    • CIDR blocks: 0.0.0.0/0

  2. Configure the security group where the VM is located to enable connections to the VM and traffic between the VM and the cluster hosts.

    For example, you can set the following rules for a VM:

    • For incoming traffic:

      • Port range: 22
      • Protocol: TCP
      • Source: CIDR
      • CIDR blocks: 0.0.0.0/0

      This rule allows you to connect to a VM over SSH.

    • For outgoing traffic:

      • Protocol: Any (Any)
      • Port range: 0-65535
      • Destination name: CIDR
      • CIDR blocks: 0.0.0.0/0

      This rule allows all outgoing traffic, thus enabling you not only to connect to the cluster but also to install the certificates and utilities your VM needs for the connection.

Note

You can specify more detailed rules for your security groups, e.g., to allow traffic only in specific subnets.

You must configure security groups correctly for all subnets in which the cluster hosts will reside. If security group settings are incomplete or incorrect, you may lose access to the cluster.

For more information about security groups, see Security groups.

Getting an SSL certificate

To use an encrypted connection, get an SSL certificate:

mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt

The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt file.

mkdir $HOME\.kafka; curl.exe -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem

The certificate will be saved to the $HOME\.kafka\YandexInternalRootCA.crt file.

Corporate policies and antivirus software can block the download of certificates. For more information, see FAQ.

This certificate is also used to access the Managed Service for Apache Kafka® REST API.

Getting FQDNs of Apache Kafka® hosts

To connect to a host, you need its fully qualified domain name (FQDN). Example of an Apache Kafka® host FQDN:

rc1a-goh2a9tr********.mdb.yandexcloud.net

You can get the FQDN by doing one of the following:

  • Look up the FQDN in the management console:

    1. Go to the cluster page.
    2. Go to Hosts.
    3. Copy the Host FQDN column value.

  • In the management console, copy the command for connecting to the cluster. This command contains the broker host FQDN. To get the command, go to the cluster page and click Connect.

  • Request the list of cluster hosts using the CLI or API.

With the Managed Service for Apache Kafka® REST API, you can send requests to any broker host: the API is available from all the cluster's broker hosts.

What's next

  • Connect to the cluster using command line tools.
  • Integrate the cluster connection into your aplication code.
Previous
Deleting a cluster
Next
Connecting from applications