Contact UsGet started

Pre-configuration for connection to an Apache Kafka® cluster

Written by
Updated at November 18, 2024

You can connect to Managed Service for Apache Kafka® cluster hosts:

  • Over the internet if you configured public access for the cluster. You can only connect to this type of cluster using an SSL connection.

  • From Yandex Cloud virtual machines located in the same cloud network. If the cluster is not publicly available, you do not need to use an SSL connection to connect to such VMs.

You can connect to the Apache Kafka® cluster both with encryption (SASL_SSL, port 9091) and without it (SASL_PLAINTEXT, port 9092).

To connect to an Apache Kafka® cluster:

  1. Create users for clients (producers and consumers) with access to the required topics.
  2. Connect the clients to the cluster:

There are ready-made Apache Kafka® API implementations for most popular programming languages. To view use examples, see the Code examples section.

Configuring security groups

To connect to a cluster, security groups must include rules allowing traffic from certain ports, IP addresses, or from other security groups.

Rule settings depend on the connection method you select:

Configure all the cluster security groups to allow incoming traffic on port 9091 from any IP address. To do this, create the following rule for incoming traffic:

  • Port range: 9091.
  • Protocol: TCP.
  • Source: CIDR.
  • CIDR blocks: 0.0.0.0/0

To allow connections to Managed Schema Registry, add a rule for incoming traffic:

  • Port range: 443.
  • Protocol: TCP.
  • Source: CIDR.
  • CIDR blocks: 0.0.0.0/0

  1. Configure all the cluster security groups to allow incoming traffic from the security group where the VM is located on ports 9091 and 9092. To do this, create the following rule for incoming traffic in these groups:

    • Port range: 9091-9092.
    • Protocol: TCP.
    • Source: Security group.
    • Security group: If your cluster and VM are in the same security group, select Current (Self). Otherwise, specify the VM security group.

    To allow connections to Managed Schema Registry, add a rule for incoming traffic:

    • Port range: 443.
    • Protocol: TCP.
    • Source: CIDR.
    • CIDR blocks: 0.0.0.0/0

  2. Configure the security group where the VM is located to enable connections to the VM and traffic between the VM and the cluster hosts.

    For example, you can set the following rules for a VM:

    • For incoming traffic:

      • Port range: 22.
      • Protocol: TCP.
      • Source: CIDR.
      • CIDR blocks: 0.0.0.0/0

      This rule allows you to connect to the VM over SSH.

    • For outgoing traffic:

      • Protocol: Any
      • Port range: 0-65535.
      • Destination name: CIDR.
      • CIDR blocks: 0.0.0.0/0

      This rule allows all outgoing traffic, thus enabling you not only to connect to the cluster but also to install the certificates and utilities your VM needs for the connection.

Note

You can specify more detailed rules for your security groups, e.g., to allow traffic only in specific subnets.

You must configure security groups correctly for all subnets in which the cluster hosts will reside. If security group settings are incomplete or incorrect, you may lose access to the cluster.

For more information about security groups, see Security groups.

Getting an SSL certificate

To use an encrypted connection, get an SSL certificate:

mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
     --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt

The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt file.

mkdir $HOME\.kafka; curl.exe -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem

The certificate is saved to the $HOME\.kafka\YandexInternalRootCA.crt file.

The resulting SSL certificate is also used when working with Managed Schema Registry.

Getting FQDNs of Apache Kafka® hosts

To connect to a host, you need its fully qualified domain name (FQDN). You can obtain it in one of the following ways:

  • Request a list of cluster hosts to get the host name. The FQDN matches the host name.

    1. In the management console, go to the relevant folder.
    2. In the list of services, select Managed Service for Kafka.
    3. Click the name of the cluster you need and select the Hosts tab.

    If you do not have the Yandex Cloud command line interface yet, install and initialize it.

    The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

    To get a list of cluster hosts, run the command:

    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    You can request the cluster ID and name with a list of clusters in the folder.

    To get a list of cluster hosts, use the listHosts REST API method for the Cluster resource or the ClusterService/ListHosts gRPC API call and provide the cluster ID in the clusterId request parameter.

    To find out the cluster ID, get a list of clusters in the folder.

  • In the management console, copy the command for connecting to the cluster. This command contains the broker host FQDN. To get the command, go to the cluster page and click Connect.

What's next

  • Connect to the cluster using command line tools.
  • Integrate the cluster connection into your aplication code.
Previous
Deleting a cluster
Next
Connecting from applications