Getting started with Managed Service for Apache Kafka®

To get started:

1. Create a cluster.
2. Create a topic.
3. Create a user.
4. Connect to the cluster.

Getting startedGetting started

1. Navigate to the management console and log in to Yandex Cloud or sign up if not signed up yet.

2. If you do not have a folder yet, create one:

   1. In the management console, in the top panel, click or and select the cloud.

   2. To the right of the cloud name, click .

   3. Select Create folder.

      

   4. Give your folder a name. The naming requirements are as follows:

      • Length: between 3 and 63 characters.
      • It can only contain lowercase Latin letters, numbers, and hyphens.
      • It must start with a letter and cannot end with a hyphen.

   5. Optionally, specify the description for your folder.

   6. Select Create a default network. This will create a network with subnets in each availability zone. Within this network, you will also have a default security group, within which all network traffic will be allowed.

   7. Click Create.

      

3. Assign the vpc.user role and the managed-kafka.editor role (or higher) to your Yandex Cloud account. These roles allow creating a cluster.

   Note

   If you cannot manage roles, contact your cloud or organization administrator.

4. You can connect to an Apache Kafka® cluster from both inside and outside Yandex Cloud:

   • To connect from inside Yandex Cloud, create a Linux VM in the same network as the cluster.

   • To connect to the cluster from the internet, enable public access when creating it.

5. Connect to your VM over SSH.

   Note

   It is assumed that you are connecting to the cluster from a Linux VM.

6. Install kafkactl, a command line interface for Apache Kafka®:

   wget https://github.com/deviceinsight/kafkactl/releases/download/v<version_number>/<archive_name>.tar.gz
   tar xzf <archive_name>.tar.gz kafkactl
   sudo mv kafkactl /usr/local/bin
   

   You can look up the archive version and name on the kafkactl releases page.

   Check that you can connect to the Managed Service for Apache Kafka® source cluster over SSL with kafkactl.

Create a clusterCreate a cluster

To create a cluster:

1. In the management console, select the folder where you want to create a cluster.
2. Navigate to Managed Service for Kafka.
3. Click Create cluster.
4. Specify the cluster settings and click Create. For more information, see Creating a cluster.
5. Wait until the cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to Running, and its state, to Alive. This may take a while.

Then create a topic in the cluster.

Create a topicCreate a topic

A topic is used to group message streams by category. Producers write messages to a topic, and consumers read messages from it.

To create a topic:

1. In the management console, select the folder containing the cluster.
2. Navigate to Managed Service for Kafka.
3. Click the cluster name and select the Topics tab.
4. Click Create topic.
5. Specify the topic settings and click Create. For details, see Managing Apache Kafka® topics.

Then create users for producers and consumers.

Create a userCreate a user

User settings enable you to manage producer and consumer permissions for cluster topics.

Learn more about the permissions you get with each role here.

To create a user:

1. In the management console, select the folder containing the cluster.
2. Navigate to Managed Service for Kafka.
3. Click the cluster name and select the Users tab.
4. Click Create user.
5. Enter a username and password (from 8 to 128 characters).
6. Click Add topic and select the previously created topic from the drop-down list.
7. Grant access permissions for this topic to the producer and consumer. For more information, see User management.
8. Click Create.

Then connect to the cluster as this user.

Connect to the clusterConnect to the cluster

You can connect both the producer and consumer to the cluster as the same user. Both the producer and the consumer can work only with the topics that this user has access to.

To connect to your cluster:

1. If you are using security groups for your cloud network, configure them to allow all relevant traffic between the cluster and the connecting host.

2. Install an SSL certificate on the VM:

   Linux (Bash)/macOS (Zsh)

   Windows (PowerShell)

   mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
   wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
        --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
   chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
   

   The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt file.

   mkdir $HOME\.kafka; curl.exe -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
   

   The certificate will be saved to the $HOME\.kafka\YandexInternalRootCA.crt file.

   Your corporate security policies and antivirus software may block the certificate download. For more information, see FAQ.

3. Create a folder named ~/.config/kafkactl and place the config.yml configuration file with your Apache Kafka® cluster connection parameters in that folder:

   cd ~/ && \
   mkdir --parents .config/kafkactl && \
   cd ~/.config/kafkactl
   

   Here is an example of the config.yml file:

   contexts:
     default:
       brokers:
       - <broker_FQDN>:9091
       sasl:
         enabled: true
         username: <consumer_login>
         password: <consumer_password>
         mechanism: scram-sha512
       tls:
         enabled: true
         ca: /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
   

   In the configuration file, specify the broker FQDN as well as the login and password of the Apache Kafka® user you created earlier.

   To learn how to get a broker host FQDN, see this guide.

4. To send a message to a topic, run this command:

   echo "test message" | kafkactl produce <topic_name>
   

5. To get messages from a topic, run the following command:

   kafkactl consume <topic_name>
   

For a detailed description of the Managed Service for Apache Kafka® cluster connection process, see Connecting to topics in a cluster.

What's nextWhat's next

• Read about the service concepts.
• Learn more about creating a cluster and connecting to a cluster.