Yandex Cloud
Search
Contact UsGet started
  • Blog
  • Pricing
  • Documentation
  • All Services
  • System Status
    • Featured
    • Infrastructure & Network
    • Data Platform
    • Containers
    • Developer tools
    • Serverless
    • Security
    • Monitoring & Resources
    • ML & AI
    • Business tools
  • All Solutions
    • By industry
    • By use case
    • Economics and Pricing
    • Security
    • Technical Support
    • Customer Stories
    • Gateway to Russia
    • Cloud for Startups
    • Education and Science
  • Blog
  • Pricing
  • Documentation
Yandex project
© 2025 Yandex.Cloud LLC
Yandex Managed Service for Apache Kafka®
  • Getting started
  • Access management
  • Pricing policy
  • Terraform reference
  • Yandex Monitoring metrics
  • Audit Trails events
  • Public materials
  • Release notes
  • FAQ

In this article:

  • Getting started
  • Create a cluster
  • Create the topic
  • Create a user
  • Connect to the cluster
  • What's next

Getting started with Managed Service for Apache Kafka®

Written by
Yandex Cloud
Improved by
Danila N.
Updated at April 22, 2025
  • Getting started
  • Create a cluster
  • Create the topic
  • Create a user
  • Connect to the cluster
  • What's next

To get started:

  1. Create a cluster.
  2. Create a topic.
  3. Create a user.
  4. Connect to the cluster.

Getting startedGetting started

  1. Go to the management console and log in to Yandex Cloud or sign up if not signed up yet.

  2. If you do not have a folder yet, create one:

    1. In the management console, select the appropriate cloud from the list on the left.

    2. At the top right, click Create folder.

    3. Give your folder a name. The naming requirements are as follows:

      • It must be from 2 to 63 characters long.
      • It may contain lowercase Latin letters, numbers, and hyphens.
      • It must start with a letter and cannot end with a hyphen.
    4. Optionally, specify the description for your folder.

    5. Select Create a default network. This will create a network with subnets in each availability zone. Within this network, you will also have a default security group, inside which all network traffic will be allowed.

    6. Click Create.

  3. Assign the vpc.user role and the managed-kafka.editor role or higher to your Yandex Cloud account. These roles allow you to create a cluster.

    Note

    If you are unable to manage roles, contact your cloud or organization administrator.

  4. You can connect to an Apache Kafka® cluster from both inside and outside Yandex Cloud:

    • To connect from inside Yandex Cloud, create a Linux-based VM in the same network as the cluster.

    • To connect to a cluster from the internet, enable public access to the cluster when creating it.

  5. Connect to the VM over SSH.

    Note

    It is assumed that you are connecting to the cluster from a Linux-based VM.

  6. Install the kafkacat utility, which is an open source app that can function as a universal data producer or consumer:

    sudo apt-get install kafkacat
    

    Check that you can use it to connect to the Managed Service for Apache Kafka® source cluster over SSL.

Create a clusterCreate a cluster

To create a cluster:

  1. In the management console, select the folder where you want to create a cluster.
  2. Select Managed Service for Kafka.
  3. Click Create cluster.
  4. Set the cluster parameters and click Create. For more information, see Creating clusters.
  5. Wait until the cluster is ready: its status on the Managed Service for Apache Kafka® dashboard will change to Running, and its state, to Alive. This may take some time.

Then create a topic in the cluster.

Create the topicCreate the topic

A topic is a way to group message streams into categories. Producers write messages to a topic and consumers read messages from it.

To create a topic:

  1. In the management console, select the folder where the cluster is located.
  2. Select Managed Service for Kafka.
  3. Click the name of the cluster you created and select the Topics tab.
  4. Click Create topic.
  5. Specify the topic settings and click Create. This process is described in detail in Managing Apache Kafka® topics.

Then create users for producers and consumers.

Create a userCreate a user

User settings let you manage producer and consumer permissions to cluster topics.

To create a user:

  1. In the management console, select the folder where the cluster is located.
  2. Select Managed Service for Kafka.
  3. Click the name of the cluster you created and select the Users tab.
  4. Click Create user.
  5. Enter a username and password (from 8 to 128 characters).
  6. Click Add topic and select the previously created topic from the drop-down list.
  7. Add permissions to this topic for the producer and consumer. For more information, see User management.
  8. Click Create.

Then connect to the cluster using this username.

Connect to the clusterConnect to the cluster

You can connect the producer and consumer to the cluster on behalf of one user. Both the producer and consumer will only be able to work with the topics that this user is allowed to access.

To connect to a cluster:

  1. If you are using security groups for a cloud network, configure them to enable all relevant traffic between the cluster and the connecting host.

  2. Install an SSL certificate on the VM:

    Linux (Bash)/macOS (Zsh)
    Windows (PowerShell)
    mkdir -p /usr/local/share/ca-certificates/Yandex/ && \
    wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
         --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
    chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
    

    The certificate will be saved to the /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt file.

    mkdir $HOME\.kafka; curl.exe -o $HOME\.kafka\YandexInternalRootCA.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
    

    The certificate will be saved to the $HOME\.kafka\YandexInternalRootCA.crt file.

    Corporate policies and antivirus software can block the download of certificates. For more information, see FAQ.

  3. To send a message to a topic, run the command:

    echo "test message" | kafkacat -P \
        -b <broker_FQDN>:9091 \
        -t <topic_name> \
        -k key \
        -X security.protocol=SASL_SSL \
        -X sasl.mechanism=SCRAM-SHA-512 \
        -X sasl.username="<producer_login>" \
        -X sasl.password="<producer_password>" \
        -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
    

    In the command, specify the broker FQDN, topic name, username and password of the Apache Kafka® user you created earlier.

    For info on how to get a broker host's FQDN, see this guide.

  4. To get messages from a topic, run the following command:

    kafkacat -C \
             -b <broker_FQDN>:9091 \
             -t <topic_name> \
             -X security.protocol=SASL_SSL \
             -X sasl.mechanism=SCRAM-SHA-512 \
             -X sasl.username="<consumer_username>" \
             -X sasl.password="<consumer_password>" \
             -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z -K:
    

    In the command, specify the broker FQDN, topic name, username and password of the Apache Kafka® user you created earlier.

    For info on how to get a broker host's FQDN, see this guide.

For more information about connecting to a Managed Service for Apache Kafka® cluster, see Connecting to topics in a cluster.

What's nextWhat's next

  • Read about service concepts.
  • Learn more about creating a cluster and connecting to a cluster.

Was the article helpful?

Next
All guides
Yandex project
© 2025 Yandex.Cloud LLC