Syncing data from Apache Kafka® topics to an Object Storage bucket without using the internet

You can synchronize data from Apache Kafka® topics to a Yandex Object Storage bucket without using the internet via a service connection in the user network hosting the Managed Service for Apache Kafka® cluster. To do this:

1. Send data to the topic.
2. Make sure the bucket is not accessible from the external network.
3. Check if there is data in the bucket.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost for this solution includes:

• Fee for an Object Storage bucket: Covers data storage and bucket operations (see Object Storage pricing).
• Managed Service for Apache Kafka® cluster fee: Covers the use of computating resources allocated to hosts and disk space (see Managed Service for Apache Kafka® pricing).
• Fee for using public IP addresses for cluster hosts (see Virtual Private Cloud pricing).

Getting startedGetting started

1. Set up the infrastructure:

   Manually

   Terraform

   1. Create a network named my-private-network. Disable Create subnets when creating it.

   2. Create a subnet in any availability zone.

   3. Create a service connection to Object Storage in my-private-network and write down its ID.

   4. Create a service account named storage-pe-admin with the storage.admin role. The Managed Service for Apache Kafka® cluster will use it to access the bucket.

   5. Create a static access key for the storage-pe-admin service account.

   6. Create a Yandex Object Storage bucket and configure its bucket policy:

      • Result: Allow.
      • Action: All actions.
      • Resource: <bucket_name> and <bucket_name>/*.
      • Condition: Select the yc:private-endpoint-id key from the list and specify the ID of the created service connection as its value.

   7. Create a Managed Service for Apache Kafka® cluster of any suitable configuration and the following settings:

      • Broker hosts: One
      • Access to cluster hosts: Public
      • Network: my-private-network

   8. If using security groups in your cluster, make sure they are configured correctly and allow connecting to the cluster.

   9. In the cluster, create a topic named my-private-topic.

   10. In the cluster, create a user named mkf-user with the ACCESS_ROLE_PRODUCER access permission for the new topic.

   11. In the cluster, create a connector with the following settings:

       • Under Additional properties, specify the connector properties:
         • key.converter: org.apache.kafka.connect.storage.StringConverter
         • value.converter: org.apache.kafka.connect.converters.ByteArrayConverter
         • format.output.fields.value.encoding: none
       • Select the S3 Sink connector type.
       • In the Topics field, specify my-private-topic.
       • Under S3 connection, specify the following settings:
         • Bucket: Bucket you created previously.
         • Endpoint: storage.pe.yandexcloud.net.
         • Access key ID, Secret access key: ID and secret key of the previously created static access key.

   12. Create a VM with a public IP address in the created my-private-network to connect to the bucket.

   1. If you do not have Terraform yet, install it.

   2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

   3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

   4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

   5. Download the kafka-objstorage-sync-private-network.tf configuration file to the same working directory.

      This file describes:

      • Network.
      • Subnet.
      • Service connection.
      • Security group required for cluster connection.
      • Service account for creating and accessing the bucket.
      • Object Storage bucket.
      • Managed Service for Apache Kafka® cluster.
      • Apache Kafka® topic.
      • Apache Kafka® user.
      • Apache Kafka® connector.
      • VM to read data from the bucket.

   6. In the kafka-objstorage-sync-private-network.tf file, specify the following:

      • tf_account_name: Service account name, the same as the one in the provider settings.
      • bucket_name: Bucket name consistent with the naming conventions.
      • mkf_version: Apache Kafka® version.
      • mkf_user_password: Apache Kafka® user password.
      • vm_image_id: ID of the public image.
      • vm_username and vm_ssh_key: Username and absolute path to the public key, for access to the VM.

   7. Validate your Terraform configuration files using this command:

      terraform validate
      

      Terraform will display any configuration errors detected in your files.

   8. Create the required infrastructure:

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

2. Make sure that Yandex Cloud DNS has the *.storage.pe.yandexcloud.net record in the . service zone of the new network.

3. Install kafkacat to write data to the Apache Kafka® topic.

   sudo apt update && sudo apt install --yes kafkacat
   

   Make sure you can use it to connect to the previously created Managed Service for Apache Kafka® cluster over SSL.

Send data to the topicSend data to the topic

1. Send test data to my-private-topic using kafkacat:

   for i in $(echo {1..50})
       do
          echo "test message_"$i | kafkacat -P \
             -b <broker_host_FQDN>:9091 \
             -t my-private-topic \
             -k key \
             -X security.protocol=SASL_SSL \
             -X sasl.username="<username_in_source_cluster>" \
             -X sasl.password="<user_password_in_source_cluster>" \
             -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt -Z
       done
   

2. Make sure data is replicated to the bucket. To do this, open the bucket upload statistics and check the Modify Requests chart for requests.

Make sure the bucket is not accessible from the external networkMake sure the bucket is not accessible from the external network

1. If you do not have the AWS CLI yet, install and configure it.

2. Run this command to get a list of bucket objects at a public network address:

   aws s3 ls s3://<bucket_name> \
       --endpoint-url=https://storage.yandexcloud.net \
       --recursive
   

   Result:

   An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
   

3. Run this command to get a list of bucket objects at a private network address:

   aws s3 ls s3://<bucket_name> \
       --endpoint-url=https://storage.pe.yandexcloud.net \
       --recursive
   

   Result:

   Could not connect to the endpoint URL: "https://storage.pe.yandexcloud.net/<bucket_name>?list-type=2&prefix=&encoding-type=url"
   

Check if there is data in the bucket via the service connectionCheck if there is data in the bucket via the service connection

Connect to the previously created VM over SSH and do the following:

1. Install and configure the AWS CLI.

2. Run this command to get a list of bucket objects at a private network address:

   aws s3 ls s3://<bucket_name> \
       --endpoint-url=https://storage.pe.yandexcloud.net \
       --recursive
   

   Result:

   2025-08-01 14:38:23         20 my-private-topic-1-0
   2025-08-01 14:38:23        587 my-private-topic-1-1
   2025-08-01 14:38:24        440 my-private-topic-1-29
   

This result means Apache Kafka® topic data has been successfully synchronized via the service connection.

Delete the resources you createdDelete the resources you created

Some resources incur charges. To avoid paying for them, delete the resources you no longer need: