Migrating from Yandex IoT Core to open-source MQTT solutions

After Yandex IoT Core is phased out, you can deploy an alternative MQTT solution on your own: either on a Yandex Compute Cloud VM instance or in a Yandex Managed Service for Kubernetes cluster.

In place of Yandex IoT Core, you can use the following open-source solutions, all of which can run in Yandex Cloud:

• Eclipse Mosquitto: Recommended for most scenarios where Yandex IoT Core was used as an MQTT broker.
• TBMQ Community Edition: Suitable for high-load MQTT scenarios requiring clustering and long-term message delivery.
• ThingsBoard Community Edition: Suitable for scenarios where, apart from MQTT, you also need device management, dashboards, a rule engine, and a UI.

How to choose your target solutionHow to choose your target solution

The section below briefly compares the three options. This guide gives detailed deployment steps for Eclipse Mosquitto. For TBMQ and ThingsBoard, refer to relevant official guides.

Choose Eclipse Mosquitto if:

• Devices use MQTT and connect to Yandex IoT Core over TLS.
• Your main logic centers around publishing and subscribing to topics.
• You are ready to replace your Yandex IoT Core integrations with Yandex Cloud Functions, Yandex Serverless Containers, or Yandex Data Streams with a proprietary message handler.
• You are happy with a single broker or active/passive level fault tolerance.

Choose TBMQ Community Edition if:

• Your MQTT broker deals with high workloads.
• You need clustering, horizontal scaling, and more reliable message delivery.
• You are ready to maintain additional infrastructure, e.g., Apache Kafka® and a database.

Choose ThingsBoard Community Edition if:

• Yandex IoT Core was part of a broader IoT system.
• You need device management, dashboards, a rule engine, a UI, telemetry storage, and event handling.
• You are ready to adapt your devices and backend to ThingsBoard's data model.

What will change after the migrationWhat will change after the migration

Migrating from Yandex IoT Core to a custom MQTT solution will affect several key parts of your architecture.

EndpointEndpoint

In Yandex IoT Core, devices were connecting to a managed endpoint provided by the service. After the migration, you will have to create an endpoint of your own, for example:

mqtt.example.com:8883

The endpoint may point to:

• Public IP address of a VM running Eclipse Mosquitto.
• Network load balancer upstream of the broker.
• LoadBalancer-type Service in Yandex Managed Service for Kubernetes.

For MQTT, use TCP port 8883 with TLS. An L7 load balancer is not suitable for standard MQTT because MQTT uses TCP, not HTTP.

If some of the clients connect via MQTT over WebSocket (e.g., web or browser clients), configure a separate Mosquitto listener with protocol websockets and TLS (typically on port 443) and open this port in your security group.

AuthenticationAuthentication

In Yandex IoT Core, devices could use the following:

• X.509 certificates.
• username and password.

Mosquitto supports the same mechanisms:

• password_file: For authentication by username and password.
• require_certificate and use_identity_as_username: For client certificate-based authentication.
• acl_file: For topic-level access control.

TopicsTopics

In Yandex IoT Core, devices and registries use the following topics:

$devices/<device_id>/events
$devices/<device_id>/state
$devices/<device_id>/commands
$devices/<device_id>/config
$registries/<registry_id>/events
$registries/<registry_id>/state
$registries/<registry_id>/commands
$registries/<registry_id>/config

The state and config topics are persistent: the broker stores the last message in them. In standard MQTT, a similar thing is implemented via retained messages (see below).

For the migration, keep this structure unchanged ($devices/..., $registries/...) in Mosquitto to avoid tweaking the firmware of devices. All examples below use exactly this structure. Switching to non-$ topics is a separate optional step. If you do it, you need to update device firmware, ACLs, and subscribers at the same time.

Persistent state and config topicsPersistent state and config topics

In Yandex IoT Core, state and config are persistent topics used by the service to store the last message.

Standard MQTT replicates this behavior using the retain flag. After the migration:

• When publishing device state to state, set retain = true in the message.
• When publishing device configuration to config, also use retain = true.
• Make sure your target broker stores retained messages on a persistent disk.

Example of publishing a retained message via mosquitto_pub:

mosquitto_pub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -u <device_id> \
  -P '<password>' \
  -t 'devices/<device_id>/state' \
  -m '{"online":true}' \
  -q 1 \
  -r

Triggers and message exportTriggers and message export

If you were using triggers to invoke Cloud Functions functions or Serverless Containers containers, replace them with a custom MQTT subscriber after the migration.

Such a subscriber must:

• Subscribe to relevant MQTT topics.
• Convert messages to the right format.
• Invoke functions, containers, HTTP APIs, or send data to Yandex Data Streams streams.
• Log errors and retry the procedure on failure.

For an example, see Replace triggers.

Mosquitto migration architectureMosquitto migration architecture

We recommend this minimal architecture:

1. Devices connect to Mosquitto over MQTTS, port 8883.
2. Mosquitto checks the device's username/password or client certificate.
3. Mosquitto applies ACLs thus allowing the device to publish and read it own topics only.
4. The backend subscriber receives messages from the events and state topics.
5. The backend subscriber forwards messages to your handlers, databases, Cloud Functions functions, Serverless Containers containers, or Yandex Data Streams streams.
6. The backend application publishes the commands and configuration to the commands and config topics.

┌─────────────┐        MQTT/TLS          ┌──────────────┐
│ Devices  │  ─────────────────────▶  │  Mosquitto   │
└─────────────┘                          └──────┬───────┘
                                                │ MQTT subscribe
                                                ▼
                                      ┌────────────────────┐
                                      │ Backend subscriber │
                                      └─────────┬──────────┘
                                                │
                 ┌──────────────────────────────┼──────────────────────────────┐
                 ▼                              ▼                              ▼
        Cloud Functions              Serverless Containers      Yandex Data Streams

Export data from Yandex IoT CoreExport data from Yandex IoT Core

Before migrating, export the lists of registries, devices, certificates, and topic aliases.

Install and configure the Yandex Cloud CLI if you have not done that already:

yc init

Verify your active CLI configuration:

yc config get cloud-id
yc config get folder-id
yc config get compute-default-zone

Create an export folder:

FOLDER_ID="$(yc config get folder-id)"
mkdir -p iot-export

Export the list of registries and brokers. Provide --folder-id explicitly for your export not to depend on the folder specified in your local CLI configuration:

yc iot registry list --folder-id "$FOLDER_ID" --format json > iot-export/registries.json
yc iot broker list --folder-id "$FOLDER_ID" --format json > iot-export/brokers.json

Export your devices, registry certificates, and topic aliases:

for registry_id in $(jq -r '.[].id' iot-export/registries.json); do
  yc iot device list \
    --folder-id "$FOLDER_ID" \
    --registry-id "$registry_id" \
    --format json > "iot-export/devices-$registry_id.json"

  yc iot registry certificate list \
    --folder-id "$FOLDER_ID" \
    --registry-id "$registry_id" \
    --format json > "iot-export/registry-certs-$registry_id.json"

  yc iot registry list-device-topic-aliases \
    --folder-id "$FOLDER_ID" \
    "$registry_id" \
    --format json > "iot-export/aliases-$registry_id.json"
done

Export your device certificates:

for file in iot-export/devices-*.json; do
  registry_id="${file#iot-export/devices-}"
  registry_id="${registry_id%.json}"

  mkdir -p "iot-export/device-certs-$registry_id"

  jq -r '.[].id' "$file" | while read -r device_id; do
    yc iot device certificate list \
      --folder-id "$FOLDER_ID" \
      --device-id "$device_id" \
      --format json > "iot-export/device-certs-$registry_id/$device_id.json"
  done
done

Deploy Mosquitto on a VMDeploy Mosquitto on a VM

The example below shows how to deploy Mosquitto on a VM in Docker. For a production scenario, configure disk backup, monitoring, and a recovery procedure.

Create a VMCreate a VM

Create a Yandex Compute Cloud VM with a Linux image and a public IP address or access via a network load balancer.

Recommendations:

• Use a dedicated service account for infrastructure management.
• Open only port 8883/TCP in the security group's inbound rules.
• Restrict SSH access.
• Store TLS keys and passwords in Yandex Lockbox or another secure secret vault.
• Connect Yandex Monitoring and Yandex Cloud Logging.

Before you create a VM, select an SSH key and an administrative CIDR allowed to be used for SSH access:

ls -1 ~/.ssh/*.pub

SSH_PUBLIC_KEY="$HOME/.ssh/<key>.pub"
SSH_PRIVATE_KEY="${SSH_PUBLIC_KEY%.pub}"

ssh-keygen -y -f "$SSH_PRIVATE_KEY" > /tmp/mqtt-broker-key.pub
diff -u /tmp/mqtt-broker-key.pub "$SSH_PUBLIC_KEY"
chmod 600 "$SSH_PRIVATE_KEY"

ADMIN_CIDR="<your-admin-ip-or-vpn-cidr>/32"

If your private key is protected by a passphrase, load it into ssh-agent before you run automation or use a separate key for migration. For CI and sandbox, do not rely solely on external HTTP services for auto-detection of your public IP: the source SSH address may be different from what such services see. In this case, specify ADMIN_CIDR explicitly (e.g., your VPN or bastion host CIDR).

Create a security group and configure the minimal inbound rules:

yc vpc security-group create \
  --name mqtt-broker-sg \
  --network-name default

yc vpc security-group update-rules mqtt-broker-sg \
  --add-rule "direction=ingress,protocol=tcp,port=8883,v4-cidrs=0.0.0.0/0" \
  --add-rule "direction=ingress,protocol=tcp,port=22,v4-cidrs=${ADMIN_CIDR}"

Some Yandex Cloud CLI versions do not support the yc vpc security-group rule add command. Use yc vpc security-group update-rules --add-rule ... or look up the syntax via yc vpc security-group update-rules --help.

Example of creating a VM using the CLI:

yc compute instance create \
  --name mqtt-broker \
  --zone ru-central1-a \
  --network-interface subnet-name=default-ru-central1-a,nat-ip-version=ipv4,security-group-names=mqtt-broker-sg \
  --create-boot-disk image-folder-id=standard-images,image-family=ubuntu-2204-lts,size=15,type=network-hdd \
  --memory 2 \
  --cores 2 \
  --core-fraction 20 \
  --metadata-from-file user-data=cloud-init.yaml \
  --metadata ssh-keys="ubuntu:$(cat "$SSH_PUBLIC_KEY")"

Install Docker and Docker Compose on the VM.

Prepare the Mosquitto catalogPrepare the Mosquitto catalog

Run the following commands:

mkdir -p mosquitto/config/certs mosquitto/data mosquitto/log
cd mosquitto

sudo chown -R 1883:1883 data log

Create a file named docker-compose.yaml:

services:
  mosquitto:
    image: eclipse-mosquitto:2
    container_name: mosquitto
    restart: unless-stopped
    ports:
      - "8883:8883"
    volumes:
      - ./config:/mosquitto/config:ro
      - ./data:/mosquitto/data
      - ./log:/mosquitto/log

Prepare the server TLS certificatePrepare the server TLS certificate

For a production environment, use a certificate for the domain devices will be using to connect to the broker, for example:

mqtt.example.com

If devices can validate public root certificates, a public TLS certificate may be used.

If devices use a custom root certificate, put the server certificate and the key in this catalog:

mosquitto/config/certs/server.crt
mosquitto/config/certs/server.key

If devices are authenticated using client certificates, also include the root certificate used to sign the client certificates:

mosquitto/config/certs/device-ca.crt

Option 1. Configure username and passwordOption 1. Configure username and password

Create a file named config/mosquitto.conf:

persistence true
persistence_location /mosquitto/data/
autosave_interval 60

log_dest stdout

listener 8883 0.0.0.0
protocol mqtt

certfile /mosquitto/config/certs/server.crt
keyfile /mosquitto/config/certs/server.key

allow_anonymous false
password_file /mosquitto/config/passwords
acl_file /mosquitto/config/acl

Create a password file. For the first user, use the -c flag and do not pre-create the file: mosquitto_passwd -c must create the new password_file itself.

docker run --rm -it \
  -v "$PWD/config:/mosquitto/config" \
  eclipse-mosquitto:2 \
  mosquitto_passwd -c /mosquitto/config/passwords <device_id>

For non-interactive setup, use -b but do not save plaintext passwords in shell history:

docker run --rm \
  -v "$PWD/config:/mosquitto/config" \
  eclipse-mosquitto:2 \
  mosquitto_passwd -b -c /mosquitto/config/passwords <device_id> '<device_password>'

For subsequent users, run the command without -c to avoid overwriting the file:

docker run --rm -it \
  -v "$PWD/config:/mosquitto/config" \
  eclipse-mosquitto:2 \
  mosquitto_passwd /mosquitto/config/passwords <another_device_id>

Add a dedicated user for backend applications:

docker run --rm -it \
  -v "$PWD/config:/mosquitto/config" \
  eclipse-mosquitto:2 \
  mosquitto_passwd /mosquitto/config/passwords Backend

Option 2. Configure client certificatesOption 2. Configure client certificates

If devices already use X.509 certificates and you want keep it that way, configure mTLS.

Create a file named config/mosquitto.conf:

persistence true
persistence_location /mosquitto/data/
autosave_interval 60

log_dest stdout

listener 8883 0.0.0.0
protocol mqtt

certfile /mosquitto/config/certs/server.crt
keyfile /mosquitto/config/certs/server.key
cafile /mosquitto/config/certs/device-ca.crt

require_certificate true
use_identity_as_username true

allow_anonymous false
acl_file /mosquitto/config/acl

With this configuration, Mosquitto will use the common name from the client certificate as username. Ensure the common name matches the device ID or another stable ID used in your ACL.

Configure an ACLConfigure an ACL

Create a file named config/acl.

ACL example when the username matches the device ID:

pattern write $devices/%u/events/#
pattern write $devices/%u/state/#
pattern read $devices/%u/commands/#
pattern read $devices/%u/config/#

pattern write devices/%u/events/#
pattern write devices/%u/state/#
pattern read devices/%u/commands/#
pattern read devices/%u/config/#

user Backend
topic read $devices/+/events/#
topic read $devices/+/state/#
topic write $devices/+/commands/#
topic write $devices/+/config/#

topic read devices/+/events/#
topic read devices/+/state/#
topic write devices/+/commands/#
topic write devices/+/config/#

In this example:

• Device can publish only its own events and state.
• Device can read only its own commands and configuration.
• Backend user can read telemetry and publish commands.

If you have used registry topics, add rules for registries/... or generate individual access rules for each device.

Run MosquittoRun Mosquitto

Run the following commands:

docker compose up -d
docker compose logs -f mosquitto

Make sure the broker listens on port 8883:

docker compose ps

Alternatively, install Mosquitto via an apt packageAlternatively, install Mosquitto via an apt package

If your infrastructure does not use Docker, you can install Mosquitto as an Ubuntu system package. While being simpler for cloud-init, this approach requires careful configuration.

sudo apt-get update
sudo apt-get install -y mosquitto mosquitto-clients

Put additional settings in /etc/mosquitto/conf.d/<name>.conf, not the main /etc/mosquitto/mosquitto.conf.

persistence true
persistence_location /var/lib/mosquitto/

Error: Duplicate persistence_location value in configuration.

sudo grep -RhE '^(persistence|persistence_location|password_file|acl_file|listener)' \
  /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/

If deploying Mosquitto via cloud-init, pay attention to the module execution order: write_files runs before packages, so the mosquitto group may not exist yet. Write files as root:root, then change the owner in runcmd after installing the package.

It is safer to provide ACLs containing the $devices/... and $registries/... topics to cloud-init in Base64 encoding: this way $ will not be interpreted by the shell wrapper during metadata preparation.

The /etc/mosquitto/passwd file you provide to cloud-init must be generated by a compatible mosquitto_passwd version. If the target VM uses the Mosquitto 2.0.18 Ubuntu package, do not inject a file created by a newer local mosquitto_passwd version because your broker may reject users due to unreadable hash.

Example cloud-init snippet for apt package:

#cloud-config
packages:
  - mosquitto
  - mosquitto-clients

write_files:
  - path: /etc/mosquitto/conf.d/iot-migration.conf
    owner: root:root
    permissions: "0644"
    content: |
      listener 8883 0.0.0.0
      protocol mqtt
      certfile /etc/mosquitto/certs/server.crt
      keyfile /etc/mosquitto/certs/server.key
      allow_anonymous false
      password_file /etc/mosquitto/passwd
      acl_file /etc/mosquitto/acl
  - path: /etc/mosquitto/acl
    owner: root:root
    permissions: "0640"
    encoding: b64
    content: <base64-encoded-acl>
  - path: /etc/mosquitto/passwd
    owner: root:root
    permissions: "0640"
    encoding: b64
    content: <base64-encoded-password-file>

runcmd:
  - mkdir -p /etc/mosquitto/certs
  - chown mosquitto:mosquitto /etc/mosquitto/acl /etc/mosquitto/passwd
  - chmod 0640 /etc/mosquitto/acl /etc/mosquitto/passwd
  - systemctl reset-failed mosquitto
  - systemctl restart mosquitto
  - systemctl is-active mosquitto

The Mosquitto package may attempt to start the service immediately after the installation, even before the execution of runcmd (i.e., before certificates and access file permissions are ready). Several failed starts may cause systemd to set the service to the Start request repeated too quickly state. Run systemctl reset-failed mosquitto before the final restart to clear this start-limit.

Do not add persistence or persistence_location to this conf.d file because these are already present in Ubuntu's default /etc/mosquitto/mosquitto.conf.

After you edit the configuration, run these commands:

sudo systemctl restart mosquitto
sudo systemctl status mosquitto
sudo ss -tlnp | grep 8883

For the Mosquitto 2.0.18 Ubuntu package, do not use mosquitto -t for configuration validation, as this version treats -t as an unknown modifier. If the service fails to start, check the diagnostics:

sudo journalctl -u mosquitto -n 100 --no-pager
sudo grep -RhE '^(persistence|persistence_location|password_file|acl_file|listener)' \
  /etc/mosquitto/mosquitto.conf /etc/mosquitto/conf.d/

If using apt, remember to grant permissions for data and log directories to the mosquitto user (chown -R mosquitto:mosquitto /var/lib/mosquitto /var/log/mosquitto) and to open only 8883/TCP in the security group.

Test the connectionTest the connection

Install the Mosquitto client utilities on your production or test VM.

Subscribe to a topic as a backend user:

mosquitto_sub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -u Backend \
  -P '<Backend_password>' \
  -t '$devices/+/events/#' \
  -q 1

Publish a test message as a device:

mosquitto_pub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -u <device_id> \
  -P '<device_password>' \
  -t '$devices/<device_id>/events' \
  -m '{"temperature":25}' \
  -q 1

Check the retained message for the state persistent topic analog:

mosquitto_pub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -u <device_id> \
  -P '<device_password>' \
  -t '$devices/<device_id>/state' \
  -m '{"online":true}' \
  -q 1 \
  -r

Subscribe to this topic as a new client. It should immediately receive the last retained message:

mosquitto_sub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -u Backend \
  -P '<Backend_password>' \
  -t '$devices/<device_id>/state' \
  -q 1

Check device isolation via ACLCheck device isolation via ACL

ACL is the most common point where errors reveal themselves only after migration. Always make certain that one device cannot read from or write to another device's topics.

Scenario 1. Foreign commands are not received. First, as <device_a>, subscribe to a topic of another device (<device_b>):

mosquitto_sub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -u <device_a> \
  -P '<device_a_password>' \
  -t '$devices/<device_b>/commands' \
  -q 1 \
  -W 5

The -W 5 parameter limits execution time to 5 seconds. While the subscriber is working, publish a backend-initiated message to $devices/<device_b>/commands and make sure the subscriber receives nothing. The broker logs for <device_a> should now contain an ACL denial entry.

Scenario 2. Foreign events cannot be published. As <device_a>, try publishing to another device's topic:

mosquitto_pub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -u <device_a> \
  -P '<device_a_password>' \
  -t '$devices/<device_b>/events' \
  -m '{"spoofed":true}' \
  -q 1

The command may appear to succeed on the client side, but the message should not reach the backend subscription on $devices/<device_b>/events/#. Check specifically for the absence of the message at the backend subscriber and an ACL denial in broker logs, not just the mosquitto_pub response code.

Scenario 3. Anonymous connection is denied. Connect without credentials:

mosquitto_sub \
  -h mqtt.example.com \
  -p 8883 \
  --cafile rootCA.crt \
  -t '$devices/<device_a>/events' \
  -W 5

The client should receive the Connection Refused: not authorised error.

Replace triggersReplace triggers

In Yandex IoT Core, you could configure a trigger to invoke a Cloud Functions function or a Serverless Containers container upon receiving an MQTT message. In a custom MQTT broker this integration should be implemented via a dedicated subscriber service.

Below is an example of a simple Python subscriber which receives an MQTT message and forwards it to an HTTP handler.

Create a file named subscriber.py:

import base64
import json
import os
import queue
import threading
import time
from urllib import request

import paho.mqtt.client as mqtt


MQTT_HOST = os.environ["MQTT_HOST"]
MQTT_PORT = int(os.getenv("MQTT_PORT", "8883"))
MQTT_USERNAME = os.environ["MQTT_USERNAME"]
MQTT_PASSWORD = os.environ["MQTT_PASSWORD"]
MQTT_TOPIC = os.getenv("MQTT_TOPIC", "$devices/+/events/#")
MQTT_CLIENT_ID = os.getenv("MQTT_CLIENT_ID", "iot-core-migration-subscriber")
MQTT_CA_FILE = os.getenv("MQTT_CA_FILE", "rootCA.crt")
WORKER_COUNT = int(os.getenv("WORKER_COUNT", "4"))

HANDLER_URL = os.environ["HANDLER_URL"]

# The queue and worker pool separate receiving MQTT messages and their handling.
# If the HTTP request is handled directly in on_message, a slow handler
# will block the MQTT network loop, causing the client to disconnect due to keepalive timeout.
message_queue: queue.Queue = queue.Queue(maxsize=10000)


def build_event(msg: mqtt.MQTTMessage) -> dict:
    payload_base64 = base64.b64encode(msg.payload).decode("ascii")
    topic_parts = msg.topic.split("/")

    event = {
        "mqtt_topic": msg.topic,
        "payload": payload_base64,
        "is_base64": True,
        "qos": msg.qos,
        "retain": msg.retain,
    }

    if len(topic_parts) >= 2 and topic_parts[0] in ("$devices", "devices"):
        event["device_id"] = topic_parts[1]

    if len(topic_parts) >= 2 and topic_parts[0] in ("$registries", "registries"):
        event["registry_id"] = topic_parts[1]

    return event


def send_to_handler(event: dict) -> None:
    body = json.dumps(event).encode("utf-8")
    http_request = request.Request(
        HANDLER_URL,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )

    with request.urlopen(http_request, timeout=10) as response:
        response.read()


def on_connect(client, userdata, flags, rc):
    if rc != 0:
        print(f"MQTT connection failed with rc={rc}")
        return

    print(f"Connected to MQTT broker. Subscribing to {MQTT_TOPIC}")
    client.subscribe(MQTT_TOPIC, qos=1)


def worker() -> None:
    while True:
        event = message_queue.get()
        try:
            for attempt in range(3):
                try:
                    send_to_handler(event)
                    break
                except Exception as error:
                    print(f"Handler call failed, attempt={attempt + 1}, error={error}")
                    time.sleep(2 ** attempt)
            else:
                print(f"Message was not processed: topic={event.get('mqtt_topic')}")
        finally:
            message_queue.task_done()


def on_message(client, userdata, msg):
    # Put the event into the queue and immediately return control to the network loop:
    # put() pauses reception only when all workers are busy.
    message_queue.put(build_event(msg))


for _ in range(WORKER_COUNT):
    threading.Thread(target=worker, daemon=True).start()

client = mqtt.Client(client_id=MQTT_CLIENT_ID)
client.username_pw_set(MQTT_USERNAME, MQTT_PASSWORD)
client.tls_set(ca_certs=MQTT_CA_FILE)

client.on_connect = on_connect
client.on_message = on_message

client.connect(MQTT_HOST, MQTT_PORT, keepalive=60)
client.loop_forever()

Create a file named requirements.txt:

paho-mqtt==1.6.1

Run the subscriber:

export MQTT_HOST=mqtt.example.com
export MQTT_PORT=8883
export MQTT_USERNAME=Backend
export MQTT_PASSWORD='<Backend_password>'
export MQTT_TOPIC='$devices/+/events/#'
export MQTT_CA_FILE=./rootCA.crt
export HANDLER_URL='https://<your-handler-url>'

python3 -m pip install -r requirements.txt
python3 subscriber.py

Such a subscriber can be deployed:

• On a Compute Cloud VM instance.
• In a Managed Service for Kubernetes cluster.
• In a Serverless Containers container if the application maintains a persistent MQTT connection.
• Alongside other backend services used to process telemetry.

Replace export to Yandex Data StreamsReplace export to Yandex Data Streams

If you were using message export from Yandex IoT Core to Yandex Data Streams data streams, use one of the following options:

• MQTT subscriber reads messages from Mosquitto and sends them to a Yandex Data Streams stream.
• Backend application publishes data to MQTT and to a Yandex Data Streams stream at the same time.
• Stream processing is performed by a separate service subscribed to MQTT topics.

General principle:

Mosquitto → MQTT subscriber → Yandex Data Streams

In your implementation, consider the following:

• Message size.
• Message order.
• Resending on errors.
• Idempotency of writes.
• Write latency and error monitoring.

Migrate devicesMigrate devices

After you have prepared the broker and backend integrations, begin migrating devices.

We recommend the following order:

1. Create a new DNS name for the MQTT broker, e.g., mqtt.example.com.
2. Configure a TLS certificate for this address.
3. Create a test device or test device group.
4. Check that events and state can be published.
5. Check that commands and config can be received.
6. Check that the backend receives and correctly processes messages.
7. Migrate a small pilot group of devices.
8. Compare telemetry between the old and new environments.
9. Migrate remaining devices in batches.
10. After the migration is complete, disable publishing to Yandex IoT Core.

If device firmware supports simultaneous publishing to two MQTT brokers, use a dual-send period. This will help you compare data and reduce the risk of telemetry loss.

Delete test resourcesDelete test resources

If you had created a test VM or security group for smoke testing, delete them only after you make certain that production traffic has already been routed to the permanent endpoint:

yc compute instance delete mqtt-broker
yc vpc security-group delete mqtt-broker-sg

Do not add the --quiet flag to these commands without checking your installed Yandex Cloud CLI version because some versions do not support it for yc compute instance delete. For non-interactive cleanup, first check the current syntax:

yc compute instance delete --help

Deploying Mosquitto in Yandex Managed Service for KubernetesDeploying Mosquitto in Yandex Managed Service for Kubernetes

Mosquitto can be deployed in a Managed Service for Kubernetes cluster. For a production environment, use:

• StatefulSet
• PersistentVolumeClaim for /mosquitto/data
• Secret for TLS keys and passwords
• ConfigMap for mosquitto.conf and ACLs
• Service of the LoadBalancer type for TCP port 8883
• PodDisruptionBudget
• Regular persistent volume backups

Example of Service:

apiVersion: v1
kind: Service
metadata:
  name: mosquitto
spec:
  type: LoadBalancer
  selector:
    app: mosquitto
  ports:
    - name: mqtts
      port: 8883
      targetPort: 8883
      protocol: TCP

When to choose TBMQ Community EditionWhen to choose TBMQ Community Edition

TBMQ Community Edition is an open-source MQTT broker designed for high workloads and clustered architecture.

Consider TBMQ if:

• A single Mosquitto broker is becoming a bottleneck.
• You need horizontal scalability.
• Long-term message delivery is critical.
• You need broker cluster-level fault tolerance.
• You are ready to maintain Apache Kafka®, a database, and other components.

Typical architecture:

Devices → TBMQ → Kafka → backend / handlers / storages

TBMQ is more difficult to operate than Mosquitto but better suited for scenarios where the MQTT broker is a critical, high-load component.

You can keep your TBMQ or ThingsBoard database in Yandex Managed Service for PostgreSQL to avoid administering a DBMS by yourself or deploy it in the cluster alongside your other components.

When to choose ThingsBoard Community EditionWhen to choose ThingsBoard Community Edition

ThingsBoard Community Edition is an open-source IoT platform featuring an MQTT API, device management, telemetry storage, dashboards, and a rule engine.

Consider ThingsBoard CE if:

• You need more than just MQTT message exchange but an IoT platform as well.
• Your users need a UI for devices and telemetry.
• You need dashboards and a rule engine.
• You want to manage device account data within the platform.
• You are ready to adapt your topics, payload, and backend integrations to ThingsBoard.

ThingsBoard CE is not a drop-in replacement for Yandex IoT Core. Devices and backend applications typically need to be adapted to ThingsBoard's API and data model.

Typical architecture:

Devices → ThingsBoard CE → Rule Engine → dashboard / backend / storages

You can also build an architecture where Mosquitto remains the MQTT broker and ThingsBoard receives data via an integration or an API gateway:

Devices → Mosquitto → ThingsBoard CE → dashboard / Rule Engine

What is not migrated automaticallyWhat is not migrated automatically

When you migrate to an open-source solution, the following components are not migrated automatically:

• Registries and devices as managed Yandex Cloud resources.
• Topic aliases.
• Device and registry passwords.
• IAM integrations.
• Message publication via the Yandex IoT Core gRPC interface.
• Triggers.
• Export to Yandex Data Streams.
• Yandex IoT Core monitoring topics.
• Behavior of persistent topics without retained-messages configured.
• Yandex Cloud-level roles and access permissions.

For each item, you need to implement a replacement:

• Registries and devices: a custom database, broker configuration, or IoT platform.
• Passwords: a new password_file, Dynamic Security plugin for Mosquitto, or account data managed by an IoT platform.
• Certificates: a custom root certificate and mTLS.
• Aliases: explicit MQTT topics or a compatibility layer in the backend.
• Publication via gRPC: no direct compatible replacement exists because Yandex IoT Core was using IAM-based authentication. If this scenario is what you need, build a dedicated backend adapter with a suitable authorization scheme to accept gRPC/API requests from your systems and publish messages to Mosquitto.
• Triggers: an MQTT subscriber.
• Export: an MQTT subscriber able to write to Yandex Data Streams streams or another storage system.
• Monitoring: broker metrics, logs, health checks, and custom heartbeat topics.

Operational recommendationsOperational recommendations

SecuritySecurity

• Use MQTTS on port 8883.
• Disable anonymous access.
• Restrict devices with the help of ACLs.
• Do not use a single shared password for all devices.
• Store your passwords, private keys, and the root certificate in a secure vault.
• Rotate your credentials regularly.
• Limit inbound traffic to security groups.
• Do not make the administrative interfaces open to the internet.

ReliabilityReliability

• Store broker persistence data on a persistent disk.
• Configure backups for the disk or data catalog.
• Document your recovery procedure.
• Test recovery in a test environment.
• Use QoS 1 for critical messages.
• Make sure retained messages are getting saved.
• For critical scenarios, consider an active/passive setup or a clustered broker.

ObservabilityObservability

• Collect your Mosquitto logs in Yandex Cloud Logging.
• Set up port 8883 unavailability alerts.
• Monitor CPU, RAM, disk, and network traffic of your VMs or pods.
• Track the number of connected clients.
• Track authentication errors.
• Add backend subscriber metrics: message count, handling errors, delivery latency.

PerformancePerformance

• Perform load testing before migrating all devices.
• Check your CPU, RAM, and network bandwidth limits.
• Separate topics for backend subscribers not to receive irrelevant messages.
• Do not use wildcard subscriptions that are to broad unless you have to.
• For big installations, consider sharding or migration to a clustered broker.

Migration planMigration plan

Use this checklist for migration control:

• Target solution selected: Eclipse Mosquitto, TBMQ CE, or ThingsBoard CE.
• List of registries, devices, certificates, and topic aliases exported.
• New device passwords generated; a plan for their delivery to firmware is in place.
• New MQTT endpoint created.
• TLS configured.
• Device authentication configured.
• ACLs configured for both prefixes, $devices/... and devices/... (if migrating part of devices in a phased way).
• events publication checked.
• state publication with retained messages checked.
• commands delivery checked.
• config delivery checked.
• ACL isolation smoke test done (device cannot read/write foreign topics; anonymous connection is rejected).
• Mosquitto configuration checked for duplicate settings (persistence, persistence_location, listener, etc., appear exactly once across all configuration files).
• Trigger replacement mechanism implemented.
• Export to Yandex Data Streams replacement mechanism implemented (if used).
• Logs, metrics, and alerts configured.
• Backups configured.
• Pilot migration of small device group completed.
• Data comparison between old and new environments completed.
• All devices migrated.
• Production system dependency on Yandex IoT Core removed.

Useful linksUseful links

• Eclipse Mosquitto guides
• TBMQ guides
• ThingsBoard guides
• Yandex Compute Cloud guides
• Yandex Managed Service for Kubernetes guides
• Yandex Object Storage guides
• Yandex Lockbox guides
• Yandex Monitoring guides
• Yandex Cloud Logging guides