Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Autoscaling an instance group to process messages in Yandex Message Queue

Written by
Updated at August 14, 2025

This is an integration scenario for Yandex Compute Cloud, Yandex Message Queue, and Yandex Monitoring.

In this scenario, we will create an instance group. All instances in this group process messages from the same queue in Message Queue (in our example, they receive messages and delete them after a while). The system registers the number of enqueued messages in the Monitoring metric and autoscales the instance group based on it.

For scaling, we are going to use the opportunistic shutdown strategy. This means the VMs get stopped once they are done processing all messages, and only then get removed from the group. This ensures that the system will not forcibly remove the VMs and they will process all messages even if the queue gets empty before the processing is complete.

To work with this scenario, you will need to:

  1. Set up your environment.
  2. Create a queue in Message Queue.
  3. Create an application image.
  4. Create an instance group.
  5. Check the instance group scalability.

If you no longer need the resources you created, delete them.

Getting started

Sign up for Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or create a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure.

Learn more about clouds and folders here.

As an example, we will use a folder named example-folder.

The infrastructure cost includes:

Set up your environment

Create auxiliary Yandex Cloud resources

  1. Create the queue-autoscale-sa service account and assign it the editor role for example-folder.

    1. In the management console, select example-folder.
    2. In the list of services, select Identity and Access Management.
    3. Click Create service account.
    4. In the Name field, specify queue-autoscale-sa.
    5. Click Add role and select the editor role.
    6. Click Create.

  2. Create a static access key to enable the service account to work with Message Queue and save the key into the access_key file:

    1. In the list of service accounts, select queue-autoscale-sa.

    2. Click Create new key and select Create static access key.

    3. In the Description field, specify Message Queue.

    4. Click Create.

    5. Add the key ID and secret key to the access_key file in the following format:

      access_key:
  key_id: <key_ID>
secret: <secret_key>

  3. Create a cloud network named queue-autoscale-network:

    1. In the management console, select example-folder.
    2. In the list of services, select Virtual Private Cloud.
    3. Click Create network.
    4. In the Name field, specify queue-autoscale-network.
    5. Disable Create subnets.
    6. Click Create network.

  4. Create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone:

    1. In the list of networks, select queue-autoscale-network.

    2. Click Create subnet.

    3. Specify the subnet parameters:

      • Name: queue-autoscale-subnet-d
      • Availability zone: ru-central1-d
      • CIDR: 192.168.1.0/24

    4. Click Create subnet.

  1. Create a service account named queue-autoscale-sa:

    yc iam service-account create --name queue-autoscale-sa

    Where --name is the service account name, queue-autoscale-sa.

    Result:

    id: aje6brh817ro********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:19:10.153346507Z"
name: queue-autoscale-sa

  2. Assign the editor role for example-folder to the service account:

    yc resource-manager folder add-access-binding example-folder \
  --service-account-name queue-autoscale-sa \
  --role editor

    Where:

    • --service-account-name: Service account name, queue-autoscale-sa.
    • --role: editor role.

    Result:

    done (1s)

  3. Create a static access key to enable the service account to work with Message Queue and save the key into the access_key file:

    yc iam access-key create \
  --service-account-name queue-autoscale-sa \
  --description "Message Queue" \
| tee access_key

    Where:

    • --service-account-name: Service account name, queue-autoscale-sa.
    • --description: Key description.

    Result:

    access_key:
  id: ajevb6bx51w3********
  service_account_id: aje6brh817ro********
  created_at: "2021-10-04T16:25:19.143847764Z"
  description: Message Queue
  key_id: g3VybpVKdq_Y********
secret: WVUqnwmC4LSUep0GTKGkbcht9K3Xav7V********

  4. Create a cloud network named queue-autoscale-network:

    yc vpc network create queue-autoscale-network

    Result:

    id: enpabce123hd********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:28:15.905337847Z"
name: queue-autoscale-network

  5. Create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone:

    yc vpc subnet create queue-autoscale-subnet-d \
  --network-name queue-autoscale-network 
  --range 192.168.1.0/24 
  --zone ru-central1-d

    Where:

    • --network-name: queue-autoscale-network, the one where you are creating the subnet.
    • --range: Subnet CIDR.
    • --zone: Subnet availability zone, ru-central1-d.

    Result:

    id: e1lnabc23r1c********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:29:12.450858436Z"
name: queue-autoscale-subnet-d
network_id: enpabce123hd********
zone_id: ru-central1-d
v4_cidr_blocks:
- 192.168.1.0/24

  1. To create the queue-autoscale-sa service account, use the ServiceAccountService/Create gRPC API call or the create REST API method. The response will contain the service account ID.

  2. To assign the service account the editor role for example-folder, use the FolderService/UpdateAccessBindings gRPC API call or the updateAccessBindings REST API method. In the request body, specify the service account ID.

  3. Create a static access key to enable the service account to work with Message Queue and save the key into the access_key file:

    1. Use the AccessKeyService/Create gRPC API call or the create REST API method.

    2. Add the key ID and secret key to the access_key file in the following format:

      access_key:
  key_id: <key_ID>
secret: <secret_key>

  4. To create the queue-autoscale-network cloud network, use the NetworkService/Create gRPC API call or the create REST API method. The response will contain the network ID.

  5. To create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone with the 192.168.1.0/24 CIDR, use the SubnetService/Create gRPC API call or the create REST API method. In the request body, specify the network ID.

Set up the AWS CLI

Note

You will need the AWS CLI to complete the final step of this scenario, i.e., to check the instance group scalability. This is why you have to install and configure it.

  1. Download and install the AWS CLI as described in the relevant AWS guide.

  2. Configure the AWS CLI to work with Yandex Cloud:

    1. In your home folder, create a file named .aws/credentials with the previously received static access key and its ID:

      [default]
    aws_access_key_id     = <key_ID>
    aws_secret_access_key = <secret_key>

    2. In your home folder, create the .aws/config file; in this file, specify the default region: ru-central1:

      [default]
    region = ru-central1

Create a queue in Message Queue

  1. Create a queue named queue-autoscale-queue:

    1. In the management console, select example-folder.
    2. In the list of services, select Message Queue.
    3. Click Create queue.
    4. In the Name field, specify queue-autoscale-queue.
    5. Click Create.

  2. Save the queue URL to the queue file:

    1. In the list of queues, select queue-autoscale-queue.

    2. Copy the queue URL.

    3. Paste the URL into the queue file in the following format:

      QueueUrl: <queue_URL>

  1. Create a queue named queue-autoscale-queue and save its URL to the queue file:

    aws sqs create-queue \
  --endpoint https://message-queue.api.cloud.yandex.net \
  --queue-name queue-autoscale-queue \
  --output yaml \
| tee queue

    Where:

    • --endpoint: Message Queue API root URL, https://message-queue.api.cloud.yandex.net.
    • --queue-name: Queue name, queue-autoscale-queue.

    Result:

    QueueUrl: https://message-queue.api.cloud.yandex.net/b1gvlrnlei4l********/dj6000000003********/queue-autoscale-queue

  1. To create the queue named queue-autoscale-queue, use the CreateQueue API method. The response will contain the queue URL.

  2. Paste the queue URL into the queue file in the following format:

    QueueUrl: <queue_URL>

Create an application image

  1. If you do not have Packer installed, download and install it by following this tutorial on the official HashiCorp website. You can also download Packer for your platform here.

    Note

    Yandex Cloud requires Packer 1.5 or higher.

  2. Configure the Yandex Compute Builder plugin:

    1. Create a file named config.pkr.hcl with the following contents:

      packer {
  required_plugins {
    yandex = {
      version = ">= 1.1.2"
      source  = "github.com/hashicorp/yandex"
    }
  }
}

    2. Install the plugin:

      packer init <path_to_config.pkr.hcl>

      Result:

      Installed plugin github.com/hashicorp/yandex v1.1.2 in ...

  3. Download the image file archive (ZIP, 3 KB) and unpack it:

    wget https://storage.yandexcloud.net/doc-files/queue-autoscale-server.zip
unzip queue-autoscale-server.zip

  4. In the server-packer.json file, change the object in the builders array:

    • In the service_account_id field, specify the queue-autoscale-sa ID.

      How to find the service account ID
      1. In the management console, select example-folder.
      2. In the list of services, select Identity and Access Management.
      3. In the left-hand panel, select Service accounts.
      4. In the list that opens, select queue-autoscale-sa.
      5. Copy the service account ID.

      Run this command:

      yc iam service-account get queue-autoscale-sa

      Result:

      id: aje6brh817ro********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:19:10.153346507Z"
name: queue-autoscale-sa

      Use the list REST API method for the ServiceAccount resource or the ServiceAccountService/List gRPC API call.

    • In the folder_id field, specify the Yandex Cloud folder ID.

      How to find the folder ID

      On the cloud page, find example-folder in the Folders list and copy its ID.

      Run the following command:

      yc resource-manager folder get example-folder

      Result:

      id: b1g9hv2loamq********
cloud_id: b1g2y61sxwqx********
created_at: "2021-10-04T13:21:32.788067492Z"
name: example-folder
status: ACTIVE

      Use the list REST API method for the Folder resource or the FolderService/List gRPC API call.

    • In the subnet_id field, specify the queue-autoscale-subnet-d ID.

      How to find the subnet ID
      1. In the management console, select example-folder.
      2. In the list of services, select Virtual Private Cloud.
      3. Navigate to the Subnets tab.
      4. In the list of subnets, find queue-autoscale-subnet-d and copy its ID.

      Run the following command:

      yc vpc subnet get queue-autoscale-subnet-d

      Result:

      id: e1lnabc23r1c********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:29:12.450858436Z"
name: queue-autoscale-subnet-d
network_id: enpabce123hd********
zone_id: ru-central1-d
v4_cidr_blocks:
- 192.168.1.0/24

      Use the list REST API method for the Subnet resource or the SubnetService/List gRPC API call.

  5. Get an IAM token allowing the service account to work with Compute Cloud and save it to the key.json file:

    yc iam key create \
  --service-account-name queue-autoscale-sa \
  --description "Compute Cloud" \
  --output key.json

    Where:

    • --service-account-name: Service account name, queue-autoscale-sa.
    • --description: Key description.
    • --output: Path to the key storage file, key.json.

    Result:

    id: ajevh3a0hrqf********
service_account_id: aje6brh817ro********
created_at: "2021-10-04T16:35:19.057777570Z"
description: Compute Cloud
key_algorithm: RSA_2048

    Use the create REST API method for the Key resource or the KeyService/Create gRPC API call.

  6. To create an image, run the following command:

    packer build server-packer.json

    Result:

    ...
Build 'builder' finished after 2 minutes 48 seconds.

==> Wait completed after 2 minutes 48 seconds

==> Builds finished. The artifacts of successful builds are:
--> builder: A disk image was created: queue-autoscale-image-v1633354734 (id: fd8hlbuhjc4m********) with family name queue-autoscale-image

Create an instance group

  1. In the management console, select example-folder.

  2. From the list of services, select Compute Cloud.

  3. In the left-hand panel, select Instance groups.

  4. Click Create group of virtual machines.

  5. Under Basic parameters:

    • In the Name field, specify queue-autoscale-ig.
    • Select queue-autoscale-sa for Service account.

  6. Under Allocation, select ru-central1-d in the Availability zone field.

  7. Under Instance template, click Define and do the following in the window that opens:

    1. Under Boot disk image, navigate to the Custom tab and click Select.

    2. In the window that opens, select the previously created image as a boot disk:

      1. Navigate to the Image tab.
      2. Select an image with a name that begins with queue-autoscale-image.
      3. Click Apply.

    3. Under Network settings:

      • In the Network field, select queue-autoscale-network.
      • In the Public address field, select Auto so that your VMs have the internet access and can receive messages from the queue.

    4. Under Access:

      • In the Service account field, select queue-autoscale-sa.
      • Enter your username and the contents of the public key to connect to the VM.

    5. Click Save.

  8. Under Changes during creation and updates, specify the following values:

    • Stop simultaneously: 1
    • Stop VMs by strategy: Opportunistic

  9. Under Scaling, specify the following values:

    • Type: Automatic
    • Autoscaling type: Regional
    • Minimum number of instances per zone: 0
    • Maximum group size: 5
    • Utilization measurement period: 60 seconds
    • Instance warmup period: 0 minutes
    • Stabilization period: 5 minutes
    • Initial group size: 1

  10. Under Metrics, specify the following values:

    • Metric: Yandex Monitoring
    • Folder ID: message-queue
    • Metric name: queue.messages.stored_count
    • Labels: queue as the key and queue-autoscale-queue as the value
    • Metric type: GAUGE
    • Rule type: WORKLOAD
    • Target: 5

  11. Click Create.

  1. Get the ID of the image you created in the previous steps:

    yc compute image get-latest-from-family queue-autoscale-image

    Result:

    id: fd8hlbuhjc4m********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:38:54.345168059Z"
name: queue-autoscale-image-v1633354734
description: Image for autoscaling an instance group based on the size of a queue
family: queue-autoscale-image
storage_size: "3858759680"
min_disk_size: "5368709120"
product_ids:
- f2e6fnj3erf1********
status: READY
os:
  type: LINUX

  2. Edit the spec.yaml file specifying the instance group:

    • In the folder_id field, specify the Yandex Cloud folder ID.

    • In the image_id field, specify the image ID.

    • In the network_id field, specify the queue-autoscale-network ID, and in the subnet_ids field, the queue-autoscale-subnet-d ID.

      How to find the network ID

      Run this command:

      yc vpc network get queue-autoscale-network

      Result:

      id: enpabce123hd********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:28:15.905337847Z"
name: queue-autoscale-network
      1. In the management console, select example-folder.
      2. In the list of services, select Virtual Private Cloud.
      3. In the list of networks, find queue-autoscale-network and copy its ID.

      Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.

    • In the service_account_id field, specify the queue-autoscale-sa service account ID.

    To learn how to get the required IDs (except for the image ID and network ID), see Create an application image.

  3. Create the queue-autoscale-ig instance group using the following command:

    yc compute instance-group create --file spec.yaml

    Where --file is the path to the instance group specification file: spec.yaml.

  1. To get the ID of the previously created image from the queue-autoscale-image family, use the ImageService/GetLatestByFamily gRPC API call or the getLatestByFamily REST API method.

  2. Edit the spec.yaml file specifying the instance group:

    • In the folder_id field, specify the Yandex Cloud folder ID.

    • In the image_id field, specify the image ID.

    • In the network_id field, specify the queue-autoscale-network ID, and in the subnet_ids field, the queue-autoscale-subnet-d ID.

      How to find the network ID
      1. In the management console, select example-folder.
      2. In the list of services, select Virtual Private Cloud.
      3. In the list of networks, find queue-autoscale-network and copy its ID.

      Run the following command:

      yc vpc network get queue-autoscale-network

      Result:

      id: enpabce123hd********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:28:15.905337847Z"
name: queue-autoscale-network

      Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.

    • In the service_account_id field, specify the queue-autoscale-sa service account ID.

    To learn how to get the required IDs (except for the image ID and network ID), see this section: Create an application image.

  3. To create the queue-autoscale-ig instance group as specified in spec.yaml, use the InstanceGroupService/CreateFromYaml gRPC API call or the createFromYaml REST API method.

Check the instance group scalability

  1. Edit the messages.sh file: in the --queue-url parameter, specify the URL of the queue from the queue file.

  2. Run messages.sh. It will start sending messages to queue-autoscale-queue one by one; overall, it will send 100 messages.

  3. Open dashboards in Yandex Monitoring:

    1. Open the instance group information dashboard:

      1. In Monitoring, select the Compute — Instance Groups dashboard.
      2. In the Instance Group ID field, select queue-autoscale-ig.

    2. Open the queue information dashboard:

      1. In Monitoring, select the Message Queue dashboard.
      2. In the Queue ID field, select queue-autoscale-queue.

  4. Check the dashboard data to make sure the following events took place in this order:

    1. Number of enqueued messages in queue-autoscale-queue reached 100 (the Message Queue dashboard, the Sent messages, count and Messages in queue, count charts).
    2. VM instances from the queue-autoscale-ig group started receiving messages (the Received messages, count chart) and deleting them from the queue (the Deleted messages, count and Messages in queue, count charts).
    3. While processing the messages, the number of instances in the group increased from one to five and, after all messages were processed, the group became empty (the Compute — Instance Groups dashboard, the Number of instances in zone B chart).

Delete the resources you created

To shut down the infrastructure and stop paying for the resources you created:

  1. Delete the queue-autoscale-ig instance group.
  2. Delete the image from the queue-autoscale-image family.
  3. Delete queue-autoscale-queue.
  4. Delete queue-autoscale-subnet-d.
  5. Delete queue-autoscale-network.
  6. Delete the queue-autoscale-sa service account.
Previous
Terraform
Next
Updating an instance group under load