Autoscaling an instance group to process messages in Yandex Message Queue

Written by

Updated at February 27, 2025

Getting started
- Required paid resources
Prepare your environment
- Create auxiliary Yandex Cloud resources
- Set up the AWS CLI
Create a queue in Message Queue
Create an application image
Create an instance group
Check the instance group scalability
Delete the resources you created

This is an integration scenario for Yandex Compute Cloud, Yandex Message Queue, and Yandex Monitoring.

In this scenario, we will create an instance group. All instances in this group process messages from the same queue in Message Queue (in our example, they receive messages and delete them after a while). The system registers the number of enqueued messages in the Monitoring metric and autoscales the instance group based on it.

For scaling, we are going to use the opportunistic shutdown strategy. This means the VMs get stopped once they are done processing all messages, and only then get removed from the group. This ensures that the system will not forcibly remove the VMs and they will process all messages even if the queue gets empty before the processing is complete.

To work with this scenario, you will need to:

If you no longer need the resources you created, delete them.

Getting started

Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

As an example, we will use a folder named example-folder.

Required paid resources

The infrastructure cost includes:

Fee for using VMs and storing an image (see Compute Cloud pricing).
Fee for public IP addresses and outbound VM traffic (see Virtual Private Cloud pricing).
Fee for queue requests (see Message Queue pricing).

Prepare your environment

Create auxiliary Yandex Cloud resources

Management console

CLI

API

Create the queue-autoscale-sa service account and assign it the editor role for example-folder.
1. In the management console, select example-folder.
2. In the list of services, select Identity and Access Management.
3. Click Create service account.
4. In the Name field, specify queue-autoscale-sa.
5. Click Add role and select the editor role.
6. Click Create.
Create a static access key to enable the service account to work with Message Queue and save the key into the access_key file:
1. In the list of service accounts, select queue-autoscale-sa.
2. Click Create new key and select Create static access key.
3. In the Description field, specify Message Queue.
4. Click Create.
5. Add the key ID and secret key to the access_key file in the following format:
```
access_key:
  key_id: <key_ID>
secret: <secret_key>
```
Create a cloud network named queue-autoscale-network:
1. In the management console, select example-folder.
2. In the list of services, select Virtual Private Cloud.
3. Click Create network.
4. In the Name field, specify queue-autoscale-network.
5. Disable the Create subnets option.
6. Click Create network.
Create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone:
1. In the list of networks, select queue-autoscale-network.
2. Click Create subnet.
3. Specify the subnet parameters:
  - Name: queue-autoscale-subnet-d
  - Zone: ru-central1-d
  - CIDR: 192.168.1.0/24
4. Click Create subnet.

Create the queue-autoscale-sa service account:

yc iam service-account create --name queue-autoscale-sa

Where --name is the service account name: queue-autoscale-sa.

Result:

id: aje6brh817ro********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:19:10.153346507Z"
name: queue-autoscale-sa

Assign the editor role for example-folder to the service account:
```
yc resource-manager folder add-access-binding example-folder \
  --service-account-name queue-autoscale-sa \
  --role editor
```
Where:
- --service-account-name: Service account name, queue-autoscale-sa.
- --role: editor role.
Result:
```
done (1s)
```

Create a static access key to enable the service account to work with Message Queue and save the key into the access_key file:

yc iam access-key create \
  --service-account-name queue-autoscale-sa \
  --description "Message Queue" \
| tee access_key

Where:

--service-account-name: Service account name, queue-autoscale-sa.
--description: Key description.

Result:

access_key:
  id: ajevb6bx51w3********
  service_account_id: aje6brh817ro********
  created_at: "2021-10-04T16:25:19.143847764Z"
  description: Message Queue
  key_id: g3VybpVKdq_Y********
secret: WVUqnwmC4LSUep0GTKGkbcht9K3Xav7V********

Create a cloud network named queue-autoscale-network:

yc vpc network create queue-autoscale-network

Result:

id: enpabce123hd********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:28:15.905337847Z"
name: queue-autoscale-network

Create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone:

yc vpc subnet create queue-autoscale-subnet-d \
  --network-name queue-autoscale-network 
  --range 192.168.1.0/24 
  --zone ru-central1-d

Where:

--network-name: queue-autoscale-network, the one where you are creating the subnet.
--range: Subnet CIDR.
--zone: Subnet availability zone, ru-central1-d.

Result:

id: e1lnabc23r1c********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:29:12.450858436Z"
name: queue-autoscale-subnet-d
network_id: enpabce123hd********
zone_id: ru-central1-d
v4_cidr_blocks:
- 192.168.1.0/24

To create the queue-autoscale-sa service account, use the ServiceAccountService/Create gRPC API call or the create REST API method. The response will contain the service account ID.
To assign the service account the editor role for example-folder, use the FolderService/UpdateAccessBindings gRPC API call or the updateAccessBindings REST API method. In the request body, specify the service account ID.
Create a static access key to enable the service account to work with Message Queue and save the key into the access_key file:
1. Use the AccessKeyService/Create gRPC API call or the create REST API method.
2. Add the key ID and secret key to the access_key file in the following format:
```
access_key:
  key_id: <key_ID>
secret: <secret_key>
```
To create the queue-autoscale-network cloud network, use the NetworkService/Create gRPC API call or the create REST API method. The response will contain the network ID.
To create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone with the 192.168.1.0/24 CIDR, use the SubnetService/Create gRPC API call or the create REST API method. In the request body, specify the network ID.

Set up the AWS CLI

Note

You will need the AWS CLI to complete the final step of this scenario, i.e., to check the instance group scalability. This is why you have to install and configure it.

Download and install the AWS CLI as described in the relevant AWS guide.
Configure the AWS CLI to work with Yandex Cloud:
1. In your home folder, create a file named .aws/credentials with the previously received static access key and its ID:
```
[default]
    aws_access_key_id     = <key_ID>
    aws_secret_access_key = <secret_key>
```
2. In your home folder, create the .aws/config file; in this file, specify the default region: ru-central1:
```
[default]
    region = ru-central1
```

Create a queue in Message Queue

Management console

AWS CLI

API

Create a queue named queue-autoscale-queue:
1. In the management console, select example-folder.
2. In the list of services, select Message Queue.
3. Click Create queue.
4. In the Name field, specify queue-autoscale-queue.
5. Click Create.
Save the queue URL to the queue file:
1. In the list of queues, select queue-autoscale-queue.
2. Copy the queue URL.
3. Paste the URL into the queue file in the following format:
```
QueueUrl: <queue_URL>
```

Create a queue named queue-autoscale-queue and save its URL to the queue file:

aws sqs create-queue \
  --endpoint https://message-queue.api.cloud.yandex.net \
  --queue-name queue-autoscale-queue \
  --output yaml \
| tee queue

Where:

--endpoint: Message Queue API root URL, https://message-queue.api.cloud.yandex.net.
--queue-name: Queue name, queue-autoscale-queue.

Result:

QueueUrl: https://message-queue.api.cloud.yandex.net/b1gvlrnlei4l********/dj6000000003********/queue-autoscale-queue

To create the queue named queue-autoscale-queue, use the CreateQueue API method. The response will contain the queue URL.
Paste the queue URL into the queue file in the following format:
```
QueueUrl: <queue_URL>
```

Create an application image

If you do not have Packer installed, download and install it by following this tutorial on the official HashiCorp website. You can also download Packer for your platform here.

Note

Yandex Cloud requires Packer version 1.5 or higher.

Configure the Yandex Compute Builder plugin:

Create the config.pkr.hcl file with the following contents:

packer {
  required_plugins {
    yandex = {
      version = ">= 1.1.2"
      source  = "github.com/hashicorp/yandex"
    }
  }
}

Install the plugin:

packer init <path_to_config.pkr.hcl>

Result:

Installed plugin github.com/hashicorp/yandex v1.1.2 in ...

Download the image file archive (ZIP, 3 KB) and unpack it:

wget https://storage.yandexcloud.net/doc-files/queue-autoscale-server.zip
unzip queue-autoscale-server.zip

In the server-packer.json file, change the object in the builders array:
- In the service_account_id field, specify the queue-autoscale-sa service account ID.
  How to find the service account ID
  Management console
  
  CLI
  
  API
  
  In the management console, select example-folder.
  
  In the list of services, select Identity and Access Management.
  
  In the left-hand panel, select Service accounts.
  
  In the list that opens, select the queue-autoscale-sa service account.
  
  Copy the service account ID.
  
  Run the following command:
  
  yc iam service-account get queue-autoscale-sa
  
  Result:
  
  id: aje6brh817ro******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:19:10.153346507Z" name: queue-autoscale-sa
  
  Use the list REST API method for the ServiceAccount resource or the ServiceAccountService/List gRPC API call.
- In the folder_id field, specify the folder ID in Yandex Cloud.
  How to find the folder ID
  Management console
  
  CLI
  
  API
  
  On the cloud page, find example-folder in the Folders list and copy its ID.
  
  Run the following command:
  
  yc resource-manager folder get example-folder
  
  Result:
  
  id: b1g9hv2loamq******** cloud_id: b1g2y61sxwqx******** created_at: "2021-10-04T13:21:32.788067492Z" name: example-folder status: ACTIVE
  
  Use the list REST API method for the Folder resource or the FolderService/List gRPC API call.
- In the subnet_id field, specify the queue-autoscale-subnet-d subnet ID.
  How to find the subnet ID
  Management console
  
  CLI
  
  API
  
  In the management console, select example-folder.
  
  In the list of services, select Virtual Private Cloud.
  
  Navigate to the Subnets tab.
  
  In the list of subnets, find queue-autoscale-subnet-d and copy its ID.
  
  Run the following command:
  
  yc vpc subnet get queue-autoscale-subnet-d
  
  Result:
  
  id: e1lnabc23r1c******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:29:12.450858436Z" name: queue-autoscale-subnet-d network_id: enpabce123hd******** zone_id: ru-central1-d v4_cidr_blocks: - 192.168.1.0/24
  
  Use the list REST API method for the Subnet resource or the SubnetService/List gRPC API call.
Get an IAM token allowing the service account to work with Compute Cloud and save it to the key.json file:
CLI

API
```
yc iam key create \
  --service-account-name queue-autoscale-sa \
  --description "Compute Cloud" \
  --output key.json
```
Where:
- --service-account-name: Service account name, queue-autoscale-sa.
- --description: Key description.
- --output: Path to the key storage file, key.json.
Result:
```
id: ajevh3a0hrqf********
service_account_id: aje6brh817ro********
created_at: "2021-10-04T16:35:19.057777570Z"
description: Compute Cloud
key_algorithm: RSA_2048
```
Use the create REST API method for the Key resource or the KeyService/Create gRPC API call.

To create an image, run the following command:

packer build server-packer.json

Result:

...
Build 'builder' finished after 2 minutes 48 seconds.

==> Wait completed after 2 minutes 48 seconds

==> Builds finished. The artifacts of successful builds are:
--> builder: A disk image was created: queue-autoscale-image-v1633354734 (id: fd8hlbuhjc4m********) with family name queue-autoscale-image

Create an instance group

Management console

CLI

API

In the management console, select example-folder.
In the list of services, select Compute Cloud.
In the left-hand panel, select Instance groups.
Click Create group of virtual machines.
Under Basic parameters:
- In the Name field, specify queue-autoscale-ig.
- Select Service account queue-autoscale-sa.
Under Allocation, select ru-central1-d in the Availability zone field.
Under Instance template, click Define and do the following in the window that opens:
1. Under Boot disk image, navigate to the Custom tab and click Select.
2. In the window that opens, select the previously created image as a boot disk:
  1. Navigate to the Image tab.
  2. Select an image with a name that begins with queue-autoscale-image.
  3. Click Apply.
3. Under Network settings:
  - In the Network field, select queue-autoscale-network.
  - In the Public address field, select Auto so that your VMs have the internet access and can receive messages from the queue.
4. Under Access:
  - In the Service account field, select queue-autoscale-sa.
  - Enter your username and the contents of the public key to connect to the VM.
5. Click Save.
Under Changes during creation and updates, specify the following values:
- Stop simultaneously: 1
- Stop VMs by strategy: Opportunistic
Under Scaling, specify the following values:
- Type: Automatic
- Autoscaling type: Regional
- Minimum number of instances per zone: 0
- Maximum group size: 5
- Utilization measurement period: 60 seconds
- Instance warmup period: 0 minutes
- Stabilization period: 5 minutes
- Initial group size: 1
Under Metrics, specify the following values:
- Metric: Yandex Monitoring
- Folder ID: message-queue
- Metric name: queue.messages.stored_count
- Labels: queue as the key and queue-autoscale-queue as the value
- Metric type: GAUGE
- Rule type: WORKLOAD
- Target: 5
Click Create.

Get the ID of the image you created in the previous steps:

yc compute image get-latest-from-family queue-autoscale-image

Result:

id: fd8hlbuhjc4m********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:38:54.345168059Z"
name: queue-autoscale-image-v1633354734
description: Image for autoscaling an instance group based on the size of a queue
family: queue-autoscale-image
storage_size: "3858759680"
min_disk_size: "5368709120"
product_ids:
- f2e6fnj3erf1********
status: READY
os:
  type: LINUX

Edit the instance group specification file: spec.yaml:
- In the folder_id field, specify the folder ID in Yandex Cloud.
- In the image_id field, specify the image ID.
- In the network_id field, specify the queue-autoscale-network ID, and in the subnet_ids field, the queue-autoscale-subnet-d ID.
  How to find the network ID
  CLI
  
  Management console
  
  API
  
  Run this command:
  
  yc vpc network get queue-autoscale-network
  
  Result:
  
  id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
  
  In the management console, select example-folder.
  
  In the list of services, select Virtual Private Cloud.
  
  In the list of networks, find queue-autoscale-network and copy its ID.
  
  Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the service_account_id field, specify the queue-autoscale-sa service account ID.
To learn how to get the required IDs (except for the image ID and network ID), see this section: Create an application image.
Create the queue-autoscale-ig instance group using the following command:
```
yc compute instance-group create --file spec.yaml
```
Where --file is the path to the instance group specification file: spec.yaml.

To get the ID of the previously created image from the queue-autoscale-image family, use the ImageService/GetLatestByFamily gRPC API call or the getLatestByFamily REST API method.
Edit the spec.yaml instance group specification file:
- In the folder_id field, specify the folder ID in Yandex Cloud.
- In the image_id field, specify the image ID.
- In the network_id field, specify the queue-autoscale-network ID, and in the subnet_ids field, the queue-autoscale-subnet-d ID.
  How to find the network ID
  Management console
  
  CLI
  
  API
  
  In the management console, select example-folder.
  
  In the list of services, select Virtual Private Cloud.
  
  In the list of networks, find queue-autoscale-network and copy its ID.
  
  Run the following command:
  
  yc vpc network get queue-autoscale-network
  
  Result:
  
  id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
  
  Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the service_account_id field, specify the queue-autoscale-sa service account ID.
To learn how to get the required IDs (except for the image ID and network ID), see this section: Create an application image.
To create the queue-autoscale-ig instance group to the spec.yaml specification, use the InstanceGroupService/CreateFromYaml gRPC API call or the createFromYaml REST API method.

Check the instance group scalability

Edit the messages.sh file: in the --queue-url parameter, specify the URL of the queue from the queue file.
Run messages.sh. It will start sending messages to queue-autoscale-queue one by one; overall, it will send 100 messages.
Open dashboards in Yandex Monitoring:
Management console
1. Open the instance group information dashboard:
  1. In Monitoring, select the Compute — Instance Groups dashboard.
  2. In the Instance Group ID field, select queue-autoscale-ig.
2. Open the queue information dashboard:
  1. In Monitoring, select the Message Queue dashboard.
  2. In the Queue ID field, select queue-autoscale-queue.
Check the dashboard data to make sure the following events took place in this order:
1. Number of enqueued messages in queue-autoscale-queue reached 100 (the Message Queue dashboard, the Sent messages, count and Messages in queue, count charts).
2. VM instances from the queue-autoscale-ig group started receiving messages (the Received messages, count chart) and deleting them from the queue (the Deleted messages, count and Messages in queue, count charts).
3. While processing the messages, the number of instances in the group increased from one to five and, after all messages were processed, the group became empty (the Compute — Instance Groups dashboard, the Number of instances in zone B chart).

Delete the resources you created

To shut down the infrastructure and stop paying for the resources you created:

Delete the queue-autoscale-ig instance group.
Delete the image from the queue-autoscale-image family.
Delete queue-autoscale-queue.
Delete queue-autoscale-subnet-d.
Delete queue-autoscale-network.
Delete the queue-autoscale-sa service account.

Autoscaling an instance group to process messages in Yandex Message Queue

Getting startedGetting started

Required paid resourcesRequired paid resources

Prepare your environmentPrepare your environment

Create auxiliary Yandex Cloud resourcesCreate auxiliary Yandex Cloud resources

Set up the AWS CLISet up the AWS CLI

Create a queue in Message QueueCreate a queue in Message Queue

Create an application imageCreate an application image

Create an instance groupCreate an instance group

Check the instance group scalabilityCheck the instance group scalability

Delete the resources you createdDelete the resources you created

Was the article helpful?

Getting started

Required paid resources

Prepare your environment

Create auxiliary Yandex Cloud resources

Set up the AWS CLI

Create a queue in Message Queue

Create an application image

Create an instance group

Check the instance group scalability

Delete the resources you created