Autoscaling an instance group to process messages from a queue in Yandex Message Queue

Written by

Updated at December 4, 2024

Getting started
- Required paid resources
Prepare the environment
- Create the supporting Yandex Cloud resources
- Set up the AWS CLI
Create a queue in Message Queue
Create an image with an application
Create an instance group
Check the instance group scalability
Delete the resources you created

This is an integration scenario for Yandex Compute Cloud, Yandex Message Queue, and Yandex Monitoring.

In this scenario, we will create an instance group. All instances in the group process messages from the same Message Queue queue (for example, they receive messages and delete them after a while). The number of enqueued messages is registered in a Monitoring metric, and the instance group is autoscaled based on this metric.

The opportunistic shutdown strategy has been chosen for scaling: the instances stop on their own once they process all the messages and are then deleted from the group as the group size is reduced. This ensures that the instances will not be forcibly deleted and will process the last messages even if the queue empties before message processing is completed.

To perform this use case:

If you no longer need the resources you created, delete them.

Getting started

Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

We will use a folder named example-folder as an example.

Required paid resources

The infrastructure cost includes:

Fee for using VMs and storing an image (see Compute Cloud pricing).
Fee for using public IP addresses and outgoing VM traffic (see Virtual Private Cloud pricing).
Fee for queue requests (see Message Queue pricing).

Prepare the environment

Create the supporting Yandex Cloud resources

Management console

CLI

API

Create a service account named queue-autoscale-sa and assign it the editor role for example-folder.
1. In the management console, select example-folder.
2. In the list of services, select Identity and Access Management.
3. Click Create service account.
4. In the Name field, specify queue-autoscale-sa.
5. Click Add role and select the editor role.
6. Click Create.
Create a static access key to enable the service account to work with Message Queue and save the key to the access_key file:
1. In the list of service accounts, select queue-autoscale-sa.
2. Click Create new key and select Create static access key.
3. In the Description field, specify Message Queue.
4. Click Create.
5. Add the key ID and secret key to the access_key file in the following format:
```
access_key:
  key_id: <key_ID>
secret: <secret_key>
```
Create a cloud network named queue-autoscale-network:
1. In the management console, select example-folder.
2. In the list of services, select Virtual Private Cloud.
3. Click Create network.
4. In the Name field, specify queue-autoscale-network.
5. Disable the Create subnets option.
6. Click Create network.
Create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone:
1. In the list of networks, select queue-autoscale-network.
2. Click Create subnet.
3. Specify the subnet parameters:
  - Name: queue-autoscale-subnet-d.
  - Zone: ru-central1-d.
  - CIDR: 192.168.1.0/24.
4. Click Create subnet.

Create a service account named queue-autoscale-sa:

yc iam service-account create --name queue-autoscale-sa

Where --name is the service account name: queue-autoscale-sa.

Result:

id: aje6brh817ro********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:19:10.153346507Z"
name: queue-autoscale-sa

Assign the editor role for example-folder to the service account:
```
yc resource-manager folder add-access-binding example-folder \
  --service-account-name queue-autoscale-sa \
  --role editor
```
Where:
- --service-account-name: Name of the service account created at step 1, queue-autoscale-sa.
- --role: Role to assign, which is editor.
Result:
```
done (1s)
```

Create a static access key to enable the service account to work with Message Queue and save the key to the access_key file:

yc iam access-key create \
  --service-account-name queue-autoscale-sa \
  --description "Message Queue" \
| tee access_key

Where:

--service-account-name: Name of the service account created at step 1, queue-autoscale-sa.
--description: Key description.

Result:

access_key:
  id: ajevb6bx51w3********
  service_account_id: aje6brh817ro********
  created_at: "2021-10-04T16:25:19.143847764Z"
  description: Message Queue
  key_id: g3VybpVKdq_Y********
secret: WVUqnwmC4LSUep0GTKGkbcht9K3Xav7V********

Create a cloud network named queue-autoscale-network:

yc vpc network create queue-autoscale-network

Result:

id: enpabce123hd********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:28:15.905337847Z"
name: queue-autoscale-network

Create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone:

yc vpc subnet create queue-autoscale-subnet-d \
  --network-name queue-autoscale-network 
  --range 192.168.1.0/24 
  --zone ru-central1-d

Where:

--network-name: Name of the network you are creating the subnet in, queue-autoscale-network.
--range: Subnet CIDR.
--zone: Availability zone to create the subnet in, ru-central1-d.

Result:

id: e1lnabc23r1c********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:29:12.450858436Z"
name: queue-autoscale-subnet-d
network_id: enpabce123hd********
zone_id: ru-central1-d
v4_cidr_blocks:
- 192.168.1.0/24

Create a service account named queue-autoscale-sa using the ServiceAccountService/Create gRPC API call or the create REST API method. The response will contain the service account ID.
Assign the service account the editor role for example-folder using the FolderService/UpdateAccessBindings gRPC API call or the updateAccessBindings REST API method. In the request body, specify the service account ID.
Create a static access key to enable the service account to work with Message Queue and save the key to the access_key file:
1. Use the AccessKeyService/Create gRPC API call or the create REST API method.
2. Add the key ID and secret key to the access_key file in the following format:
```
access_key:
  key_id: <key_ID>
secret: <secret_key>
```
Create a cloud network named queue-autoscale-network using the NetworkService/Create gRPC API call or the create REST API method. The response will contain the network ID.
Create a subnet named queue-autoscale-subnet-d in the ru-central1-d availability zone with the 192.168.1.0/24 CIDR using the SubnetService/Create gRPC API call or the create REST API method. In the request body, specify the network ID.

Set up the AWS CLI

Note

You will use the AWS CLI to perform the final step of the script: checking instance group scalability. This is why you must install and configure it, even if you aren't going to use it for any other steps.

Download and install the AWS CLI as described in its documentation.
Configure the AWS CLI to work with Yandex Cloud:
1. In your home folder, create a file named .aws/credentials with the previously received static access key and its ID:
```
[default]
    aws_access_key_id     = <key_ID>
    aws_secret_access_key = <secret_key>
```
2. In your home folder, create a file named .aws/config with the default region set to ru-central1:
```
[default]
    region = ru-central1
```

Create a queue in Message Queue

Management console

AWS CLI

API

Create a queue named queue-autoscale-queue:
1. In the management console, select example-folder.
2. In the list of services, select Message Queue.
3. Click Create queue.
4. In the Name field, specify queue-autoscale-queue.
5. Click Create.
Save the queue URL to the queue file:
1. In the list of queues, select queue-autoscale-queue.
2. Copy the queue URL.
3. Paste the URL into the queue file in the following format:
```
QueueUrl: <queue_URL>
```

Create a queue named queue-autoscale-queue and save its URL to the queue file:

aws sqs create-queue \
  --endpoint https://message-queue.api.cloud.yandex.net \
  --queue-name queue-autoscale-queue \
  --output yaml \
| tee queue

Where:

--endpoint: Root URL of the Message Queue API, which is https://message-queue.api.cloud.yandex.net.
--queue-name: Queue name, queue-autoscale-queue.

Result:

QueueUrl: https://message-queue.api.cloud.yandex.net/b1gvlrnlei4l********/dj6000000003********/queue-autoscale-queue

Create a queue named queue-autoscale-queue using the CreateQueue API method. The response will contain the queue URL.
Paste the queue URL into the queue file in the following format:
```
QueueUrl: <queue_URL>
```

Create an image with an application

If you do not have Packer installed yet, download and install it by following the instructions on the official website. You can also download a Packer distribution for your platform from a mirror.

Note

Yandex Cloud requires Packer 1.5 or higher.

Configure the Yandex Compute Builder plugin:

Create a file named config.pkr.hcl with the following contents:

packer {
  required_plugins {
    yandex = {
      version = ">= 1.1.2"
      source  = "github.com/hashicorp/yandex"
    }
  }
}

Install the plugin:

packer init <path_to_config.pkr.hcl>

Result:

Installed plugin github.com/hashicorp/yandex v1.1.2 in ...

Download the archive with files for the image (ZIP, 3 KB) and unpack it:

wget https://storage.yandexcloud.net/doc-files/queue-autoscale-server.zip
unzip queue-autoscale-server.zip

In the server-packer.json file, change the object in the builders array:
- In the service_account_id field, enter the queue-autoscale-sa service account ID.
  How do I find out the service account ID
  Management console
  
  CLI
  
  API
  
  In the management console, select example-folder.
  
  In the list of services, select Identity and Access Management.
  
  In the left-hand panel, select Service accounts.
  
  Select the queue-autoscale-sa service account from the list that opens.
  
  Copy the service account ID.
  
  Run the following command:
  
  yc iam service-account get queue-autoscale-sa
  
  Result:
  
  id: aje6brh817ro******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:19:10.153346507Z" name: queue-autoscale-sa
  
  Use the list REST API method for the ServiceAccount resource or the ServiceAccountService/List gRPC API call.
- In the folder_id field, specify the ID of the folder in Yandex Cloud.
  How do I find out the folder ID
  Management console
  
  CLI
  
  API
  
  On the cloud page, find example-folder in the Folders list and copy its ID.
  
  Run the following command:
  
  yc resource-manager folder get example-folder
  
  Result:
  
  id: b1g9hv2loamq******** cloud_id: b1g2y61sxwqx******** created_at: "2021-10-04T13:21:32.788067492Z" name: example-folder status: ACTIVE
  
  Use the list REST API method for the Folder resource or the FolderService/List gRPC API call.
- In the subnet_id field, specify the queue-autoscale-subnet-d subnet ID.
  How do I find out the subnet ID
  Management console
  
  CLI
  
  API
  
  In the management console, select the example-folder folder.
  
  In the list of services, select Virtual Private Cloud.
  
  Go to the Subnets tab.
  
  In the list of subnets, find queue-autoscale-subnet-d and copy its ID.
  
  Run the following command:
  
  yc vpc subnet get queue-autoscale-subnet-d
  
  Result:
  
  id: e1lnabc23r1c******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:29:12.450858436Z" name: queue-autoscale-subnet-d network_id: enpabce123hd******** zone_id: ru-central1-d v4_cidr_blocks: - 192.168.1.0/24
  
  Use the list REST API method for the Subnet resource or the SubnetService/List gRPC API call.
Get an IAM token to enable the service account to work with Compute Cloud and save the token to the key.json file:
CLI

API
```
yc iam key create \
  --service-account-name queue-autoscale-sa \
  --description "Compute Cloud" \
  --output key.json
```
Where:
- --service-account-name: Name of the previously created service account, queue-autoscale-sa.
- --description: Key description.
- --output: Path to the file to save the key to, which is key.json.
Result:
```
id: ajevh3a0hrqf********
service_account_id: aje6brh817ro********
created_at: "2021-10-04T16:35:19.057777570Z"
description: Compute Cloud
key_algorithm: RSA_2048
```
Use the create REST API method for the Key resource or the KeyService/Create gRPC API call.

Run the following command to create an image:

packer build server-packer.json

Result:

...
Build 'builder' finished after 2 minutes 48 seconds.

==> Wait completed after 2 minutes 48 seconds

==> Builds finished. The artifacts of successful builds are:
--> builder: A disk image was created: queue-autoscale-image-v1633354734 (id: fd8hlbuhjc4m********) with family name queue-autoscale-image

Create an instance group

Management console

CLI

API

In the management console, select example-folder.
In the list of services, select Compute Cloud.
In the left-hand panel, select Instance groups.
Click Create group of virtual machines.
Under Basic parameters:
- In the Name field, specify queue-autoscale-ig.
- Select Service account queue-autoscale-sa.
Under Allocation, select ru-central1-d in the Availability zone field.
Under Instance template, click Define and do the following in the window that opens:
1. Under Boot disk image, go to the Custom tab and click Select.
2. In the window that opens, select the created image as a boot disk:
  1. Go to the Image tab.
  2. Select an image whose name begins with queue-autoscale-image.
  3. Click Apply.
3. Under Network settings:
  - In the Network field, select queue-autoscale-network.
  - In the Public address field, select Auto so that your VMs have internet access and can receive messages from the queue.
4. Under Access:
  - In the Service account field, select queue-autoscale-sa.
  - Enter your username and the contents of the public key to connect to the VM.
5. Click Save.
Under Allow when creating and updating, specify the following values:
- Stop simultaneously: 1.
- Stop VMs by strategy: Opportunistic.
Under Scaling, specify the following values:
- Type: Automatic.
- Autoscaling type: Regional.
- Minimum number of instances per zone: 0.
- Maximum group size: 5.
- Utilization measurement period: 60 seconds.
- Instance warmup period: 0 minutes.
- Stabilization period: 5 minutes.
- Initial group size: 1.
Under Metrics, specify the following values:
- Metric: Yandex Monitoring.
- Folder ID: message-queue.
- Metric name: queue.messages.stored_count.
- Labels: queue as the key and queue-autoscale-queue as the value.
- Metric type: GAUGE.
- Rule type: WORKLOAD.
- Target: 5.
Click Create.

Get the ID of the image you created at the previous steps:

yc compute image get-latest-from-family queue-autoscale-image

Result:

id: fd8hlbuhjc4m********
folder_id: b0g12ga82bcv********
created_at: "2021-10-04T16:38:54.345168059Z"
name: queue-autoscale-image-v1633354734
description: Image for autoscaling an instance group based on the size of a queue
family: queue-autoscale-image
storage_size: "3858759680"
min_disk_size: "5368709120"
product_ids:
- f2e6fnj3erf1********
status: READY
os:
  type: LINUX

Edit the spec.yaml instance group specification file:
- In the folder_id field, specify the ID of the folder in Yandex Cloud.
- In the image_id field, specify the ID of the created image.
- In the network_id field, specify the queue-autoscale-network ID, and in the subnet_ids field, the queue-autoscale-subnet-d ID.
  How do I find out the network ID
  CLI
  
  Management console
  
  API
  
  Run this command:
  
  yc vpc network get queue-autoscale-network
  
  Result:
  
  id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
  
  In the management console, select the example-folder folder.
  
  In the list of services, select Virtual Private Cloud.
  
  In the list of networks, find queue-autoscale-network and copy its ID.
  
  Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the service_account_id field, enter the queue-autoscale-sa service account ID.
To learn how to get the IDs of resources (except for images and networks), see step 3 in the section Create an image with an application.
Create an instance group named queue-autoscale-ig based on the following specification:
```
yc compute instance-group create --file spec.yaml
```
Where --file is the path to the instance group specification file: spec.yaml.

Get the ID of the previously created image from the queue-autoscale-image family. To do this, use the ImageService/GetLatestByFamily gRPC API call or the getLatestByFamily REST API method.
Edit the spec.yaml instance group specification file:
- In the folder_id field, specify the ID of the folder in Yandex Cloud.
- In the image_id field, specify the ID of the created image.
- In the network_id field, specify the queue-autoscale-network ID, and in the subnet_ids field, the queue-autoscale-subnet-d ID.
  How do I find out the network ID
  Management console
  
  CLI
  
  API
  
  In the management console, select the example-folder folder.
  
  In the list of services, select Virtual Private Cloud.
  
  In the list of networks, find queue-autoscale-network and copy its ID.
  
  Run the following command:
  
  yc vpc network get queue-autoscale-network
  
  Result:
  
  id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
  
  Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the service_account_id field, enter the queue-autoscale-sa service account ID.
To learn how to get the IDs of resources (except for images and networks), see step 3 in the section Create an image with an application.
Create an instance group named queue-autoscale-ig based on the spec.yaml specification. To do this, use the InstanceGroupService/CreateFromYaml gRPC API call or the createFromYaml REST API method.

Check the instance group scalability

Edit the messages.sh file: in the --queue-url parameter, specify the URL of the queue from the queue file.
Run the messages.sh file to start sending 100 messages to queue-autoscale-queue one by one.
Open dashboards in Yandex Monitoring:
Management console
1. Open the dashboard with information about the instance group:
  1. In Monitoring, select the Compute — Instance Groups service dashboard.
  2. In the Instance Group ID field, select queue-autoscale-ig.
2. Open the dashboard with queue information:
  1. In Monitoring, select the Message Queue service dashboard.
  2. In the Queue ID field, select queue-autoscale-queue.
Check the dashboard data to make sure the following events took place in this order:
1. Number of enqueued messages in queue-autoscale-queue reached 100 (the Message Queue dashboard or the Sent messages, count and Messages in queue, count charts).
2. Instances from the queue-autoscale-ig group started receiving messages (the Received messages, count chart) and deleting them from the queue (the Deleted messages, count and Messages in queue, count charts).
3. While processing the messages, the number of instances in the group increased from one to five and, after all messages were processed, the group became empty (the Compute — Instance Groups dashboard, the Number of instances in zone B chart).

Delete the resources you created

To shut down the infrastructure and stop paying for the resources you created:

Delete the queue-autoscale-ig instance group.
Delete the image from the queue-autoscale-image family.
Delete the queue-autoscale-queue queue.
Delete the queue-autoscale-subnet-d subnet.
Delete the queue-autoscale-network network.
Delete the queue-autoscale-sa service account.

Autoscaling an instance group to process messages from a queue in Yandex Message Queue

Getting startedGetting started

Required paid resourcesRequired paid resources

Prepare the environmentPrepare the environment

Create the supporting Yandex Cloud resourcesCreate the supporting Yandex Cloud resources

Set up the AWS CLISet up the AWS CLI

Create a queue in Message QueueCreate a queue in Message Queue

Create an image with an applicationCreate an image with an application

Create an instance groupCreate an instance group

Check the instance group scalabilityCheck the instance group scalability

Delete the resources you createdDelete the resources you created

Was the article helpful?

Getting started

Required paid resources

Prepare the environment

Create the supporting Yandex Cloud resources

Set up the AWS CLI

Create a queue in Message Queue

Create an image with an application

Create an instance group

Check the instance group scalability

Delete the resources you created