Autoscaling an instance group to process messages in Yandex Message Queue
This is an integration scenario for Yandex Compute Cloud, Yandex Message Queue, and Yandex Monitoring.
In this scenario, we will create an instance group. All instances in this group process messages from the same queue in Message Queue (in our example, they receive messages and delete them after a while). The system registers the number of enqueued messages in the Monitoring metric and autoscales the instance group based on it.
For scaling, we are going to use the opportunistic shutdown strategy. This means the VMs get stopped once they are done processing all messages, and only then get removed from the group. This ensures that the system will not forcibly remove the VMs and they will process all messages even if the queue gets empty before the processing is complete.
To work with this scenario, you will need to:
- Get your cloud ready.
- Create a queue in Message Queue.
- Create an application image.
- Create an instance group.
- Check the instance group scalability.
If you no longer need the resources you created, delete them.
Getting started
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
As an example, we will use a folder named example-folder
.
Required paid resources
The infrastructure cost includes:
- Fee for using VMs and storing an image (see Compute Cloud pricing).
- Fee for public IP addresses and outbound VM traffic (see Virtual Private Cloud pricing).
- Fee for queue requests (see Message Queue pricing).
Prepare your environment
Create auxiliary Yandex Cloud resources
-
Create the
queue-autoscale-sa
service account and assign it theeditor
role forexample-folder
.- In the management console
, selectexample-folder
. - In the list of services, select Identity and Access Management.
- Click Create service account.
- In the Name field, specify
queue-autoscale-sa
. - Click
Add role and select theeditor
role. - Click Create.
- In the management console
-
Create a static access key to enable the service account to work with Message Queue and save the key into the
access_key
file:-
In the list of service accounts, select
queue-autoscale-sa
. -
Click
Create new key and select Create static access key. -
In the Description field, specify
Message Queue
. -
Click Create.
-
Add the key ID and secret key to the
access_key
file in the following format:access_key: key_id: <key_ID> secret: <secret_key>
-
-
Create a cloud network named
queue-autoscale-network
:- In the management console
, selectexample-folder
. - In the list of services, select Virtual Private Cloud.
- Click Create network.
- In the Name field, specify
queue-autoscale-network
. - Disable the Create subnets option.
- Click Create network.
- In the management console
-
Create a subnet named
queue-autoscale-subnet-d
in theru-central1-d
availability zone:-
In the list of networks, select
queue-autoscale-network
. -
Click
Create subnet. -
Specify the subnet parameters:
- Name:
queue-autoscale-subnet-d
- Zone:
ru-central1-d
- CIDR:
192.168.1.0/24
- Name:
-
Click Create subnet.
-
-
Create the
queue-autoscale-sa
service account:yc iam service-account create --name queue-autoscale-sa
Where
--name
is the service account name:queue-autoscale-sa
.Result:
id: aje6brh817ro******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:19:10.153346507Z" name: queue-autoscale-sa
-
Assign the
editor
role forexample-folder
to the service account:yc resource-manager folder add-access-binding example-folder \ --service-account-name queue-autoscale-sa \ --role editor
Where:
--service-account-name
: Service account name,queue-autoscale-sa
.--role
:editor
role.
Result:
done (1s)
-
Create a static access key to enable the service account to work with Message Queue and save the key into the
access_key
file:yc iam access-key create \ --service-account-name queue-autoscale-sa \ --description "Message Queue" \ | tee access_key
Where:
--service-account-name
: Service account name,queue-autoscale-sa
.--description
: Key description.
Result:
access_key: id: ajevb6bx51w3******** service_account_id: aje6brh817ro******** created_at: "2021-10-04T16:25:19.143847764Z" description: Message Queue key_id: g3VybpVKdq_Y******** secret: WVUqnwmC4LSUep0GTKGkbcht9K3Xav7V********
-
Create a cloud network named
queue-autoscale-network
:yc vpc network create queue-autoscale-network
Result:
id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
-
Create a subnet named
queue-autoscale-subnet-d
in theru-central1-d
availability zone:yc vpc subnet create queue-autoscale-subnet-d \ --network-name queue-autoscale-network --range 192.168.1.0/24 --zone ru-central1-d
Where:
--network-name
:queue-autoscale-network
, the one where you are creating the subnet.--range
: Subnet CIDR.--zone
: Subnet availability zone,ru-central1-d
.
Result:
id: e1lnabc23r1c******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:29:12.450858436Z" name: queue-autoscale-subnet-d network_id: enpabce123hd******** zone_id: ru-central1-d v4_cidr_blocks: - 192.168.1.0/24
-
To create the
queue-autoscale-sa
service account, use the ServiceAccountService/Create gRPC API call or the create REST API method. The response will contain the service account ID. -
To assign the service account the
editor
role forexample-folder
, use the FolderService/UpdateAccessBindings gRPC API call or the updateAccessBindings REST API method. In the request body, specify the service account ID. -
Create a static access key to enable the service account to work with Message Queue and save the key into the
access_key
file:-
Use the AccessKeyService/Create gRPC API call or the create REST API method.
-
Add the key ID and secret key to the
access_key
file in the following format:access_key: key_id: <key_ID> secret: <secret_key>
-
-
To create the
queue-autoscale-network
cloud network, use the NetworkService/Create gRPC API call or the create REST API method. The response will contain the network ID. -
To create a subnet named
queue-autoscale-subnet-d
in theru-central1-d
availability zone with the192.168.1.0/24
CIDR, use the SubnetService/Create gRPC API call or the create REST API method. In the request body, specify the network ID.
Set up the AWS CLI
Note
You will need the AWS CLI to complete the final step of this scenario, i.e., to check the instance group scalability. This is why you have to install and configure it.
-
Download and install the AWS CLI as described in the relevant AWS guide
. -
Configure the AWS CLI to work with Yandex Cloud:
-
In your home folder, create a file named
.aws/credentials
with the previously received static access key and its ID:[default] aws_access_key_id = <key_ID> aws_secret_access_key = <secret_key>
-
In your home folder, create the
.aws/config
file; in this file, specify the default region:ru-central1
:[default] region = ru-central1
-
Create a queue in Message Queue
-
Create a queue named
queue-autoscale-queue
:- In the management console
, selectexample-folder
. - In the list of services, select Message Queue.
- Click Create queue.
- In the Name field, specify
queue-autoscale-queue
. - Click Create.
- In the management console
-
Save the queue URL to the
queue
file:-
In the list of queues, select
queue-autoscale-queue
. -
Copy the queue URL.
-
Paste the URL into the
queue
file in the following format:QueueUrl: <queue_URL>
-
-
Create a queue named
queue-autoscale-queue
and save its URL to thequeue
file:aws sqs create-queue \ --endpoint https://message-queue.api.cloud.yandex.net \ --queue-name queue-autoscale-queue \ --output yaml \ | tee queue
Where:
--endpoint
: Message Queue API root URL,https://message-queue.api.cloud.yandex.net
.--queue-name
: Queue name,queue-autoscale-queue
.
Result:
QueueUrl: https://message-queue.api.cloud.yandex.net/b1gvlrnlei4l********/dj6000000003********/queue-autoscale-queue
-
To create the queue named
queue-autoscale-queue
, use the CreateQueue API method. The response will contain the queue URL. -
Paste the queue URL into the
queue
file in the following format:QueueUrl: <queue_URL>
Create an application image
-
If you do not have Packer installed, download and install it by following this tutorial
on the official HashiCorp website. You can also download Packer for your platform here .Note
Yandex Cloud requires Packer version 1.5 or higher.
-
Configure the Yandex Compute Builder plugin
:-
Create the
config.pkr.hcl
file with the following contents:packer { required_plugins { yandex = { version = ">= 1.1.2" source = "github.com/hashicorp/yandex" } } }
-
Install the plugin:
packer init <path_to_config.pkr.hcl>
Result:
Installed plugin github.com/hashicorp/yandex v1.1.2 in ...
-
-
Download the image file archive
(ZIP, 3 KB) and unpack it:wget https://storage.yandexcloud.net/doc-files/queue-autoscale-server.zip unzip queue-autoscale-server.zip
-
In the
server-packer.json
file, change the object in thebuilders
array:-
In the
service_account_id
field, specify thequeue-autoscale-sa
service account ID.How to find the service account ID
Management consoleCLIAPI- In the management console
, selectexample-folder
. - In the list of services, select Identity and Access Management.
- In the left-hand panel, select
Service accounts. - In the list that opens, select the
queue-autoscale-sa
service account. - Copy the service account ID.
Run the following command:
yc iam service-account get queue-autoscale-sa
Result:
id: aje6brh817ro******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:19:10.153346507Z" name: queue-autoscale-sa
Use the list REST API method for the ServiceAccount resource or the ServiceAccountService/List gRPC API call.
- In the management console
-
In the
folder_id
field, specify the folder ID in Yandex Cloud.How to find the folder ID
Management consoleCLIAPIOn the cloud page
, findexample-folder
in the Folders list and copy its ID.Run the following command:
yc resource-manager folder get example-folder
Result:
id: b1g9hv2loamq******** cloud_id: b1g2y61sxwqx******** created_at: "2021-10-04T13:21:32.788067492Z" name: example-folder status: ACTIVE
Use the list REST API method for the Folder resource or the FolderService/List gRPC API call.
-
In the
subnet_id
field, specify thequeue-autoscale-subnet-d
subnet ID.How to find the subnet ID
Management consoleCLIAPI- In the management console
, selectexample-folder
. - In the list of services, select Virtual Private Cloud.
- Navigate to the
Subnets tab. - In the list of subnets, find
queue-autoscale-subnet-d
and copy its ID.
Run the following command:
yc vpc subnet get queue-autoscale-subnet-d
Result:
id: e1lnabc23r1c******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:29:12.450858436Z" name: queue-autoscale-subnet-d network_id: enpabce123hd******** zone_id: ru-central1-d v4_cidr_blocks: - 192.168.1.0/24
Use the list REST API method for the Subnet resource or the SubnetService/List gRPC API call.
- In the management console
-
-
Get an IAM token allowing the service account to work with Compute Cloud and save it to the
key.json
file:CLIAPIyc iam key create \ --service-account-name queue-autoscale-sa \ --description "Compute Cloud" \ --output key.json
Where:
--service-account-name
: Service account name,queue-autoscale-sa
.--description
: Key description.--output
: Path to the key storage file,key.json
.
Result:
id: ajevh3a0hrqf******** service_account_id: aje6brh817ro******** created_at: "2021-10-04T16:35:19.057777570Z" description: Compute Cloud key_algorithm: RSA_2048
Use the create REST API method for the Key resource or the KeyService/Create gRPC API call.
-
To create an image, run the following command:
packer build server-packer.json
Result:
... Build 'builder' finished after 2 minutes 48 seconds. ==> Wait completed after 2 minutes 48 seconds ==> Builds finished. The artifacts of successful builds are: --> builder: A disk image was created: queue-autoscale-image-v1633354734 (id: fd8hlbuhjc4m********) with family name queue-autoscale-image
Create an instance group
-
In the management console
, selectexample-folder
. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Instance groups. -
Click Create group of virtual machines.
-
Under Basic parameters:
- In the Name field, specify
queue-autoscale-ig
. - Select Service account
queue-autoscale-sa
.
- In the Name field, specify
-
Under Allocation, select
ru-central1-d
in the Availability zone field. -
Under Instance template, click Define and do the following in the window that opens:
-
Under Boot disk image, navigate to the Custom tab and click Select.
-
In the window that opens, select the previously created image as a boot disk:
- Navigate to the Image tab.
- Select an image with a name that begins with
queue-autoscale-image
. - Click Apply.
-
Under Network settings:
- In the Network field, select
queue-autoscale-network
. - In the Public address field, select
Auto
so that your VMs have the internet access and can receive messages from the queue.
- In the Network field, select
-
Under Access:
- In the Service account field, select
queue-autoscale-sa
. - Enter your username and the contents of the public key to connect to the VM.
- In the Service account field, select
-
Click Save.
-
-
Under Changes during creation and updates, specify the following values:
- Stop simultaneously:
1
- Stop VMs by strategy:
Opportunistic
- Stop simultaneously:
-
Under Scaling, specify the following values:
- Type:
Automatic
- Autoscaling type:
Regional
- Minimum number of instances per zone:
0
- Maximum group size:
5
- Utilization measurement period:
60 seconds
- Instance warmup period:
0 minutes
- Stabilization period:
5 minutes
- Initial group size:
1
- Type:
-
Under Metrics, specify the following values:
- Metric:
Yandex Monitoring
- Folder ID:
message-queue
- Metric name:
queue.messages.stored_count
- Labels:
queue
as the key andqueue-autoscale-queue
as the value - Metric type:
GAUGE
- Rule type:
WORKLOAD
- Target:
5
- Metric:
-
Click Create.
-
Get the ID of the image you created in the previous steps:
yc compute image get-latest-from-family queue-autoscale-image
Result:
id: fd8hlbuhjc4m******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:38:54.345168059Z" name: queue-autoscale-image-v1633354734 description: Image for autoscaling an instance group based on the size of a queue family: queue-autoscale-image storage_size: "3858759680" min_disk_size: "5368709120" product_ids: - f2e6fnj3erf1******** status: READY os: type: LINUX
-
Edit the instance group specification file:
spec.yaml
:-
In the
folder_id
field, specify the folder ID in Yandex Cloud. -
In the
image_id
field, specify the image ID. -
In the
network_id
field, specify thequeue-autoscale-network
ID, and in thesubnet_ids
field, thequeue-autoscale-subnet-d
ID.How to find the network ID
CLIManagement consoleAPIRun this command:
yc vpc network get queue-autoscale-network
Result:
id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
- In the management console
, selectexample-folder
. - In the list of services, select Virtual Private Cloud.
- In the list of networks, find
queue-autoscale-network
and copy its ID.
Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the management console
-
In the
service_account_id
field, specify thequeue-autoscale-sa
service account ID.
To learn how to get the required IDs (except for the image ID and network ID), see this section: Create an application image.
-
-
Create the
queue-autoscale-ig
instance group using the following command:yc compute instance-group create --file spec.yaml
Where
--file
is the path to the instance group specification file:spec.yaml
.
-
To get the ID of the previously created image from the
queue-autoscale-image
family, use the ImageService/GetLatestByFamily gRPC API call or the getLatestByFamily REST API method. -
Edit the
spec.yaml
instance group specification file:-
In the
folder_id
field, specify the folder ID in Yandex Cloud. -
In the
image_id
field, specify the image ID. -
In the
network_id
field, specify thequeue-autoscale-network
ID, and in thesubnet_ids
field, thequeue-autoscale-subnet-d
ID.How to find the network ID
Management consoleCLIAPI- In the management console
, selectexample-folder
. - In the list of services, select Virtual Private Cloud.
- In the list of networks, find
queue-autoscale-network
and copy its ID.
Run the following command:
yc vpc network get queue-autoscale-network
Result:
id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the management console
-
In the
service_account_id
field, specify thequeue-autoscale-sa
service account ID.
To learn how to get the required IDs (except for the image ID and network ID), see this section: Create an application image.
-
-
To create the
queue-autoscale-ig
instance group to thespec.yaml
specification, use the InstanceGroupService/CreateFromYaml gRPC API call or the createFromYaml REST API method.
Check the instance group scalability
-
Edit the
messages.sh
file: in the--queue-url
parameter, specify the URL of the queue from thequeue
file. -
Run
messages.sh
. It will start sending messages toqueue-autoscale-queue
one by one; overall, it will send 100 messages. -
Open dashboards in Yandex Monitoring:
Management console-
Open the instance group information dashboard:
- In Monitoring
, select the Compute — Instance Groups dashboard. - In the Instance Group ID field, select
queue-autoscale-ig
.
- In Monitoring
-
Open the queue information dashboard:
- In Monitoring
, select the Message Queue dashboard. - In the Queue ID field, select
queue-autoscale-queue
.
- In Monitoring
-
-
Check the dashboard data to make sure the following events took place in this order:
- Number of enqueued messages in
queue-autoscale-queue
reached 100 (the Message Queue dashboard, the Sent messages, count and Messages in queue, count charts). - VM instances from the
queue-autoscale-ig
group started receiving messages (the Received messages, count chart) and deleting them from the queue (the Deleted messages, count and Messages in queue, count charts). - While processing the messages, the number of instances in the group increased from one to five and, after all messages were processed, the group became empty (the Compute — Instance Groups dashboard, the Number of instances in zone B chart).
- Number of enqueued messages in
Delete the resources you created
To shut down the infrastructure and stop paying for the resources you created: