Autoscaling an instance group to process messages from a queue in Yandex Message Queue
This is an integration scenario for Yandex Compute Cloud, Yandex Message Queue, and Yandex Monitoring.
In this scenario, we will create an instance group. All instances in the group process messages from the same Message Queue queue (for example, they receive messages and delete them after a while). The number of enqueued messages is registered in a Monitoring metric, and the instance group is autoscaled based on this metric.
The opportunistic shutdown strategy has been chosen for scaling: the instances stop on their own once they process all the messages and are then deleted from the group as the group size is reduced. This ensures that the instances will not be forcibly deleted and will process the last messages even if the queue empties before message processing is completed.
To perform this use case:
- Prepare the environment.
- Create a queue in Message Queue.
- Create an image with an application.
- Create an instance group.
- Check the instance group scalability.
If you no longer need the resources you created, delete them.
Getting started
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
We will use a folder named example-folder
as an example.
Required paid resources
The infrastructure cost includes:
- Fee for using VMs and storing an image (see Compute Cloud pricing).
- Fee for using public IP addresses and outgoing VM traffic (see Virtual Private Cloud pricing).
- Fee for queue requests (see Message Queue pricing).
Prepare the environment
Create the supporting Yandex Cloud resources
-
Create a service account named
queue-autoscale-sa
and assign it theeditor
role forexample-folder
.- In the management console
, selectexample-folder
. - At the top of the screen, go to the Service accounts tab.
- Click Create service account.
- In the Name field, specify
queue-autoscale-sa
. - Click
Add role and select theeditor
role. - Click Create.
- In the management console
-
Create a static access key to enable the service account to work with Message Queue and save the key to the
access_key
file:-
In the list of service accounts, select
queue-autoscale-sa
. -
Click
Create new key and select Create static access key. -
In the Description field, specify
Message Queue
. -
Click Create.
-
Add the key ID and secret key to the
access_key
file in the following format:access_key: key_id: <key_ID> secret: <secret_key>
-
-
Create a cloud network named
queue-autoscale-network
:- In the management console
, selectexample-folder
. - In the list of services, select Virtual Private Cloud.
- Click Create network.
- In the Name field, specify
queue-autoscale-network
. - Disable the Create subnets option.
- Click Create network.
- In the management console
-
Create a subnet named
queue-autoscale-subnet-a
in theru-central1-a
availability zone:-
In the list of networks, select
queue-autoscale-network
. -
Click
Create subnet. -
Specify the subnet parameters:
- Name:
queue-autoscale-subnet-a
- Zone:
ru-central1-a
- CIDR:
192.168.1.0/24
- Name:
-
Click Create subnet.
-
-
Create a service account named
queue-autoscale-sa
:yc iam service-account create --name queue-autoscale-sa
Where
--name
is the service account name:queue-autoscale-sa
.Result:
id: aje6brh817ro******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:19:10.153346507Z" name: queue-autoscale-sa
-
Assign the
editor
role forexample-folder
to the service account:yc resource-manager folder add-access-binding example-folder \ --service-account-name queue-autoscale-sa \ --role editor
Where:
--service-account-name
: Name of the service account created at step 1,queue-autoscale-sa
.--role
: Role to assign, which iseditor
.
Result:
done (1s)
-
Create a static access key to enable the service account to work with Message Queue and save the key to the
access_key
file:yc iam access-key create \ --service-account-name queue-autoscale-sa \ --description "Message Queue" \ | tee access_key
Where:
--service-account-name
: Name of the service account created at step 1,queue-autoscale-sa
.--description
: Key description.
Result:
access_key: id: ajevb6bx51w3******** service_account_id: aje6brh817ro******** created_at: "2021-10-04T16:25:19.143847764Z" description: Message Queue key_id: g3VybpVKdq_Y******** secret: WVUqnwmC4LSUep0GTKGkbcht9K3Xav7V********
-
Create a cloud network named
queue-autoscale-network
:yc vpc network create queue-autoscale-network
Result:
id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
-
Create a subnet named
queue-autoscale-subnet-a
in theru-central1-a
availability zone:yc vpc subnet create queue-autoscale-subnet-a \ --network-name queue-autoscale-network --range 192.168.1.0/24 --zone ru-central1-a
Where:
--network-name
: Name of the network to create the subnet in,queue-autoscale-network
.--range
: Subnet CIDR.--zone
: Availability zone to create the subnet in,ru-central1-a
.
Result:
id: e1lnabc23r1c******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:29:12.450858436Z" name: queue-autoscale-subnet-a network_id: enpabce123hd******** zone_id: ru-central1-a v4_cidr_blocks: - 192.168.1.0/24
-
Create a service account named
queue-autoscale-sa
using the ServiceAccountService/Create gRPC API call or the create REST API method. The response will contain the service account ID. -
Assign the service account the
editor
role forexample-folder
using the FolderService/UpdateAccessBindings gRPC API call or the updateAccessBindings REST API method. In the request body, specify the service account ID. -
Create a static access key to enable the service account to work with Message Queue and save the key to the
access_key
file:-
Use the AccessKeyService/Create gRPC API call or the create REST API method.
-
Add the key ID and secret key to the
access_key
file in the following format:access_key: key_id: <key_ID> secret: <secret_key>
-
-
Create a cloud network named
queue-autoscale-network
using the NetworkService/Create gRPC API call or the create REST API method. The response will contain the network ID. -
Create a subnet named
queue-autoscale-subnet-a
in theru-central1-a
availability zone with the192.168.1.0/24
CIDR using the SubnetService/Create gRPC API call or the create REST API method. In the request body, specify the network ID.
Set up the AWS CLI
Note
You will use the AWS CLI to perform the final step of the script: checking instance group scalability. This is why you must install and configure it, even if you aren't going to use it for any other steps.
-
Download and install the AWS CLI as described in its documentation
. -
Configure the AWS CLI to work with Yandex Cloud:
-
In your home folder, create a file named
.aws/credentials
with the previously received static access key and its ID:[default] aws_access_key_id = <key_ID> aws_secret_access_key = <secret_key>
-
In your home folder, create a file named
.aws/config
with the default region set toru-central1
:[default] region = ru-central1
-
Create a queue in Message Queue
-
Create a queue named
queue-autoscale-queue
:- In the management console
, selectexample-folder
. - In the list of services, select Message Queue.
- Click Create queue.
- In the Name field, specify
queue-autoscale-queue
. - Click Create.
- In the management console
-
Save the queue URL to the
queue
file:-
In the list of queues, select
queue-autoscale-queue
. -
Copy the queue URL.
-
Paste the URL into the
queue
file in the following format:QueueUrl: <queue_URL>
-
-
Create a queue named
queue-autoscale-queue
and save its URL to thequeue
file:aws sqs create-queue \ --endpoint https://message-queue.api.cloud.yandex.net \ --queue-name queue-autoscale-queue \ --output yaml \ | tee queue
Where:
--endpoint
: Root URL of the Message Queue API, which ishttps://message-queue.api.cloud.yandex.net
.--queue-name
: Queue name,queue-autoscale-queue
.
Result:
QueueUrl: https://message-queue.api.cloud.yandex.net/b1gvlrnlei4l********/dj6000000003********/queue-autoscale-queue
-
Create a queue named
queue-autoscale-queue
using the CreateQueue API method. The response will contain the queue URL. -
Paste the queue URL into the
queue
file in the following format:QueueUrl: <queue_URL>
Create an image with an application
-
If you do not have Packer installed yet, download and install it by following the instructions
on the official website. You can also download a Packer distribution for your platform from a mirror .Note
Yandex Cloud requires Packer 1.5 or higher.
-
Configure the Yandex Compute Builder plugin
:-
Create a file named
config.pkr.hcl
with the following contents:packer { required_plugins { yandex = { version = ">= 1.1.2" source = "github.com/hashicorp/yandex" } } }
-
Install the plugin:
packer init <path_to_config.pkr.hcl>
Result:
Installed plugin github.com/hashicorp/yandex v1.1.2 in ...
-
-
Download the archive with files for the image
(ZIP, 3 KB) and unpack it:wget https://storage.yandexcloud.net/doc-files/queue-autoscale-server.zip unzip queue-autoscale-server.zip
-
In the
server-packer.json
file, change the object in thebuilders
array:-
In the
service_account_id
field, enter thequeue-autoscale-sa
service account ID.How do I find out the service account ID
Management consoleCLIAPI- In the management console
, select theexample-folder
folder. - At the top of the screen, go to the Service accounts tab.
- In the list of service accounts, select
queue-autoscale-sa
. - Copy the service account ID.
Run the following command:
yc iam service-account get queue-autoscale-sa
Result:
id: aje6brh817ro******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:19:10.153346507Z" name: queue-autoscale-sa
Use the list REST API method for the ServiceAccount resource or the ServiceAccountService/List gRPC API call.
- In the management console
-
In the
folder_id
field, specify the ID of the folder in Yandex Cloud.How do I find out the folder ID
Management consoleCLIAPIOn the cloud page
, findexample-folder
in the Folders list and copy its ID.Run the following command:
yc resource-manager folder get example-folder
Result:
id: b1g9hv2loamq******** cloud_id: b1g2y61sxwqx******** created_at: "2021-10-04T13:21:32.788067492Z" name: example-folder status: ACTIVE
Use the list REST API method for the Folder resource or the FolderService/List gRPC API call.
-
In the
subnet_id
field, specify thequeue-autoscale-subnet-a
subnet ID.How do I find out the subnet ID
Management consoleCLIAPI- In the management console
, select theexample-folder
folder. - In the list of services, select Virtual Private Cloud.
- Go to the
Subnets tab. - In the list of subnets, find
queue-autoscale-subnet-a
and copy its ID.
Run the following command:
yc vpc subnet get queue-autoscale-subnet-a
Result:
id: e1lnabc23r1c******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:29:12.450858436Z" name: queue-autoscale-subnet-a network_id: enpabce123hd******** zone_id: ru-central1-a v4_cidr_blocks: - 192.168.1.0/24
Use the list REST API method for the Subnet resource or the SubnetService/List gRPC API call.
- In the management console
-
-
Get an IAM token to enable the service account to work with Compute Cloud and save the token to the
key.json
file:CLIAPIyc iam key create \ --service-account-name queue-autoscale-sa \ --description "Compute Cloud" \ --output key.json
Where:
--service-account-name
: Name of the previously created service account,queue-autoscale-sa
.--description
: Key description.--output
: Path to the file to save the key to, which iskey.json
.
Result:
id: ajevh3a0hrqf******** service_account_id: aje6brh817ro******** created_at: "2021-10-04T16:35:19.057777570Z" description: Compute Cloud key_algorithm: RSA_2048
Use the create REST API method for the Key resource or the KeyService/Create gRPC API call.
-
Run the following command to create an image:
packer build server-packer.json
Result:
... Build 'builder' finished after 2 minutes 48 seconds. ==> Wait completed after 2 minutes 48 seconds ==> Builds finished. The artifacts of successful builds are: --> builder: A disk image was created: queue-autoscale-image-v1633354734 (id: fd8hlbuhjc4m********) with family name queue-autoscale-image
Create an instance group
-
In the management console
, selectexample-folder
. -
In the list of services, select Compute Cloud.
-
In the left-hand panel, select
Instance groups. -
Click Create group of virtual machines.
-
Under Basic parameters:
- In the Name field, specify
queue-autoscale-ig
. - Select the Service account :
queue-autoscale-sa
.
- In the Name field, specify
-
Under Allocation, select
ru-central1-a
in the Availability zone field. -
Under Instance template, click Define and do the following in the window that opens:
-
Under Boot disk image, go to the Custom tab and click Select.
-
In the window that opens, select the created image as a boot disk:
- Go to the Image tab.
- Select an image whose name begins with
queue-autoscale-image
. - Click Apply.
-
Under Network settings:
- In the Network field, select
queue-autoscale-network
. - In the Public address field, select
Auto
so that your VMs have internet access and can receive messages from the queue.
- In the Network field, select
-
Under Access:
- In the Service account field, select
queue-autoscale-sa
. - Enter your username and the contents of the public key to connect to the VM.
- In the Service account field, select
-
Click Save.
-
-
Under Allow when creating and updating, specify the following values:
- Stop simultaneously:
1
- Stop VMs by strategy:
Opportunistic
- Stop simultaneously:
-
Under Scaling, specify the following values:
- Type:
Automatic
- Autoscaling type:
Regional
- Minimum number of instances per zone:
0
- Maximum group size:
5
- Utilization measurement period:
60 seconds
- Instance warmup period:
0 minutes
- Stabilization period:
5 minutes
- Initial group size:
1
- Type:
-
Under Metrics, specify the following values:
- Metric:
Yandex Monitoring
- Folder ID:
message-queue
- Metric name:
queue.messages.stored_count
- Labels:
queue
as the key andqueue-autoscale-queue
as the value - Metric type:
GAUGE
- Rule type:
WORKLOAD
- Target:
5
- Metric:
-
Click Create.
-
Get the ID of the image you created at the previous steps:
yc compute image get-latest-from-family queue-autoscale-image
Result:
id: fd8hlbuhjc4m******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:38:54.345168059Z" name: queue-autoscale-image-v1633354734 description: Image for autoscaling an instance group based on the size of a queue family: queue-autoscale-image storage_size: "3858759680" min_disk_size: "5368709120" product_ids: - f2e6fnj3erf1******** status: READY os: type: LINUX
-
Edit the
spec.yaml
instance group specification file:-
In the
folder_id
field, specify the ID of the folder in Yandex Cloud. -
In the
image_id
field, specify the ID of the created image. -
In the
network_id
field, specify thequeue-autoscale-network
ID, and in thesubnet_ids
field, thequeue-autoscale-subnet-a
ID.How do I find out the network ID
CLIManagement consoleAPIRun this command:
yc vpc network get queue-autoscale-network
Result:
id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
- In the management console
, select theexample-folder
folder. - In the list of services, select Virtual Private Cloud.
- In the list of networks, find
queue-autoscale-network
and copy its ID.
Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the management console
-
In the
service_account_id
field, enter thequeue-autoscale-sa
service account ID.
To learn how to get the IDs of resources (except for images and networks), see step 3 in the section Create an image with an application.
-
-
Create an instance group named
queue-autoscale-ig
based on the following specification:yc compute instance-group create --file spec.yaml
Where
--file
is the path to the instance group specification file:spec.yaml
.
-
Get the ID of the previously created image from the
queue-autoscale-image
family. To do this, use the ImageService/GetLatestByFamily gRPC API call or the getLatestByFamily REST API method. -
Edit the
spec.yaml
instance group specification file:-
In the
folder_id
field, specify the ID of the folder in Yandex Cloud. -
In the
image_id
field, specify the ID of the created image. -
In the
network_id
field, specify thequeue-autoscale-network
ID, and in thesubnet_ids
field, thequeue-autoscale-subnet-a
ID.How do I find out the network ID
Management consoleCLIAPI- In the management console
, select theexample-folder
folder. - In the list of services, select Virtual Private Cloud.
- In the list of networks, find
queue-autoscale-network
and copy its ID.
Run the following command:
yc vpc network get queue-autoscale-network
Result:
id: enpabce123hd******** folder_id: b0g12ga82bcv******** created_at: "2021-10-04T16:28:15.905337847Z" name: queue-autoscale-network
Use the list REST API method for the Network resource or the NetworkService/List gRPC API call.
- In the management console
-
In the
service_account_id
field, enter thequeue-autoscale-sa
service account ID.
To learn how to get the IDs of resources (except for images and networks), see step 3 in the section Create an image with an application.
-
-
Create an instance group named
queue-autoscale-ig
based on thespec.yaml
specification. To do this, use the InstanceGroupService/CreateFromYaml gRPC API call or the createFromYaml REST API method.
Check the instance group scalability
-
Edit the
messages.sh
file: in the--queue-url
parameter, specify the URL of the queue from thequeue
file. -
Run the
messages.sh
file to start sending 100 messages toqueue-autoscale-queue
one by one. -
Open dashboards in Yandex Monitoring:
Management console-
Open the dashboard with information about the instance group:
- In Monitoring
, select the Compute — Instance Groups service dashboard. - In the Instance Group ID field, select
queue-autoscale-ig
.
- In Monitoring
-
Open the dashboard with queue information:
- In Monitoring
, select the Message Queue service dashboard. - In the Queue ID field, select
queue-autoscale-queue
.
- In Monitoring
-
-
Check the dashboard data to make sure the following events took place in this order:
- Number of enqueued messages in
queue-autoscale-queue
reached 100 (the Message Queue dashboard or the Sent messages, count and Messages in queue, count charts). - Instances from the
queue-autoscale-ig
group started receiving messages (the Received messages, count chart) and deleting them from the queue (the Deleted messages, count and Messages in queue, count charts). - While processing the messages, the number of instances in the group increased from one to five and, after all messages were processed, the group became empty (the Compute — Instance Groups dashboard, the Number of instances in zone B chart).
- Number of enqueued messages in
Delete the resources you created
To shut down the infrastructure and stop paying for the resources you created: