Configuring autoscaling
Managed Service for Kubernetes has three autoscaling methods available:
Getting started
-
Create a Managed Service for Kubernetes cluster with any suitable configuration.
-
Install kubect
and configure it to work with the new cluster.
Configuring cluster autoscaling
Warning
You can only enable automatic scaling of this type when creating a Managed Service for Kubernetes node group.
To create an autoscalable Managed Service for Kubernetes node group:
Create a Managed Service for Kubernetes node group with the following parameters:
- Scaling Type:
Automatic
. - Minimum number of nodes: Specify the number of Managed Service for Kubernetes nodes to remain in the group at minimum load.
- Maximum number of nodes: Specify the maximum number of Managed Service for Kubernetes nodes allowed in the group.
- Initial number of nodes: Number of Managed Service for Kubernetes nodes to be created together with the group (this number must be between the minimum and the maximum number of nodes in the group).
If you do not have the Yandex Cloud CLI yet, install and initialize it.
The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID>
command. You can specify a different folder using the --folder-name
or --folder-id
parameter.
-
Review the command to create a Managed Service for Kubernetes node group:
yc managed-kubernetes node-group create --help
-
Create an autoscalable Managed Service for Kubernetes node group:
yc managed-kubernetes node-group create \ ... --auto-scale min=<minimum_number_of_nodes>, max=<maximum_number_of_nodes>, initial=<initial_number_of_nodes>
-
With Terraform
, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.Terraform is distributed under the Business Source License
. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.For more information about the provider resources, see the documentation on the Terraform
website or mirror website.If you do not have Terraform yet, install it and configure its Yandex Cloud provider.
-
Open the current Terraform configuration file describing the node group.
For more information about creating this file, see Creating a node group.
-
Add a description of the new node group by specifying the autoscaling settings under
scale_policy.auto_scale
:resource "yandex_kubernetes_node_group" "<node_group_name>" { ... scale_policy { auto_scale { min = <minimum_number_of_nodes_per_group> max = <maximum_number_of_nodes_per_group> initial = <initial_number_of_nodes_per_group> } } }
-
Make sure the configuration files are correct.
-
In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
-
Run this command:
terraform validate
Terraform will show any errors found in your configuration files.
-
-
Confirm updating the resources.
-
Run this command to view the planned changes:
terraform plan
If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
-
If everything looks correct, apply the changes:
-
Run this command:
terraform apply
-
Confirm updating the resources.
-
Wait for the operation to complete.
-
-
Cluster Autoscaler is managed on the Managed Service for Kubernetes side.
For more information about Cluster Autoscaler, see Cluster autoscaling. The default parameters are described in the Kubernetes documentation
See also Questions and answers about node group autoscaling in Managed Service for Kubernetes.
Configuring horizontal pod autoscaling
-
Create a Horizontal Pod Autoscaler for your application, for example:
kubectl autoscale deployment/<application_name> --cpu-percent=50 --min=1 --max=3
Where:
--cpu-percent
: Expected Managed Service for Kubernetes pod load on the vCPU.--min
: Minimum number of Managed Service for Kubernetes pods.--max
: Maximum number of Managed Service for Kubernetes pods.
-
Check the Horizontal Pod Autoscaler status:
kubectl describe hpa/<application_name>
For more information about Horizontal Pod Autoscaler, see Horizontal pod autoscaling.
Configuring vertical pod autoscaling
-
Install Vertical Pod Autoscaler from the following repository
:cd /tmp && \ git clone https://github.com/kubernetes/autoscaler.git && \ cd autoscaler/vertical-pod-autoscaler/hack && \ ./vpa-up.sh
-
Create a configuration file called
vpa.yaml
for your application:apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: <application_name> spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: <application_name> updatePolicy: updateMode: "<VPA_runtime_mode>"
Where
updateMode
is the Vertical Pod Autoscaler runtime mode,Auto
orOff
. -
Create a Vertical Pod Autoscaler for your application:
kubectl apply -f vpa.yaml
-
Check the Vertical Pod Autoscaler status:
kubectl describe vpa <application_name>
For more information about Vertical Pod Autoscaler, see Vertical pod autoscaling.
Deleting Terminated pods
Sometimes during autoscaling, Managed Service for Kubernetes node pods are not deleted and stay in the Terminated status. This happens because the Pod garbage collector (PodGC)
You can delete the stuck Managed Service for Kubernetes pods:
Manually
Run this command:
kubectl get pods --all-namespaces | grep -i Terminated \
| awk '{print $1, $2}' | xargs -n2 kubectl delete pod -n
Automatically using CronJob
To delete stuck Managed Service for Kubernetes pods automatically:
If you no longer need the CronJob, delete it.
Setting up automatic deletion in a CronJob
-
Create a
cronjob.yaml
file with a specification for the CronJob and the resources needed to run it:--- apiVersion: batch/v1 kind: CronJob metadata: name: terminated-pod-cleaner spec: schedule: "*/5 * * * *" jobTemplate: spec: template: spec: serviceAccountName: terminated-pod-cleaner containers: - name: terminated-pod-cleaner image: bitnami/kubectl imagePullPolicy: IfNotPresent command: ["/bin/sh", "-c"] args: ["kubectl get pods --all-namespaces | grep -i Terminated | awk '{print $1, $2}' | xargs --no-run-if-empty -n2 kubectl delete pod -n"] restartPolicy: Never --- apiVersion: v1 kind: ServiceAccount metadata: name: terminated-pod-cleaner --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: terminated-pod-cleaner rules: - apiGroups: [""] resources: - pods verbs: [list, delete] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: terminated-pod-cleaner subjects: - kind: ServiceAccount name: terminated-pod-cleaner namespace: default roleRef: kind: ClusterRole name: terminated-pod-cleaner apiGroup: rbac.authorization.k8s.io
The
schedule: "*/5 * * * *"
line defines a schedule in cron format: the job runs every 5 minutes. Change the interval if needed. -
Create a CronJob and its resources:
kubectl create -f cronjob.yaml
Result:
cronjob.batch/terminated-pod-cleaner created serviceaccount/terminated-pod-cleaner created clusterrole.rbac.authorization.k8s.io/terminated-pod-cleaner created clusterrolebinding.rbac.authorization.k8s.io/terminated-pod-cleaner created
-
Check that the CronJob has been created:
kubectl get cronjob terminated-pod-cleaner
Result:
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE terminated-pod-cleaner */5 * * * * False 0 <none> 4s
After the interval specified in the
SCHEDULE
, a time value will appear in theLAST SCHEDULE
column. This means that the task was successfully executed or ended with an error.
Checking the results of CronJob jobs
-
Retrieve a list of jobs:
kubectl get jobs
Result:
NAME COMPLETIONS DURATION AGE <job_name> 1/1 4s 2m1s ...
-
Get the name of the Managed Service for Kubernetes pod that ran the job:
kubectl get pods --selector=job-name=<job_name> --output=jsonpath={.items[*].metadata.name}
-
View the Managed Service for Kubernetes pod logs:
kubectl logs <pod_name>
The log will include a list of deleted Managed Service for Kubernetes pods. If the log is empty, this means that there were no Managed Service for Kubernetes pods in the Terminated status when the job ran.
Deleting the CronJob
To delete the CronJob and its resources, run this command:
kubectl delete cronjob terminated-pod-cleaner && \
kubectl delete serviceaccount terminated-pod-cleaner && \
kubectl delete clusterrole terminated-pod-cleaner && \
kubectl delete clusterrolebinding terminated-pod-cleaner