Configuring autoscaling
Managed Service for Kubernetes has three autoscaling methods available:
Getting started
-
Create a Managed Service for Kubernetes cluster with any suitable configuration.
-
Install kubectl
and configure it to work with the created cluster.
Configuring cluster autoscaling
Warning
You can only enable automatic scaling of this type when creating a Managed Service for Kubernetes node group.
To create an autoscalable Managed Service for Kubernetes node group:
Create a Managed Service for Kubernetes node group with the following parameters:
- Scaling Type:
Automatic
. - Minimum number of nodes: Specify the number of Managed Service for Kubernetes nodes to remain in the group at minimum load.
- Maximum number of nodes: Specify the maximum number of Managed Service for Kubernetes nodes allowed in the group.
- Initial number of nodes: Number of Managed Service for Kubernetes nodes to be created together with the group (this number must be between the minimum and the maximum number of nodes in the group).
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
-
Review the command to create a Managed Service for Kubernetes node group:
yc managed-kubernetes node-group create --help
-
Create an autoscalable Managed Service for Kubernetes node group:
yc managed-kubernetes node-group create \ ... --auto-scale min=<minimum_number_of_nodes>, max=<maximum_number_of_nodes>, initial=<initial_number_of_nodes>
-
Terraform
allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License .For more information about the provider resources, see the documentation on the Terraform
website or the mirror .If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.
If you don't have Terraform, install it and configure the Yandex Cloud provider.
-
Open the current Terraform configuration file describing the node group.
For more information about creating this file, see Creating a node group.
-
Add a description of the new node group by specifying the autoscaling settings under
scale_policy.auto_scale
:resource "yandex_kubernetes_node_group" "<node_group_name>" { ... scale_policy { auto_scale { min = <minimum_number_of_nodes_per_group> max = <maximum_number_of_nodes_per_group> initial = <initial_number_of_nodes_per_group> } } }
-
Make sure the configuration files are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
Cluster Autoscaler is managed on the Managed Service for Kubernetes side.
For more information about Cluster Autoscaler, see Cluster autoscaling. The default parameters are described in the Kubernetes documentation
See also Questions and answers about node group autoscaling in Managed Service for Kubernetes.
Configuring horizontal pod autoscaling
-
Create a Horizontal Pod Autoscaler for your application, for example:
kubectl autoscale deployment/<application_name> --cpu-percent=50 --min=1 --max=3
Where:
--cpu-percent
: Expected Managed Service for Kubernetes pod load on the vCPU.--min
: Minimum number of Managed Service for Kubernetes pods.--max
: Maximum number of Managed Service for Kubernetes pods.
-
Check the Horizontal Pod Autoscaler status:
kubectl describe hpa/<application_name>
For more information about Horizontal Pod Autoscaler, see Horizontal pod autoscaling.
Configuring vertical pod autoscaling
-
Install Vertical Pod Autoscaler from the following repository
:cd /tmp && \ git clone https://github.com/kubernetes/autoscaler.git && \ cd autoscaler/vertical-pod-autoscaler/hack && \ ./vpa-up.sh
-
Create a configuration file called
vpa.yaml
for your application:apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: <application_name> spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: <application_name> updatePolicy: updateMode: "<VPA_runtime_mode>"
Where
updateMode
is the Vertical Pod Autoscaler runtime mode,Auto
orOff
. -
Create a Vertical Pod Autoscaler for your application:
kubectl apply -f vpa.yaml
-
Check the Vertical Pod Autoscaler status:
kubectl describe vpa <application_name>
For more information about Vertical Pod Autoscaler, see Vertical pod autoscaling.
Deleting Terminated pods
Sometimes during autoscaling, Managed Service for Kubernetes node pods are not deleted and stay in the Terminated status. This happens because the Pod garbage collector (PodGC)
You can delete the stuck Managed Service for Kubernetes pods:
Manually
Run this command:
kubectl get pods --all-namespaces | grep -i Terminated \
| awk '{print $1, $2}' | xargs -n2 kubectl delete pod -n
Automatically using CronJob
To delete stuck Managed Service for Kubernetes pods automatically:
If you no longer need the CronJob, delete it.
Setting up automatic deletion in a CronJob
-
Create a
cronjob.yaml
file with a specification for the CronJob and the resources needed to run it:--- apiVersion: batch/v1 kind: CronJob metadata: name: terminated-pod-cleaner spec: schedule: "*/5 * * * *" jobTemplate: spec: template: spec: serviceAccountName: terminated-pod-cleaner containers: - name: terminated-pod-cleaner image: bitnami/kubectl imagePullPolicy: IfNotPresent command: ["/bin/sh", "-c"] args: ["kubectl get pods --all-namespaces | grep -i Terminated | awk '{print $1, $2}' | xargs --no-run-if-empty -n2 kubectl delete pod -n"] restartPolicy: Never --- apiVersion: v1 kind: ServiceAccount metadata: name: terminated-pod-cleaner --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: terminated-pod-cleaner rules: - apiGroups: [""] resources: - pods verbs: [list, delete] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: terminated-pod-cleaner subjects: - kind: ServiceAccount name: terminated-pod-cleaner namespace: default roleRef: kind: ClusterRole name: terminated-pod-cleaner apiGroup: rbac.authorization.k8s.io
The
schedule: "*/5 * * * *"
line defines a schedule in cron format: the job runs every 5 minutes. Change the interval if needed. -
Create a CronJob and its resources:
kubectl create -f cronjob.yaml
Result:
cronjob.batch/terminated-pod-cleaner created serviceaccount/terminated-pod-cleaner created clusterrole.rbac.authorization.k8s.io/terminated-pod-cleaner created clusterrolebinding.rbac.authorization.k8s.io/terminated-pod-cleaner created
-
Check that the CronJob has been created:
kubectl get cronjob terminated-pod-cleaner
Result:
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE terminated-pod-cleaner */5 * * * * False 0 <none> 4s
After the interval specified in the
SCHEDULE
, a time value will appear in theLAST SCHEDULE
column. This means that the task was successfully executed or ended with an error.
Checking the results of CronJob jobs
-
Retrieve a list of jobs:
kubectl get jobs
Result:
NAME COMPLETIONS DURATION AGE <job_name> 1/1 4s 2m1s ...
-
Get the name of the Managed Service for Kubernetes pod that ran the job:
kubectl get pods --selector=job-name=<job_name> --output=jsonpath={.items[*].metadata.name}
-
View the Managed Service for Kubernetes pod logs:
kubectl logs <pod_name>
The log will include a list of deleted Managed Service for Kubernetes pods. If the log is empty, this means that there were no Managed Service for Kubernetes pods in the Terminated status when the job ran.
Deleting the CronJob
To delete the CronJob and its resources, run this command:
kubectl delete cronjob terminated-pod-cleaner && \
kubectl delete serviceaccount terminated-pod-cleaner && \
kubectl delete clusterrole terminated-pod-cleaner && \
kubectl delete clusterrolebinding terminated-pod-cleaner