Troubleshooting in Managed Service for Kubernetes

• Error creating a cluster in a different folder's cloud network

• Namespace fails to delete and remains Terminating

• I am using Yandex Network Load Balancer together with an ingress controller. Why are some of my cluster's nodes UNHEALTHY?

• Why does the newly created PersistentVolumeClaim remain Pending?

• Why does my Managed Service for Kubernetes cluster fail to start after I update its node configuration?

• Error updating ingress controller certificate

• Why is DNS resolution not working in my cluster?

• Creating a node group with the CLI results in a parameter conflict. How do I fix it?

• Error connecting to a cluster using kubectl

• Errors connecting to a node over SSH

• How do I provide internet access to my Managed Service for Kubernetes cluster nodes?

• Why cannot I choose Docker as the container runtime?

• Error connecting a GitLab repository to Argo CD

• Traffic loss when deploying app updates in a cluster with Yandex Application Load Balancer

• System time displayed incorrectly in the Linux console, as well as in container and Managed Service for Kubernetes cluster pod logs

• What should I do if I deleted my Yandex Network Load Balancer or its target groups that were automatically created for a LoadBalancer service?

• Error connecting a Yandex Compute Cloud VM instance as an external Managed Service for Kubernetes node

• After changing the node subnet mask in the cluster settings, the number of pods per node is not as expected

• What should I do if I get the node(s) had untolerated taint error?

• Why does the pod remain in the Pending status?

• What should I do if I get the DEADLINE_EXCEEDED error when exporting metrics?

• What should I do if HPA does not receive metrics?

• What should I do if mounting a volume to a pod times out?

• Why does mounting a volume with a large number of files take a long time?

• What should I do if nodes remain in the RECONCILING status for a long time?

This section describes typical issues you may encounter while using Managed Service for Kubernetes and gives troubleshooting recommendations.

Error creating a cluster in a different folder's cloud networkError creating a cluster in a different folder's cloud network

Error message:

Permission denied

This error occurs when the resource service account has no required roles in the folder that contains the cloud network selected when creating the cluster.

To create a Managed Service for Kubernetes cluster in a cloud network of another folder, assign the resource service account the following roles in that folder:

• vpc.privateAdmin
• vpc.user

To use a public IP address, also assign the vpc.publicAdmin role.

Namespace fails to delete and remains TerminatingNamespace fails to delete and remains Terminating

This issue occurs when your namespace contains stuck resources that the namespace controller cannot delete.

To fix it, delete the stuck resources manually.

I am using Yandex Network Load Balancer together with an ingress controller. Why are some of my cluster's nodes UNHEALTHY?I am using Yandex Network Load Balancer together with an ingress controller. Why are some of my cluster's nodes UNHEALTHY?

This is normal behavior for a load balancer with External Traffic Policy: Local enabled. Only the Managed Service for Kubernetes nodes whose pods are ready to handle user traffic get the HEALTHY status. All other nodes are labeled as UNHEALTHY.

To check the policy type of a load balancer created using a LoadBalancer service, run this command:

kubectl describe svc <LoadBalancer_service_name> \
| grep 'External Traffic Policy'

For more information, see Parameters of a LoadBalancer service.

Why does the newly created PersistentVolumeClaim remain Pending?Why does the newly created PersistentVolumeClaim remain Pending?

This is normal for a PersistentVolumeClaim (PVC). The newly created PVC remains Pending until you create a pod that will use it.

To change the PVC status to Running:

1. View the PVC details:

   kubectl describe pvc <PVC_name> \
     --namespace=<namespace>
   

   Where --namespace is the namespace containing the PVC.

   The waiting for first consumer to be created before binding message means that the PVC is awaiting pod creation.

2. Create a pod for this PVC.

Why does my Managed Service for Kubernetes cluster fail to start after I update its node configuration?Why does my Managed Service for Kubernetes cluster fail to start after I update its node configuration?

Make sure the new configuration of Managed Service for Kubernetes nodes is within the quota:

If out of quotas, request an increase. Check both the Managed Service for Kubernetes and Compute Cloud quotas.

These checks are also useful where your cluster remains STARTING for a long time after being created or started. Also check the roles granted to the service account for cluster resources. It must have the k8s.clusters.agent role; for using public IP addresses, vpc.publicAdmin.

If the issue persists, create a support ticket. Specify the cluster ID, the time the error occurred, and the diagnostic results.

After changing the node subnet mask in the cluster settings, the number of pods per node is not as expectedAfter changing the node subnet mask in the cluster settings, the number of pods per node is not as expected

Solution: Recreate the node group.

Error updating ingress controller certificateError updating ingress controller certificate

Error message:

ERROR controller-runtime.manager.controller.ingressgroup Reconciler error
{"name": "some-prod", "namespace": , "error": "rpc error: code = InvalidArgument
desc = Validation error:\nlistener_specs[1].tls.sni_handlers[2].handler.certificate_ids:
Number of elements must be less than or equal to 1"}

The error occurs if different certificates are specified for the same ingress controller handler.

Solution: Edit and apply the ingress controller specifications so that each handler has only one certificate.

Why is DNS resolution not working in my cluster?Why is DNS resolution not working in my cluster?

A Managed Service for Kubernetes cluster may fail to resolve internal and external DNS requests for several reasons. To fix the issue:

1. Check the version of your Managed Service for Kubernetes cluster and node groups.
2. Make sure CoreDNS is up and running.
3. Make sure your Managed Service for Kubernetes cluster has enough CPU resources available.
4. Set up autoscaling.
5. Set up local DNS caching.

Check the version of your cluster and node groupsCheck the version of your cluster and node groups

1. Get the list of current Kubernetes versions:

   yc managed-kubernetes list-versions
   

2. Get the Managed Service for Kubernetes cluster version:

   yc managed-kubernetes cluster get <cluster_name_or_ID> | grep version:
   

   You can get the Managed Service for Kubernetes cluster ID and name with the list of clusters in the folder.

3. Get the Managed Service for Kubernetes node group version:

   yc managed-kubernetes node-group get <node_group_name_or_ID> | grep version:
   

   You can get the Managed Service for Kubernetes node group ID and name with the list of node groups in the cluster.

4. If the versions of your Managed Service for Kubernetes cluster and node groups are not on the list of current Kubernetes versions, upgrade them.

Make sure CoreDNS is up and runningMake sure CoreDNS is up and running

Get the list of CoreDNS pods and their statuses:

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

Make sure all pods have the Running status. If the status is different or DNS requests continue to fail, check the logs:

kubectl logs -l k8s-app=kube-dns -n kube-system --all-containers=true

For further diagnostics, follow the Kubernetes guide on checking DNS.

Make sure your cluster has enough CPU resources availableMake sure your cluster has enough CPU resources available

1. In the management console, select the folder.
2. Navigate to Managed Service for Kubernetes.
3. Locate the Managed Service for Kubernetes cluster you need in the list, click its name, and select the Node management tab.
4. Go to the Nodes tab and click the name of any Managed Service for Kubernetes node.
5. Navigate to the Monitoring tab.
6. Make sure that, in the CPU, [cores] chart, the used CPU values have not reached the total available CPU values. Check this for each Managed Service for Kubernetes cluster node.

Set up autoscalingSet up autoscaling

Set up DNS autoscaling based on the Managed Service for Kubernetes cluster size.

Set up local DNS cachingSet up local DNS caching

Set up NodeLocal DNS Cache. For optimal settings, install NodeLocal DNS Cache from Yandex Cloud Marketplace.

If the issue persists, create a support ticket. Specify the cluster ID, the time the error occurred, and the diagnostic results. Attach the CoreDNS logs and examples of failed DNS requests.

Creating a node group with the CLI results in a parameter conflict. How do I fix it?Creating a node group with the CLI results in a parameter conflict. How do I fix it?

Check whether you are specifying the --location, --network-interface, and --public-ip parameters in the same command. Providing them together causes the following errors:

• For the --location and --public-ip or --location and --network-interface pairs:

  ERROR: rpc error: code = InvalidArgument desc = Validation error:
  allocation_policy.locations[0].subnet_id: can't use "allocation_policy.locations[0].subnet_id" together with "node_template.network_interface_specs"
  

• For the --network-interface and --public-ip pair:

  ERROR: flag --public-ip cannot be used together with --network-interface. Use '--network-interface' option 'nat' to get public address
  

Make sure you only provide one of the three parameters in a command. It is enough to specify the location of a Managed Service for Kubernetes node group either in --location or in --network-interface.

To grant internet access to Managed Service for Kubernetes cluster nodes, do one of the following:

• Assign a public IP address to the cluster nodes, specifying --network-interface ipv4-address=nat or --network-interface ipv6-address=nat.
• Enable access to Managed Service for Kubernetes nodes from the internet after creating a node group.

Error connecting to a cluster using kubectlError connecting to a cluster using kubectl

Error message:

ERROR: cluster has empty endpoint

This error occurs if you try to connect to a cluster with no public IP address and get kubectl credentials for a public IP address using this command:

yc managed-kubernetes cluster \
   get-credentials <cluster_name_or_ID> \
   --external

To connect to the cluster's private IP address from a VM in the same network, get kubectl credentials using this command:

yc managed-kubernetes cluster \
   get-credentials <cluster_name_or_ID> \
   --internal

If you need to connect to a cluster from the internet, recreate the cluster and assign it a public IP address.

Errors connecting to a node over SSHErrors connecting to a node over SSH

Error messages:

Permission denied (publickey,password)

Too many authentication failures

The following situations cause errors when connecting to a Managed Service for Kubernetes node:

• No public SSH key is added to the Managed Service for Kubernetes node group metadata.

  Solution: Update the Managed Service for Kubernetes node group keys.

• An invalid public SSH key is added to the Managed Service for Kubernetes node group metadata.

  Solution: Change the format of the public key file to the appropriate one and update the Managed Service for Kubernetes node group keys.

• No private SSH key is added to an authentication agent (ssh-agent).

  Solution: Add a private key by running the ssh-add <path_to_private_key_file> command.

How do I provide internet access to my Managed Service for Kubernetes cluster nodes?How do I provide internet access to my Managed Service for Kubernetes cluster nodes?

If Managed Service for Kubernetes cluster nodes have no internet access, the following error occurs when trying to connect to the internet:

Failed to pull image "cr.yandex/***": rpc error: code = Unknown desc = Error response from daemon: Gethttps://cr.yandex/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

You can provide internet access to your Managed Service for Kubernetes cluster nodes in several ways:

• Set up a NAT gateway or NAT instance. With static routing in place, traffic will go through a gateway or a separate NAT instance.
• Assign a public IP addresses to your Managed Service for Kubernetes node group.

Why cannot I choose Docker as the container runtime?Why cannot I choose Docker as the container runtime?

Clusters running Kubernetes 1.24 or higher do not support the Docker container runtime. containerd is the only available runtime.

Error connecting a GitLab repository to Argo CDError connecting a GitLab repository to Argo CD

Error message:

FATA[0000] rpc error: code = Unknown desc = error testing repository connectivity: authorization failed

This error occurs if access to GitLab over HTTP(S) is disabled.

Solution: Enable HTTP(S) access. To do this:

1. In GitLab, in the left-hand panel, select Admin → Settings → General.
2. Under Visibility and access controls, find the Enabled Git access protocols setting.
3. In the list, select the item which allows access over HTTP(S).

For more information, see this GitLab guide.

Traffic loss when deploying app updates in a cluster with Yandex Application Load BalancerTraffic loss when deploying app updates in a cluster with Yandex Application Load Balancer

When your app traffic is managed by an Application Load Balancer and the load balancer's ingress controller traffic policy is set to externalTrafficPolicy: Local, the app processes requests on the same node they were delivered to by the load balancer. There is no traffic flow between nodes.

The default health check monitors the status of the node, not application. Therefore, Application Load Balancer traffic may go to a node where there is no application running. When you deploy a new app version in a cluster, the Application Load Balancer ingress controller requests the load balancer to update the backend group configuration. It takes at least 30 seconds to process the request, and the app may not receive any user traffic during that time.

To prevent this, we recommend setting up backend health checks on your Application Load Balancer. With health checks, the load balancer timely spots unavailable backends and reroutes traffic to healthy backends. Once the application is updated, traffic will again be distributed across all backends.

For more information, see Tips for configuring Yandex Application Load Balancer health checks and Metadata.annotations.

System time displayed incorrectly on nodes, as well as in container and Managed Service for Kubernetes cluster pod logsSystem time displayed incorrectly on nodes, as well as in container and Managed Service for Kubernetes cluster pod logs

Managed Service for Kubernetes cluster time may not match the time of other resources, such as VMs, if they use different time synchronization sources. For example, a Managed Service for Kubernetes cluster synchronizes with a time server (by default), whereas a VM synchronizes with a private or public NTP server.

Solution: Set up Managed Service for Kubernetes cluster time synchronization with your private NTP server. Follow these steps:

1. Specify the NTP server addresses in the DHCP settings of the subnets housing the master and worker nodes. If the worker nodes reside in other subnets, repeat the configuration procedure for those subnets.

   Management console

   CLI

   Terraform

   API

   1. In the management console, select the folder.
   2. Navigate to Managed Service for Kubernetes.
   3. Click the name of the Kubernetes cluster.
   4. Under Master configuration, click the subnet name.
   5. Click Edit in the top-right corner.
   6. In the window that opens, expand the DHCP settings section.
   7. Click Add and specify the IP address of your NTP server.
   8. Click Save changes.

   If you do not have the Yandex Cloud CLI yet, install and initialize it.

   The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.

   If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

   1. See the description of the CLI command for updating subnet settings:

      yc vpc subnet update --help
      

   2. Run the subnet command, specifying the NTP server IP address in the --ntp-server parameter:

      yc vpc subnet update <subnet_ID> --ntp-server <server_address>
      

   Tip

   To find out the IDs of the subnets containing the cluster, get detailed information about the cluster.

   1. In the Terraform configuration file, change the cluster subnet description. Add the dhcp_options section (if missing) with the ntp_servers parameter specifying the IP address of your NTP server:

      ...
      resource "yandex_vpc_subnet" "lab-subnet-a" {
        ...
        v4_cidr_blocks = ["<IPv4_address>"]
        network_id     = "<network_ID>"
        ...
        dhcp_options {
          ntp_servers = ["<IPv4_address>"]
          ...
        }
      }
      ...
      

      For more information about yandex_vpc_subnet properties in Terraform, see this provider guide.

   2. Apply the changes:

      1. In the terminal, navigate to the configuration file directory.

      2. Make sure the configuration is correct using this command:

         terraform validate
         

         If the configuration is valid, you will get this message:

         Success! The configuration is valid.
         

      3. Run this command:

         terraform plan
         

         You will see a list of resources and their properties. No changes will be made at this step. Terraform will show any errors in the configuration.

      4. Apply the configuration changes:

         terraform apply
         

      5. Type yes and press Enter to confirm the changes.

      Terraform will update all required resources. You can check the subnet update using the management console or this CLI command:

      yc vpc subnet get <subnet_name>
      

   Use the update method for the Subnet resource and provide the following in the request:

   • NTP server IP address in the dhcpOptions.ntpServers parameter.
   • dhcpOptions.ntpServers parameter to update in the updateMask parameter.

   Tip

   To find out the IDs of the subnets containing the cluster, get detailed information about the cluster.

   Warning

   For a highly available master hosted across three availability zones, you need to update each of the three subnets.

2. Enable connections from the cluster to NTP servers.

   Create a rule for outbound traffic in the cluster and node group security group:

   • Port range: 123. If your NTP server uses a port other than 123, specify that port.
   • Protocol: UDP.
   • Destination name: Address range.
   • IPv4 CIDR: <NTP_server_IP_address>/32. For a master hosted across three availability zones, specify three sections: <NTP_server_IP_address_in_subnet1>/32, <NTP_server_IP_address_in_subnet2>/32, and <NTP_server_IP_address_in_subnet3>/32.

3. Update the network settings in the cluster node group using one of the following methods:

   • Connect to each node in the group over SSH or via OS Login and run the sudo dhclient -v -r && sudo dhclient command.
   • Reboot the group nodes at any convenient time.

   Warning

   Updating network settings may cause the services within the cluster to become unavailable for a few minutes.

For worker nodes running systemd-timesyncd, time sources can also be configured via a DaemonSet. This option requires a privileged container and modifies the host configuration. Before proceeding, check the time synchronization service being used.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ntp-configurator
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ntp-configurator
  template:
    metadata:
      labels:
        app: ntp-configurator
    spec:
      hostPID: true
      hostNetwork: true
      initContainers:
        - name: configure-ntp
          image: ubuntu:22.04
          env:
            - name: NTP_SERVERS
              value: "<NTP_server_1> <NTP_server_2>"
          command:
            - /bin/bash
            - -ec
            - |
              nsenter --mount=/proc/1/ns/mnt -- /bin/sh -ec '
                mkdir -p /etc/systemd/timesyncd.conf.d
                printf "[Time]\nNTP=%s\n" "$1" > /etc/systemd/timesyncd.conf.d/90-custom-ntp.conf
                systemctl restart systemd-timesyncd
                systemctl is-active systemd-timesyncd
              ' sh "$NTP_SERVERS"
          securityContext:
            privileged: true
      containers:
        - name: sleep
          image: ubuntu:22.04
          command: ["/bin/sleep", "infinity"]

If the issue persists, create a support ticket. Specify the cluster ID, the time the error occurred, and the diagnostic results. Specify the NTP servers used and the exact time discrepancy.

What should I do if I deleted my Yandex Network Load Balancer or its target groups that were automatically created for a LoadBalancer service?What should I do if I deleted my Yandex Network Load Balancer or its target groups that were automatically created for a LoadBalancer service?

You cannot manually restore a Network Load Balancer or its target groups. Recreate your LoadBalancer service. This will automatically create a load balancer and target groups.

Error connecting a Yandex Compute Cloud VM instance as an external Managed Service for Kubernetes nodeError connecting a Yandex Compute Cloud VM instance as an external Managed Service for Kubernetes node

Error message:

Unable to create remote dir /home/kubernetes/bin/: ssh run `mkdir -p -m 0644 /home/kubernetes/bin/': Process exited with status 142
Please login as the user "NONE" rather than the user "root".

To fix the issue, recreate the VM instance, specifying disable_root: false in the metadata for the user-data key.

#cloud-config
datasource:
 Ec2:
  strict_id: false
disable_root: false
users:
- name: <username>
  sudo: ALL=(ALL) NOPASSWD:ALL
  shell: /bin/bash
  ssh_authorized_keys:
  - ssh-rsa <public_key_for_VM_access>

What should I do if I get the error?What should I do if I get the node(s) had untolerated taint error?

The error means that a taint is set on the node, for which the pod does not have a corresponding toleration. The taint effect determines the behavior: NoSchedule prevents new pods from being scheduled, NoExecute also evicts already running pods, and PreferNoSchedule applies a soft restriction.

Check the node taints and pod tolerations:

kubectl describe node <node_name>
kubectl describe pod <pod_name> --namespace <namespace>

If the pod must run on this node, add the corresponding toleration to the pod’s spec.tolerations. If the taint is no longer required, remove it or change its effect to PreferNoSchedule. For more details on taint and toleration, see this Kubernetes guide.

Why does the pod remain in the Pending status?Why does the pod remain in the Pending status?

View the pod description:

kubectl describe pod <pod_name> --namespace <namespace>

Messages under Events help determine the cause, e.g., insufficient resources, placement restrictions, or a volume mounting issue. The next steps depend on the error message.

If the information is insufficient, create a support ticket. Specify the cluster ID and attach the pod description. If the pod is already scheduled on a node, attach the kubelet, kernel, and system logs from that node. You can use a diagnostic script to collect the logs.

What to do if I get a error when exporting metrics?What to do if I get a DEADLINE_EXCEEDED error when exporting metrics?

When exporting Managed Service for Kubernetes metrics via the Monitoring API, the request may fail with a 504 code and a DEADLINE_EXCEEDED message. One possible reason is a large volume of metrics, causing your request to time out.

Reduce the data volume using the selectors parameter. For example, in the Prometheus configuration, specify the namespace:

scrape_configs:
  - job_name: yc-monitoring-export
    metrics_path: /monitoring/v2/prometheusMetrics
    params:
      folderId:
        - '<folder_ID>'
      service:
        - managed-kubernetes
      selectors:
        - 'namespace=<namespace>'
    bearer_token: '<IAM_token>'

This is a snippet of the metrics collection configuration: keep the server address settings from your configuration. For pod metrics, you can also use a name mask, e.g., pod=app*. You can learn more about the Monitoring query language here.

What to do if HPA does not receive metrics?What to do if HPA does not receive metrics?

If HPA reports FailedGetResourceMetric or requests to metrics.k8s.io time out, check the cluster security groups and the resources of the node running your Metrics Server.

If this node does not have enough resources, migrate the Metrics Server pod to another node:

1. Locate the pod name and node:

   kubectl get pods -n kube-system -o wide | grep metrics-server
   

2. Make sure that another node has enough resources and meets the pod placement rules. Prevent new pods from being scheduled on the original node and delete the Metrics Server pod:

   kubectl cordon <original_node>
   kubectl delete pod <metrics-server_pod_name> -n kube-system
   

   The controller will create a new pod. Until it starts, metrics may be unavailable.

3. Check the new pod’s readiness and the node it is placed on:

   kubectl get pods -n kube-system -o wide | grep metrics-server
   

4. After the pod starts, allow pod scheduling on the original node again:

   kubectl uncordon <original_node>
   

   If the migration fails, also uncordon the node before further diagnostics.

If the error persists, create a support ticket. Specify the cluster ID, the name of the affected pod, and attach the output of kubectl describe hpa --namespace <namespace>.

What to do if mounting a volume to a pod times out?What to do if mounting a volume to a pod times out?

When mounting a volume, you may get the following error:

Unable to attach or mount volumes: timed out waiting for the condition

Check the pod and PVC events to determine the cause:

kubectl describe pod <pod_name> --namespace <namespace>
kubectl describe pvc <PVC_name> --namespace <namespace>

If the kubelet logs contain a message about a slow file permission change, use this guide for volumes with a large number of files.

If the issue is related to attaching the disk to the node’s VM, stopping and restarting the VM may help. Stop the VM, wait for the STOPPED state, then start it again. This will interrupt the operation of pods on the node, so perform this operation during a period of minimal load.

Why does mounting a volume with a large number of files take a long time?Why does mounting a volume with a large number of files take a long time?

If mounting times out, check the kubelet log on the node:

sudo journalctl -u kubelet --no-pager --since today

If the volume has a lot of files then setting volume ownership could be slow... indicates that changing the owner and access permissions to the volume’s files is taking a long time.

If fsGroup is set for the pod, Kubernetes may recursively change permissions during mounting. For supported volumes, the fsGroupChangePolicy: OnRootMismatch parameter in spec.securityContext allows this operation to be skipped if the root directory’s permissions already match the expected ones. For more details on when this applies, see this Kubernetes guide.

If the issue persists, create a support ticket. Specify the cluster ID, the time the error occurred, and the diagnostic results.

What to do if nodes remain in the status for a long time?What to do if nodes remain in the RECONCILING status for a long time?

If the RECONCILING status persists for more than 20 minutes, check quota usage and the node status:

yc managed-kubernetes node-group list-nodes <node_group_ID> --format yaml

Kubelet stopped posting node status indicates that your kubelet has stopped reporting the node’s status. Connect to the node over SSH and check the services:

sudo systemctl status containerd kubelet

If the services are not working correctly, review their logs and restart the affected services:

sudo systemctl restart containerd
sudo systemctl restart kubelet

Restarting may affect the node’s applications. After restarting, recheck the status of the services and the node.

If the issue persists, create a support ticket. Specify the cluster ID, the time the error occurred, and the diagnostic results.