Creating and configuring a Managed Service for Kubernetes cluster with no internet access

You can create and configure a Managed Service for Kubernetes cluster with no internet connectivity. For this, you will need the following configuration:

• Managed Service for Kubernetes cluster and node group without a public address. You can only connect to such a cluster using a Yandex Cloud virtual machine.
• The cluster and node group are hosted by subnets with no internet access.
• Service accounts have no roles to use resources with internet access, e.g., Yandex Network Load Balancer.
• Cluster security groups restrict incoming and outgoing traffic.

To create a Managed Service for Kubernetes cluster with no internet access:

1. Prepare the infrastructure for Managed Service for Kubernetes.
2. Set up a virtual machine.
3. Check cluster availability.
4. (Optional) Connect a private Docker image registry.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost includes:

• Fee for the Managed Service for Kubernetes cluster: using the master (see Managed Service for Kubernetes pricing).
• Fee for cluster nodes and VMs: using computing resources, operating system, and storage (see Compute Cloud pricing).
• Fee for a public IP address for a VM, which is used to connect to the cluster (see Virtual Private Cloud pricing).
• Key Management Service fee: number of active key versions (with Active or Scheduled For Destruction for status) and completed cryptographic operations (see Key Management Service pricing).

Prepare the infrastructure for Managed Service for KubernetesPrepare the infrastructure for Managed Service for Kubernetes

Set up a virtual machineSet up a virtual machine

As the Managed Service for Kubernetes cluster has no internet access, you can only connect to it from a VM that is in the same network as the cluster. Therefore, to check the cluster availability, set up the infrastructure:

1. Create the required resources:

   Manually

   Terraform

   1. Create a service account named vm-sa with the k8s.cluster-api.cluster-admin and k8s.admin roles. This account will be used to connect to the Managed Service for Kubernetes cluster.

   2. Create a security group named vm-security-group and specify a rule for incoming traffic in it:

      • Port range: 22.
      • Protocol: TCP.
      • Source: CIDR.
      • CIDR blocks: 0.0.0.0/0.

   3. Create a Linux VM with the following parameters:

      • Subnet: my-subnet.
      • Public IP address: Auto or you can reserve a static public IP address and assign it to the new VM.
      • Security groups: vm-security-group.
      • Service account: vm-sa.

   1. Download the virtual-machine-for-k8s.tf configuration file to the directory containing the k8s-cluster-with-no-internet.tf file.

      This file describes:

      • Service account for VM.
      • Security group for VM.
      • VM.

   2. Specify the following in the virtual-machine-for-k8s.tf file:

      • Folder ID.
      • ID of the network created together with the Managed Service for Kubernetes cluster.
      • ID of the subnet created together with the Managed Service for Kubernetes cluster and residing in the ru-central1-a availability zone. You can find this zone in the VM settings.
      • Username to be used for connection to the VM over SSH.
      • Absolute path to the public part of the SSH key for connection to the VM.

   3. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   4. Create the required infrastructure:

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

2. Connect to the VM over SSH:

   ssh <username>@<VM_public_IP_address>
   

   Where <username> is the VM account username.

3. Install the Yandex Cloud command line interface (YC CLI).

4. Create a YC CLI profile.

5. Install kubect and set it up to work with the created cluster.

Check cluster availabilityCheck cluster availability

Run this command on the VM:

kubectl cluster-info

The command will return the following Managed Service for Kubernetes cluster information:

Kubernetes control plane is running at https://<cluster_address>
CoreDNS is running at https://<cluster_address>/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

(Optional) Connect a private Docker image registry(Optional) Connect a private Docker image registry

You can connect a private Docker image registry to your Managed Service for Kubernetes cluster. To get authenticated in the registry and connect to it over HTTPS, the cluster will need certificates issued by the CA (Certificate Authority). Use the DaemonSet controller to add and later automatically update the certificates on cluster nodes. It runs the following process in pods:

1. A Bash script constantly checks cluster nodes for required certificates.
2. If not, the certificates are copied from the Kubernetes secret and updated.
3. The containerd runtime environment is restarted.

To configure certificate updates using DaemonSet, do the following on your VM:

1. Place the .crt certificate files.

2. Create a file named certificate-updater-namespace.yaml with the namespace configuration. This namespace will be used for DaemonSet operation and isolation:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: certificate-updater
     labels:
       name: certificate-updater
   

3. Create a certificate-updater-daemonset.yaml file with the DaemonSet configuration:

   File contents

   ---
   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     name: certificate-updater-deny-all
     namespace: certificate-updater
   spec:
     podSelector:
       matchLabels:
         k8s-app: certificate-updater
     policyTypes:
       - Ingress
       - Egress
     ingress: []
     egress:  []
   ---
   apiVersion: "apps/v1"
   kind: DaemonSet
   metadata:
     name: certificate-updater
     namespace: certificate-updater
     labels:
       k8s-app: certificate-updater
       version: 1v
   spec:
     selector:
       matchLabels:
         k8s-app: certificate-updater
     template:
       metadata:
         labels:
           k8s-app: certificate-updater
       spec:
         hostPID: true
         hostIPC: true
         containers:
         - name: certificate-updater
           image: cr.yandex/yc/mk8s-openssl:stable
           command: 
             - sh
             - -c
             - |
               while true; do
                 diff -x '.*' -r /mnt/user-cert-path/ /usr/local/share/ca-certificates
                 if [ $? -ne 0 ];
                   then
                       echo "Removing all old certificates"
                       rm -r /usr/local/share/ca-certificates/*
                       echo "Copying certificates from configmap"
                       cp /mnt/sbin/update-ca-certificates /usr/sbin/
                       cp /mnt/user-cert-path/* /usr/local/share/ca-certificates
         
                       echo "Updating cerfificates authorities"
                       update-ca-certificates 
   
                       echo "Restarting containerd"
                       ps -x -o pid= -o comm= | awk '$2 ~ "^(containerd|dockerd)$" { print $1 }' | xargs kill
                       #systemd will get them back less than a minute
                   else
                     echo "Doing Nothing as no certs has not been changed"
                   fi
                 sleep 60
               done
           imagePullPolicy: Never
           securityContext:
             privileged: true
           resources:
             limits:
               memory: 200Mi
             requests:
               cpu: 100m
               memory: 200Mi
           volumeMounts:
           - mountPath: /etc/
             name: etc
           - mountPath: /usr/local/share/ca-certificates
             name: docker-cert
           - name: secret
             mountPath: /mnt/user-cert-path
           - name: sbin
             mountPath: /mnt/sbin
             readOnly: true
           - name: ca-cert
             mountPath: /usr/share/ca-certificates
         volumes:
         - name: secret
           secret:
             secretName: crt
         - name: sbin
           hostPath:
             path: /usr/sbin/
             type: Directory
         - name: ca-cert
           hostPath:
             path: /usr/share/ca-certificates
             type: Directory
         - name: docker-cert
           hostPath:
             path: /usr/local/share/ca-certificates
             type: DirectoryOrCreate
         - name: etc
           hostPath:
             path: /etc/
             type: Directory
   

4. Create a namespace:

   kubectl apply -f certificate-updater-namespace.yaml
   

5. Create a secret with the contents of the certificates issued by the CA:

   kubectl create secret generic crt \
      --from-file=<certificate_file_path>.crt \
      --namespace="certificate-updater"
   

   Specify a certificate with the .crt extension in the command. If you need to add multiple certificates, provide each one in the command using the --from-file flag.

   You can check the secret configuration using the command below and see if it contains the information about the certificates:

   kubectl get secret crt -o yaml
   

6. Create a DaemonSet:

   kubectl apply -f certificate-updater-daemonset.yaml
   

Now you can monitor the state of the DaemonSet controller. As soon as the certificates are updated, the cluster will restart the containerd runtime environment processes.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them.

If you assigned a static public IP address to your VM, release and delete it.