Creating a self-managed Kubernetes cluster using the Yandex Cloud provider for the Kubernetes Cluster API

Cluster-api-provider-yandex is a provider for deploying a self-managed Kubernetes cluster in Yandex Cloud infrastructure using the Kubernetes Cluster API.

The cluster is deployed on virtual machines Yandex Compute Cloud and an L7 Yandex Application Load Balancer.

Advantages of using Yandex Cloud provider for creating clusters:

• Integration with the Yandex Cloud API.
• Declarative approach to cluster creation and management.
• Ability to describe the cluster as a custom resource CustomResourceDefinition.
• Wide range of parameters for configuring cluster compute resources.
• Custom OS images for master and nodes.
• Custom Control Plane.
• Alternative to Terraform in CI processes.

Provider compatibility with the Kubernetes Cluster API

To deploy a Kubernetes cluster in Yandex Cloud using the Cluster API:

1. Get your cloud ready.
2. Set up your environment.
3. Prepare an OS image for cluster nodes.
4. Get a Docker image with the Yandex Cloud provider.
5. Install the Yandex Cloud provider and the Kubernetes Cluster API provider.
6. Generate cluster manifests.
7. Deploy a cluster.
8. Connect to the cluster.
9. Install the CCM.
10. Install the CNI.
11. Check the connection between the managing cluster and the new cluster.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

The infrastructure support costs include:

• Fee for computing resources and disks of VMs used for Kubernetes cluster deployment, auxiliary VM, and Managed Service for Kubernetes managing cluster nodes (see Compute Cloud pricing).
• Fee for using an L7 load balancer’s computing resources (see Yandex Application Load Balancer pricing).
• Fee for using Managed Service for Kubernetes managing cluster master and outbound traffic (see Yandex Managed Service for Kubernetes pricing).
• Fee for public IP addresses for auxiliary VMs and Managed Service for Kubernetes managing cluster (see Yandex Virtual Private Cloud pricing).
• Fee for using the NAT gateway (see Yandex Virtual Private Cloud pricing).

Optional costsOptional costs

• If intending to use a custom image for the new Kubernetes cluster nodes:
  • Fee for storing the image in the bucket and data operations (see Yandex Object Storage pricing).
  • Fee for storing the image in Compute Cloud (see Yandex Compute Cloud pricing).
• If intending to use a custom Docker image to deploy the Yandex Cloud provider in the managing cluster, fee for storing a Docker image in the registry and outgoing traffic (see Yandex Container Registry pricing).

Set up your infrastructureSet up your infrastructure

1. Prepare a Yandex Cloud service account:

   1. Create a service account you will use to create resources for the cluster.
   2. Assign the compute.editor and alb.editor roles for the folder to the service account.
   3. Create an authorized key for a service account in JSON format.

2. If your folder does not have a Virtual Private Cloud network yet, create it. Also create a subnet.

3. The new cluster infrastructure will automatically be assigned the default security group which is created together with the network. Add the following rules for incoming traffic to this group:

   Protocol	Port range	Source type	Source	Description
   TCP	0-65535	Security group	Balancer	Health checks by an L7 load balancer
   Any	8443	CIDR	0.0.0.0/0	Access to the Kubernetes API

4. The created cluster will be accessible within the cloud network via an internal IP address. To enable remote access to the cluster:

   1. Create an auxiliary VM with a public IP address and the default security group in the same network where your cluster will be deployed.
   2. Install kubectl on the auxiliary VM.

5. Create a Managed Service for Kubernetes managing cluster with a public IP address and a node group. You will need this cluster to deploy the new cluster using the Cluster API and to manage the cluster infrastructure.

   Tip

   You can also deploy the managing cluster locally, for example, using the kind utility.

6. For the new cluster to have internet access and be able to push Docker images, configure a NAT gateway for the subnet the new cluster will be located in.

Set up your environmentSet up your environment

The environment is configured on the local computer.

1. Install the following tools:

   • Go 1.22.0 or higher.
   • Docker 17.03 or higher.
   • kubectl 1.11.3 or higher.
   • clusterctl 1.5.0 or higher.

2. Configure kubectl access to the Managed Service for Kubernetes managing cluster.

   If the managing cluster is deployed locally with the help of kind, configure access to it as per this guide.

3. Clone the cluster-api-provider-yandex repository and navigate to the project directory.

   git clone https://github.com/yandex-cloud/cluster-api-provider-yandex.git
   cd cluster-api-provider-yandex
   

Prepare an OS image for cluster nodesPrepare an OS image for cluster nodes

The OS image deployed on the nodes of the new cluster must be ready to work with the Kubernetes Cluster API and compatible with Compute Cloud.

You can use a ready-made test image or build a custom one:

Get a Docker image with the Yandex Cloud providerGet a Docker image with the Yandex Cloud provider

You can use a ready-made Docker image with the Yandex Cloud provider from a public Yandex Container Registry or build your own image from the source code.

Install the Yandex Cloud provider and the Kubernetes Cluster API providerInstall the Yandex Cloud provider and the Kubernetes Cluster API provider

1. Initialize the managing cluster:

   clusterctl init
   

   The managing cluster will have the core components of the Kubernetes Cluster API and cert-manager.

2. Create a custom resource definition (CustomResourceDefinitions, CRD) for the new cluster:

   make install
   

3. Retrieve a list of installed CRDs:

   kubectl get crd | grep cluster.x-k8s.io
   

   To get a manifest for a specific CRD, run the following command:

   kubectl get crd <CRD_name> \
     --output yaml
   

4. Create a namespace for the Yandex Cloud provider:

   kubectl create namespace capy-system
   

5. Create a secret with the Yandex Cloud service account's authorized key:

   kubectl create secret generic yc-sa-key \
     --from-file=key=<path_to_file_with_authorized_key> \
     --namespace capy-system
   

6. Install the Yandex Cloud provider:

   make deploy
   

Generate cluster manifestsGenerate cluster manifests

1. Get the IDs of Yandex Cloud resources to deploy a cluster:

   • OS image
   • Folder
   • Availability zone
   • Network
   • Subnet in the selected availability zone.

2. Provide the IDs to these environment variables:

   export YANDEX_CONTROL_PLANE_MACHINE_IMAGE_ID=<image_ID>
   export YANDEX_FOLDER_ID=<folder_ID>
   export YANDEX_NETWORK_ID=<network_ID>
   export YANDEX_SUBNET_ID=<subnet_ID>
   export YANDEX_ZONE_ID=<availability_zone_ID>
   

   If you did not build a custom OS image, set the YANDEX_CONTROL_PLANE_MACHINE_IMAGE_ID variable to fd8a3kknu25826s8hbq3. This is the ID of a test Ubuntu 24.04 image compatible with Kubernetes 1.31.4.

3. Generate cluster manifests:

   clusterctl generate cluster <name_of_new_cluster> \
     --from templates/cluster-template.yaml > /tmp/capy-cluster.yaml
   

   The capy-cluster.yaml manifest will describe the following:

   • L7 Application Load Balancer with a dynamic internal IP address. You can give it a fixed IP address.

     Warning

     Once the cluster is created, you will not be able to assign a fixed IP address to the L7 load balancer.

   • Three Control Plane nodes for the cluster.

4. Optionally, to deploy workload cluster nodes right away, add their description to the manifest.

   clusterctl generate cluster <name_of_new_cluster> \
       --worker-machine-count <number_of_workload_nodes> \
       --from templates/cluster-template.yaml > /tmp/capy-cluster.yaml
   

Optionally, configure the API server endpointOptionally, configure the API server endpoint

Specify the parameters for the L7 load balancer in the capy-cluster.yaml manifest:

  loadBalancer:
    listener:
      address: <fixed_IP_address_from_subnet_range>
      subnet:
        id: <subnet_ID>

Deploy a clusterDeploy a cluster

Run this command:

kubectl apply -f /tmp/capy-cluster.yaml

You can monitor cluster creation progress from the Yandex Cloud management console and the capy-controller-manager pod logs:

kubectl logs <capy-controller-manager_pod_name> \
  --namespace capy-system \
  --follow

Connect to the clusterConnect to the cluster

The details for connection to the new cluster will be stored in the <name_of_new_cluster>-kubeconfig secret in the managing cluster.

1. Get the data from the secret:

   kubectl get secret <name_of_new_cluster>-kubeconfig \
     --output yaml | yq -r '.data.value' | base64 \
     --decode > capy-cluster-config
   

2. Provide the kubectl configuration file to the auxiliary VM:

   scp <path_to_capy-cluster-config_file_on_local_computer> \
   <username>@<VM_public_IP_address>:/home/<username>/.kube/config
   

3. Connect to the auxiliary VM over SSH.

4. Make sure the new cluster is accessible:

   kubectl cluster-info
   

Install a CCM to the new clusterInstall a CCM to the new cluster

For connection between the cluster resources and Yandex Cloud resources, install a cloud controller manager to the new cluster, e.g., the Kubernetes Cloud Controller Manager for Yandex Cloud.

Install a CNI to the new clusterInstall a CNI to the new cluster

To provide network functionality for pods in the new cluster, install to it a container network interface, e.g., Cilium or Calico.

For more information, see this documentation:

• Cilium Quick Installation.
• Quickstart for Calico on Kubernetes.

Check the connection between the managing cluster and the new clusterCheck the connection between the managing cluster and the new cluster

1. Connect to the auxiliary VM and make sure that all the pods with the necessary system components have been deployed in the cluster:

   kubectl get pods --all-namespaces
   

   Output example:

   NAMESPACE     NAME                                                       READY   STATUS    RESTARTS   AGE
   kube-system   calico-kube-controllers-695bcfd99c-rcc42                   1/1     Running   0          3h55m
   kube-system   calico-node-9qhxj                                          1/1     Running   0          3h55m
   kube-system   coredns-7c65d6cfc9-52tvn                                   1/1     Running   0          4h50m
   kube-system   coredns-7c65d6cfc9-dpgvg                                   1/1     Running   0          4h50m
   kube-system   etcd-capy-cluster-control-plane-p646q                      1/1     Running   0          4h50m
   kube-system   kube-apiserver-capy-cluster-control-plane-p646q            1/1     Running   0          4h50m
   kube-system   kube-controller-manager-capy-cluster-control-plane-p646q   1/1     Running   0          4h50m
   kube-system   kube-proxy-wb7jr                                           1/1     Running   0          4h50m
   kube-system   kube-scheduler-capy-cluster-control-plane-p646q            1/1     Running   0          4h50m
   kube-system   yandex-cloud-controller-manager-nwhwv                      1/1     Running   0          26s
   

2. Use your local computer to check the connection between the managing cluster and the new cluster:

   clusterctl describe cluster <name_of_new_cluster>
   

   Result:

   NAME                                                             READY  SEVERITY  REASON  SINCE  MESSAGE                                  
   Cluster/capy-cluster                                             True                     10s
   ├─ClusterInfrastructure - YandexCluster/capy-cluster                                   
   └─ControlPlane - KubeadmControlPlane/capy-cluster-control-plane  True                     10s                                                                                       
     └─3 Machines...                                                True                     3m9s   See capy-cluster-control-plane-cf72l, capy-cluster-control-plane-g9jw7, ...
   

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the Kubernetes cluster created using the Cluster API:

   kubectl delete -f /tmp/capy-cluster.yaml
   

2. Delete CRD from the Managed Service for Kubernetes managing cluster:

   make uninstall
   

3. Delete the Yandex Cloud provider controller from the managing cluster:

   make undeploy
   

4. Delete the auxiliary Yandex Cloud resources if you had created them:

   • Node group of the Managed Service for Kubernetes managing cluster
   • Managed Service for Kubernetes managing cluster
   • Auxiliary VM
   • NAT gateway
   • OS image in Compute Cloud
   • OS image in Object Storage
   • Bucket
   • Docker image
   • Registry