Configuring security groups

Security groups follow the All traffic that is not allowed is prohibited principle. For a cluster to work:

1. In its security groups, create rules to allow relevant traffic for the cluster nodes:

   • Service traffic within the cluster.

     These rules allow intra-cluster communication and connection to the cluster via the Kubernetes API.

   • Connections to services from the internet.

   • Connections to nodes over SSH.

   Tip

   Place each of these rule sets into a separate security group.

   This will make it easy to apply these security groups to the cluster and its node groups.

2. (Optional) If planning to use a Yandex Application Load Balancer L7 balancer together with the cluster, add rules for the balancer as well.

3. Apply security groups with these rules to the cluster and node groups.

You can specify more detailed rules for your security groups, e.g., to allow traffic only in specific subnets.

Security groups must be correctly configured for all subnets that will host the cluster. This determines the performance and availability of the cluster and the services running there.

Prior to editing security groups or the settings of any included rules, make sure this is not going to disrupt the cluster or its node groups.

Creating rules for service trafficCreating rules for service traffic

Allowing traffic for a cluster and node groupsAllowing traffic for a cluster and node groups

For the cluster to work correctly, create rules for incoming and outgoing traffic and apply them to the cluster and node groups:

1. Add rules for incoming traffic that allow:

   • Health checks of the network load balancer:
     • Port range: 0-65535.
     • Protocol: TCP.
     • Source: Load balancer healthchecks.
   • Transferring service traffic between the master and the nodes:
     • Port range: 0-65535.
     • Protocol: Any (Any).
     • Source: Security group.
     • Security group: Current (Self).
   • Health checks of nodes using ICMP requests from subnets within Yandex Cloud:
     • Protocol: ICMP.
     • Source: CIDR.
     • CIDR blocks: Address ranges of subnets within Yandex Cloud to health check the cluster from, e.g.:
       • 10.0.0.0/8
       • 192.168.0.0/16
       • 172.16.0.0/12

2. Add a rule for outgoing service traffic between the master and the nodes:

   • Port range: 0-65535.
   • Protocol: Any (Any).
   • Destination name: Security group.
   • Security group: Current (Self).

Allowing traffic for node groupsAllowing traffic for node groups

For node groups to run properly, create rules for incoming and outgoing traffic and apply them to node groups:

1. Add a rule for incoming traffic that allows traffic transfer between pods and services:

   • Port range: 0-65535.
   • Protocol: Any (Any).
   • Source: CIDR.
   • CIDR blocks: CIDRs of the cluster and services, e.g., 10.96.0.0/16 and 10.112.0.0/16.

2. Add a rule for outgoing traffic that allows node group nodes to connect to external resources, e.g., to download images from Docker Hub or work with Yandex Object Storage:

   • Port range: 0-65535.
   • Protocol: Any (Any).
   • Destination name: CIDR.
   • CIDR blocks: 0.0.0.0/0.

   Note

   If you are using a highly available master and do not want to allow nodes to connect to any addresses (CIDR: 0.0.0.0/0), allow outgoing traffic to the CIDRs of the subnets where the master hosts are located or to the cluster CIDR. This ensures node access to the master.

   Examples

   • Access to the cluster CIDR:

     • Port range: 0-65535.
     • Protocol: Any (Any).
     • Destination name: CIDR.
     • CIDR blocks: 10.131.0.32/32.

   • Access to the CIDRs of subnets with master hosts:

     • Port range: 0-65535.
     • Protocol: Any (Any).
     • Destination name: CIDR.
     • CIDR blocks: 10.128.0.0/24, 10.129.0.0/24, 10.131.0.0/24.

Allowing traffic for clusterAllowing traffic for cluster

For the cluster to work correctly and to allow incoming connections, create rules for incoming and outgoing traffic and apply them to the cluster:

1. Add rules for incoming traffic that allow connecting to the master via ports 6443 and 443. This will allow you to access the Kubernetes API and manage the cluster using kubectl and other utilities.

   Create two rules for incoming traffic, one per port:

   • Port range: 443, 6443.
   • Protocol: TCP.
   • Source: CIDR.
   • CIDR blocks: Specify the IP address range of the subnets you will manage the cluster from, e.g.:
     • 85.23.23.22/32: For an external network.
     • 192.168.0.0/24: For an internal network.

2. Add rules for outgoing traffic that allow:

   • Traffic between the master and metric-server pods:

     • Port range: 4443.
     • Protocol: TCP.
     • Destination name: CIDR.
     • CIDR blocks: Specify the cluster CIDR, e.g., 10.96.0.0/16.

   • Master host connection to NTP servers for time syncing:

     • Port range: 123.
     • Protocol: UDP.
     • Destination name: CIDR.
     • CIDR blocks: 0.0.0.0/0.

Creating a rule for connecting to services from the internetCreating a rule for connecting to services from the internet

To be sure that the services running on nodes are accessible from the internet and subnets within Yandex Cloud, create a rule for the incoming traffic and apply it to the node group:

• Port range: 30000-32767.
• Protocol: TCP.
• Source: CIDR.
• CIDR blocks: 0.0.0.0/0.

Creating a rule for connecting to nodes via SSHCreating a rule for connecting to nodes via SSH

To access the nodes via SSH, create a rule for incoming traffic and apply it to the node group:

• Port range: 22.
• Protocol: TCP.
• Source: CIDR.
• CIDR blocks: IP address ranges of subnets within Yandex Cloud and public IP addresses of computers on the internet, for example:
  • 10.0.0.0/8
  • 192.168.0.0/16
  • 172.16.0.0/12
  • 85.32.32.22/32

Applying security groups with rulesApplying security groups with rules

Depending on the rules the security groups contain, these groups must be assigned to a cluster or node group:

Examples of rulesExamples of rules

Security groups for a Managed Service for Kubernetes clusterSecurity groups for a Managed Service for Kubernetes cluster

Let’s assume you want to create a Kubernetes cluster which:

• Uses the following configuration:

  • A highly-available master in the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.
  • A node group named worker-nodes-a in the ru-central1-a availability zone.
  • Cluster CIDR: 10.96.0.0/16; CIDR of services: 10.112.0.0/16.

• Follows these traffic exchange policies:

  • Health checks of nodes using ICMP requests are allowed from the following subnets within Yandex Cloud:

    • 10.0.0.0/8
    • 172.16.0.0/12
    • 192.168.0.0/16

  • Connections to services from the internet are allowed from any addresses: 0.0.0.0/0.

  • Connections to nodes via SSH are allowed from the only address: 85.32.32.22/32.

  • Access to the Kubernetes API is allowed from the only subnet: 203.0.113.0/24.

To create such a cluster:

Security groups for an Application Load Balancer and Managed Service for Kubernetes clusterSecurity groups for an Application Load Balancer and Managed Service for Kubernetes cluster

Let's assume you need to create rules for the following conditions:

• You need to deploy a load balancer with a public IP address accepting HTTPS traffic in three subnets. These subnets' CIDR blocks are 10.128.0.0/24, 10.129.0.0/24, and 10.130.0.0/24.
• The cluster's CIDR block is 10.96.0.0/16 and the service CIDR block is 10.112.0.0/16. These were specified when creating the cluster.
• The cluster's node group resides in a subnet with the 10.140.0.0/24 CIDR block.
• SSH access to nodes and cluster management via API, kubectl and other utilities are restricted to 203.0.113.0/24.

locals {
  network-id = "<cloud_network_ID>"
}

resource "yandex_vpc_security_group" "alb" {
  description = "Load balancer security group"
  name        = "alb"
  network_id  = local.network-id
  ingress {
    description    = "Rule to allow incoming HTTP traffic"
    port           = 80
    protocol       = "TCP"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description    = "Rule to allow incoming HTTPS traffic"
    port           = 443
    protocol       = "TCP"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description       = "Rule for load balancer node health checks"
    port              = 30080
    protocol          = "TCP"
    predefined_target = "loadbalancer_healthchecks"
  }
  egress {
    description    = "Rule for sending traffic to nodes, including health checks"
    from_port      = 0
    to_port        = 65535
    protocol       = "TCP"
    v4_cidr_blocks = ["10.140.0.0/24"]
  }
}

resource "yandex_vpc_security_group" "nodegroup-backend" {
  description = "Node group's security group for backend health checks"
  name        = "nodegroup-backend"
  network_id  = local.network-id
  ingress {
    description    = "Backend health check rule"
    port           = 10501
    protocol       = "TCP"
    v4_cidr_blocks = ["10.128.0.0/24", "10.129.0.0/24", "10.130.0.0/24"]
  }
}

resource "yandex_vpc_security_group" "k8s-cluster-nodegroup-traffic" {
  description = "Cluster and node group's security group for service traffic"
  name        = "k8s-cluster-nodegroup-traffic"
  network_id  = local.network-id
  ingress {
    description       = "Network load balancer rule"
    from_port         = 0
    to_port           = 65535
    protocol          = "TCP"
    predefined_target = "loadbalancer_healthchecks"
  }
  ingress {
    description       = "Rule for traffic between the master and nodes"
    from_port         = 0
    to_port           = 65535
    protocol          = "ANY"
    predefined_target = "self_security_group"
  }
  ingress {
    description    = "Rule for health checks of nodes from subnets within Yandex Cloud"
    from_port      = 0
    to_port        = 65535
    protocol       = "IPV6_ICMP"
    v4_cidr_blocks = ["10.0.0.0/8", "192.168.0.0/16", "172.16.0.0/12"]
  }
  egress {
    description       = "Rule for traffic between the master and nodes"
    from_port         = 0
    to_port           = 65535
    protocol          = "ANY"
    predefined_target = "self_security_group"
  }
}

resource "yandex_vpc_security_group" "nodegroup-traffic" {
  description = "Node group's security group for service traffic"
  name        = "nodegroup-traffic"
  network_id  = local.network-id
  ingress {
    description    = "Rule for traffic between pods and services"
    from_port      = 0
    to_port        = 65535
    protocol       = "ANY"
    v4_cidr_blocks = ["10.96.0.0/16", "10.112.0.0/16"]
  }
  egress {
    description    = "Rule for external resource access"
    from_port      = 0
    to_port        = 65535
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "yandex_vpc_security_group" "k8s-cluster-traffic" {
  description = "Cluster security group for control plane traffic"
  name        = "k8s-cluster-traffic"
  network_id  = local.network-id
  ingress {
    description    = "Rule for access to the Kubernetes API"
    port           = 443
    protocol       = "TCP"
    v4_cidr_blocks = ["203.0.113.0/24"]
  }
  ingress {
    description    = "Rule for access to the Kubernetes API"
    port           = 6443
    protocol       = "TCP"
    v4_cidr_blocks = ["203.0.113.0/24"]
  }
  egress {
    description    = "Rule for traffic between the master and metric-server pods"
    port           = 4443
    protocol       = "TCP"
    v4_cidr_blocks = ["10.96.0.0/16"]
  }
}

resource "yandex_vpc_security_group" "nodegroup-services-access" {
  description = "Node group's security group for accessing services from the internet"
  name        = "nodegroup-services-access"
  network_id  = local.network-id
  ingress {
    description    = "Rules for accessing services from the internet and Yandex Cloud subnets"
    from_port      = 30000
    to_port        = 32767
    protocol       = "TCP"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "yandex_vpc_security_group" "nodegroup-ssh" {
  description = "Node group's security group for SSH access to nodes"
  name        = "nodegroup-ssh"
  network_id  = local.network-id
  ingress {
    description    = "Rules for SSH access to nodes"
    port           = 22
    protocol       = "TCP"
    v4_cidr_blocks = ["203.0.113.0/24"]
  }
}