Configuring security groups

Security groups follow the All traffic that is not allowed is prohibited principle. For a cluster to work:

1. In its security groups, create rules to allow relevant traffic for the cluster nodes:

   • Service traffic within the cluster.

     These rules allow intra-cluster communication and connection to the cluster via the Kubernetes API.

   • Connections to services from the internet.

   • Connections to nodes over SSH.

   Tip

   Place each of these rule sets into a separate security group.

   This will make it easy to apply these security groups to the cluster and its node groups.

2. Optionally, if you are going to use an L7 Yandex Application Load Balancer together with the cluster, add rules for the balancer as well.

3. Apply security groups with these rules to the cluster and its node groups.

You can specify more granular rules for your security groups, e.g., to allow traffic only in specific subnets.

You must configure security groups correctly for all subnets that will host the cluster. This determines the performance and availability of the cluster and services running in it.

Prior to editing the list of security groups or settings of any included rules, make sure this will not disrupt the cluster or its node groups.

Creating rules for service trafficCreating rules for service traffic

Allowing traffic for a cluster and node groupsAllowing traffic for a cluster and node groups

For the cluster to work correctly, create rules for incoming and outgoing traffic and apply them to the cluster and its node groups:

1. Add rules for incoming traffic that allow:

   • Health checks of the network load balancer:
     • Port range: 0-65535.
     • Protocol: TCP.
     • Source: Load balancer healthchecks.
   • Transmitting service traffic between the master and nodes:
     • Port range: 0-65535.
     • Protocol: Any (Any).
     • Source: Security group.
     • Security group: Current (Self).
   • Health checks of nodes using ICMP messages from subnets within Yandex Cloud:
     • Protocol: ICMP.
     • Source: CIDR.
     • CIDR blocks: Address ranges of subnets within Yandex Cloud for health checks, e.g.:
       • 10.0.0.0/8
       • 192.168.0.0/16
       • 172.16.0.0/12

2. Add a rule for outgoing service traffic between the master and nodes:

   • Port range: 0-65535.
   • Protocol: Any (Any).
   • Destination name: Security group.
   • Security group: Current (Self).

Allowing traffic for node groupsAllowing traffic for node groups

For node groups to run properly, create rules for incoming and outgoing traffic and apply them to the node groups:

1. Add an incoming traffic rule that allows traffic transfer between pods and services:

   • Port range: 0-65535.
   • Protocol: Any (Any).
   • Source: CIDR.
   • CIDR blocks: CIDRs of the cluster and services, e.g., 10.96.0.0/16 and 10.112.0.0/16.

2. Add an outgoing traffic rule that allows node group nodes to connect to external resources, e.g., Container Registry, Cloud Registry, or Docker Hub Docker image registries, as well as the Object Storage buckets:

   • Port range: 0-65535.
   • Protocol: Any (Any).
   • Destination name: CIDR.
   • CIDR blocks: 0.0.0.0/0.

   Note

   If you are using a highly available master and do not want to allow nodes to connect to any addresses (CIDR: 0.0.0.0/0), allow outgoing traffic to the CIDRs of subnets where the master hosts are located or to the cluster CIDR. This ensures node access to the master.

   Examples

   • Access to the cluster CIDR:

     • Port range: 0-65535.
     • Protocol: Any (Any).
     • Destination name: CIDR.
     • CIDR blocks: 10.131.0.32/32.

   • Access to the CIDRs of subnets with master hosts:

     • Port range: 0-65535.
     • Protocol: Any (Any).
     • Destination name: CIDR.
     • CIDR blocks: 10.128.0.0/24, 10.129.0.0/24, 10.131.0.0/24.

Allowing traffic for a clusterAllowing traffic for a cluster

For the cluster to work correctly and to allow incoming connections, create rules for incoming and outgoing traffic and apply them to the cluster:

1. Add the rules for incoming traffic that allow connecting to the master on ports 6443 and 443. This will allow you to access the Kubernetes API and manage the cluster using kubectl and other utilities.

   Create two rules for incoming traffic, one per port:

   • Port range: 443, 6443.
   • Protocol: TCP.
   • Source: CIDR.
   • CIDR blocks: Specify the IP address range of the subnets you will manage the cluster from, e.g.:
     • 85.23.23.22/32: For an external network.
     • 192.168.0.0/24: For an internal network.

2. Add rules for outgoing traffic that allow:

   • Traffic between the master and metric-server pods:

     • Port range: 4443.
     • Protocol: TCP.
     • Destination name: CIDR.
     • CIDR blocks: Specify the cluster CIDR, e.g., 10.96.0.0/16.

   • Master host connection to NTP servers for time syncing:

     • Port range: 123.
     • Protocol: UDP.
     • Destination name: CIDR.
     • CIDR blocks: 0.0.0.0/0.

Creating a rule for connecting to services from the internetCreating a rule for connecting to services from the internet

To ensure the services running on nodes are accessible from the internet and subnets within Yandex Cloud, create a rule for incoming traffic and apply it to the node group:

• Port range: 30000-32767.
• Protocol: TCP.
• Source: CIDR.
• CIDR blocks: 0.0.0.0/0.

Creating a rule for connecting to nodes over SSHCreating a rule for connecting to nodes over SSH

To access nodes over SSH, create a rule for incoming traffic and apply it to the node group:

• Port range: 22.
• Protocol: TCP.
• Source: CIDR.
• CIDR blocks: IP address ranges of subnets within Yandex Cloud and public IP addresses of computers on the internet, e.g.:
  • 10.0.0.0/8
  • 192.168.0.0/16
  • 172.16.0.0/12
  • 85.32.32.22/32

Applying security groups with rulesApplying security groups with rules

Depending on the rules the security groups contain, these groups must be associated with a cluster or node group:

Examples of rulesExamples of rules

Security groups for a Managed Service for Kubernetes clusterSecurity groups for a Managed Service for Kubernetes cluster

Let’s assume you want to create a Kubernetes cluster which:

• Uses the following configuration:

  • Highly available master in the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.
  • Node group named worker-nodes-a in the ru-central1-a availability zone.
  • Cluster CIDR: 10.96.0.0/16; CIDR of services: 10.112.0.0/16.

• Follows these traffic exchange policies:

  • Health checks of nodes using ICMP messages are allowed from the following subnets within Yandex Cloud:

    • 10.0.0.0/8
    • 172.16.0.0/12
    • 192.168.0.0/16

  • Connections to services from the internet are allowed from any addresses: 0.0.0.0/0.

  • Connections to nodes over SSH are allowed from the only address: 85.32.32.22/32.

  • Access to the Kubernetes API is allowed from the only subnet: 203.0.113.0/24.

To create such a cluster:

Security groups for an L7 Application Load Balancer and Managed Service for Kubernetes clusterSecurity groups for an L7 Application Load Balancer and Managed Service for Kubernetes cluster

Let's assume you need to create rules for the following conditions:

• You need to deploy a load balancer with a public IP address accepting HTTPS traffic in three subnets. These subnets' CIDR blocks are 10.128.0.0/24, 10.129.0.0/24, and 10.130.0.0/24.
• The cluster CIDR block is 10.96.0.0/16 and the service CIDR block is 10.112.0.0/16. These were specified when creating the cluster.
• The cluster's node group resides in the subnet with the 10.140.0.0/24 CIDR block.
• SSH connections to nodes and cluster management via the API, kubectl and other utilities are restricted to 203.0.113.0/24.

locals {
  network-id = "<cloud_network_ID>"
}

resource "yandex_vpc_security_group" "alb" {
  description = "Load balancer security group"
  name        = "alb"
  network_id  = local.network-id
  ingress {
    description    = "Rule to allow incoming HTTP traffic"
    port           = 80
    protocol       = "TCP"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description    = "Rule to allow incoming HTTPS traffic"
    port           = 443
    protocol       = "TCP"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    description       = "Rule for load balancer node health checks"
    port              = 30080
    protocol          = "TCP"
    predefined_target = "loadbalancer_healthchecks"
  }
  egress {
    description    = "Rule for sending traffic to nodes, including health checks"
    from_port      = 0
    to_port        = 65535
    protocol       = "TCP"
    v4_cidr_blocks = ["10.140.0.0/24"]
  }
}

resource "yandex_vpc_security_group" "nodegroup-backend" {
  description = "Node group's security group for backend health checks"
  name        = "nodegroup-backend"
  network_id  = local.network-id
  ingress {
    description    = "Backend health check rule"
    port           = 10501
    protocol       = "TCP"
    v4_cidr_blocks = ["10.128.0.0/24", "10.129.0.0/24", "10.130.0.0/24"]
  }
}

resource "yandex_vpc_security_group" "k8s-cluster-nodegroup-traffic" {
  description = "Cluster and node group's security group for service traffic"
  name        = "k8s-cluster-nodegroup-traffic"
  network_id  = local.network-id
  ingress {
    description       = "Network load balancer rule"
    from_port         = 0
    to_port           = 65535
    protocol          = "TCP"
    predefined_target = "loadbalancer_healthchecks"
  }
  ingress {
    description       = "Rule for traffic between the master and nodes"
    from_port         = 0
    to_port           = 65535
    protocol          = "ANY"
    predefined_target = "self_security_group"
  }
  ingress {
    description    = "Rule for health checks of nodes from subnets within Yandex Cloud"
    from_port      = 0
    to_port        = 65535
    protocol       = "IPV6_ICMP"
    v4_cidr_blocks = ["10.0.0.0/8", "192.168.0.0/16", "172.16.0.0/12"]
  }
  egress {
    description       = "Rule for traffic between the master and nodes"
    from_port         = 0
    to_port           = 65535
    protocol          = "ANY"
    predefined_target = "self_security_group"
  }
}

resource "yandex_vpc_security_group" "nodegroup-traffic" {
  description = "Node group's security group for service traffic"
  name        = "nodegroup-traffic"
  network_id  = local.network-id
  ingress {
    description    = "Rule for traffic between pods and services"
    from_port      = 0
    to_port        = 65535
    protocol       = "ANY"
    v4_cidr_blocks = ["10.96.0.0/16", "10.112.0.0/16"]
  }
  egress {
    description    = "Rule for external resource access"
    from_port      = 0
    to_port        = 65535
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "yandex_vpc_security_group" "k8s-cluster-traffic" {
  description = "Cluster security group for control plane traffic"
  name        = "k8s-cluster-traffic"
  network_id  = local.network-id
  ingress {
    description    = "Rule for access to the Kubernetes API"
    port           = 443
    protocol       = "TCP"
    v4_cidr_blocks = ["203.0.113.0/24"]
  }
  ingress {
    description    = "Rule for access to the Kubernetes API"
    port           = 6443
    protocol       = "TCP"
    v4_cidr_blocks = ["203.0.113.0/24"]
  }
  egress {
    description    = "Rule for traffic between the master and metric-server pods"
    port           = 4443
    protocol       = "TCP"
    v4_cidr_blocks = ["10.96.0.0/16"]
  }
}

resource "yandex_vpc_security_group" "nodegroup-services-access" {
  description = "Node group's security group for accessing services from the internet"
  name        = "nodegroup-services-access"
  network_id  = local.network-id
  ingress {
    description    = "Rules for accessing services from the internet and Yandex Cloud subnets"
    from_port      = 30000
    to_port        = 32767
    protocol       = "TCP"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "yandex_vpc_security_group" "nodegroup-ssh" {
  description = "Node group's security group for SSH access to nodes"
  name        = "nodegroup-ssh"
  network_id  = local.network-id
  ingress {
    description    = "Rules for SSH access to nodes"
    port           = 22
    protocol       = "TCP"
    v4_cidr_blocks = ["203.0.113.0/24"]
  }
}