9. Security Kubernetes

IntroductionIntroduction

Yandex Managed Service for Kubernetes provides an environment for managing containerized applications in Yandex Cloud infrastructure. Deploy, scale, and manage applications in containers using Kubernetes.

The user is responsible for all actions made inside the Kubernetes node. The user is responsible for the security of the nodes and their proper setup in accordance with PCI DSS requirements and other security standards.

Yandex Cloud is responsible for the Kubernetes API security.

The user is responsible for correctly choosing security settings in Managed Service for Kubernetes, including selecting the channel and the update schedule.

9.1 The use of sensitive data is limited9.1 The use of sensitive data is limited

To comply with PCI DSS or other security standards when using Managed Service for Kubernetes, do not:

• Use sensitive data in names and descriptions of clusters, node groups, namespaces, services, and pods.
• Use sensitive data in Kubernetes node labels and Yandex Cloud service resource labels.
• Use sensitive data in pod manifests.
• Use sensitive data in etcd in clear text.
• Write sensitive data to Managed Service for Kubernetes logs.

Guides and solutions to use:

Manually edit the names or contents of manifests and other configuration files if they use sensitive data.

9.2 Resources are isolated from each other9.2 Resources are isolated from each other

Wherever possible, ensure maximum isolation between resources:

• Use a separate organization for each "big" project.
• Use a separate cloud for each development team.
• Use a separate Kubernetes cluster located in a separate folder for each service.
• Use a separate namespace for each microservice.
• Your clouds must have no shared resources. Cloud members must have access only to their clouds.

Less strict isolation models are also possible, e.g., where:

• Projects are deployed in different clouds.
• Development teams use independent folders.
• Services have separate Kubernetes clusters.
• Microservices use different namespaces.

9.3 No access to the Kubernetes API and node groups from untrusted networks9.3 No access to the Kubernetes API and node groups from untrusted networks

We do not recommend granting access to the Kubernetes API and node groups from non-trusted networks, e.g., from the internet. Use firewall protection where needed (for example, security groups).

Guides and solutions to use:

• Guide on creating a cluster without internet access.

• Security group setup guide.

• Use network policy configuration tools via the Calico (basic) or Cilium CNI (advanced) plugins in Yandex Cloud. By default, apply the default deny rules for incoming and outgoing traffic with only the relevant traffic allowed.

• For online endpoints, allocate an independent Kubernetes cluster or independent node groups (using such mechanisms as Taints and Tolerations + Node affinity). By doing this, you establish a DMZ so that if your nodes are compromised online, your attack surface is small.

• To enable incoming network access to your workloads via HTTP/HTTPS, use the Ingress resource. There are at least two Ingress controller options that you can use in Yandex Cloud:

  • NGINX Ingress Controller.
  • Application Load Balancer of an Ingress controller.

9.4 Authentication and access management configured in Managed Service for Kubernetes9.4 Authentication and access management configured in Managed Service for Kubernetes

For the Kubernetes cluster to run, you need two service accounts: the service account of the cluster and the service account of the node group. The access of IAM accounts to Managed Service for Kubernetes resources is managed at the following levels:

• Managed Service for Kubernetes service roles (access to the Yandex Cloud API): These enable you to control clusters and node groups (for example, create a cluster, create/edit/delete a node group, and so on).
• Service roles to access the Kubernetes API: These let you control cluster resources via the Kubernetes API (for example, perform standard actions with Kubernetes: create, delete, view namespaces, work with pods, deployments, create roles, and so on). Only the basic global roles at the cluster level are available: k8s.cluster-api.cluster-admin, k8s.cluster-api.editor, and k8s.cluster-api.viewer.
• Primitive roles: These are global primitive IAM roles that include service roles (e.g., the primitive admin role includes both the service administration role and the administrative role to access the Kubernetes API).
• Standard Kubernetes roles: Inside the Kubernetes cluster itself, you can use Kubernetes tools to create both regular roles and cluster roles. Thus you can manage access for IAM accounts at the namespace level. To assign IAM roles at the namespace level, you can manually create RoleBinding objects in a relevant namespace stating the cloud user's IAM ID in the subjects name field.

9.5 Managed Service for Kubernetes uses a safe configuration9.5 Managed Service for Kubernetes uses a safe configuration

In Managed Service for Kubernetes, the user is fully in control of all node group settings, but only partially in control of the master settings. The user is responsible for the whole cluster's security.

The CIS Kubernetes Benchmark standard is designed to build a secure Kubernetes configuration, including node configurations. In Yandex Cloud, the Kubernetes node groups are deployed by default with the configuration that complies with CIS Kubernetes Benchmark.

9.6 Data encryption and Managed Service for Kubernetes secret management are done in ESO as a Service format9.6 Data encryption and Managed Service for Kubernetes secret management are done in ESO as a Service format

At the Kubernetes etcd level, encrypt secrets using an in-built mechanism from Yandex Cloud.

We recommend that you use SecretManager solutions to work with Kubernetes secrets. Yandex Lockbox is such a solution in Yandex Cloud.

Yandex Lockbox was integrated with Kubernetes using the External Secrets open-source project. In Cloud Marketplace, the solution is available in the basic simplified scenario: External Secrets Operator with Yandex Lockbox support.

The most secure recommended option for encrypting secrets is ESO as a Service (External Secrets Operator as a service). When using ESO, the global administrator has access to the namespace where ESO is installed, and administrators of individual namespaces create their own SecretStore objects (where they specify IAM-authorized access keys for their Lockbox secrets). If this SecretStore object is compromised, only the authorized key of one specific namespace is compromised (rather than all of them, as in the case of Shared ClusterSecretStore).

Guides and solutions to use:

• Guide on working with External Secrets and Yandex Lockbox from the project description.
• Guide on working with External Secrets and Yandex Lockbox from the Yandex Cloud documentation.

9.7 Docker images are stored in a Container Registry registry configured for regular image scanning9.7 Docker images are stored in a Container Registry registry configured for regular image scanning

To ensure effective security, we recommend using Container Registry to store Docker images to be deployed in Managed Service for Kubernetes. This allows you to quickly respond to new vulnerabilities in images using built-in recurrent vulnerability scanning.

You should perform vulnerability scanning at least once a week. This will help you detect and eliminate vulnerabilities in images in a timely manner, which significantly reduces risks of unauthorized access to your resources and increases security level of your infrastructure.

Using Container Registry to store images will also provide centralized image version control for simpler updates and security management.

Guides and solutions to use:

• Guide on scheduled scanning of Docker images.

9.8 One of the three latest Kubernetes versions is used with update monitoring9.8 One of the three latest Kubernetes versions is used with update monitoring

For Kubernetes, both automatic and manual updates are available for clusters and node groups. You can request a manual update of the Kubernetes cluster or its nodes to the latest supported version at any time. Manual updates bypass any configured maintenance windows and maintenance exceptions. Kubernetes issues updates in a regular manner. To meet the Information Security standards:

• Select a relevant update channel and enable either automatic installation of updates, or manual installation immediately after publication in the selected channel.
• Double-check that the ad settings meet the Information Security standards.
• Use one of the three latest Kubernetes versions, because updates (including security updates) are only released for these versions.

yc managed-kubernetes list-versions

Guides and solutions to use:

• Guide on how to update a cluster automatically.
• Guide on how to update a cluster manually.

9.9 Backup is configured9.9 Backup is configured

To ensure continuous operation and data protection, we recommend using backups in Managed Service for Kubernetes. With backups, you can quickly recover the service without experiencing any data or time loss in the wake of a malfunction or accident. Data in Kubernetes clusters is securely stored and replicated within the Yandex Cloud infrastructure. However, you can back up data from Kubernetes cluster node groups at any time and store them in Yandex Object Storage or other types of storage.

Guides and solutions to use:

You can create backups of Kubernetes cluster node group data using the Velero tool. It supports working with Yandex Cloud disks using the Kubernetes CSI driver and helps create snapshots of disks and volumes.

When working with Velero that is installed manually, you can use nfs, emptyDir, local, or any other type of volumes without built-in support for snapshots. To use such a volume type, install Velero with the restic plugin. Velero installed from Cloud Marketplace does not include the restic plugin.

• Guide on Kubernetes cluster backup in Object Storage.

9.10 Check lists are used for secure creation and use of Docker images9.10 Check lists are used for secure creation and use of Docker images

Practices for secure creation and use of Docker images must be in place for protection against potential vulnerabilities, malware, and unauthorized access to data. They ensure image integrity and security compliance while also preventing potential threats coming from its use in the infrastructure. Use these check lists to meet requirements for secure creation of images:

• Dockerfile best practices.
• Kubernetes Security Checklist and Requirements.

You can control Dockerfile in your CI/CD pipeline using the Conftest utility.

When using minimal images or distroless images without a shell, ephemeral containers are recommended.

9.11 The Kubernetes security policy is used9.11 The Kubernetes security policy is used

The requirements listed in Kubernetes Pod Security Standards allow you to prevent threats related to Kubernetes objects, such as unauthorized access to confidential data and execution of malicious code.

These requirements allow you to ensure security and reliability of applications in a Kubernetes cluster. To achieve compliance, you can either use the built-in Kubernetes tool called Pod Security Admission Controller or open-source software, e.g., OPA Gatekeeper or Kyverno.

Guides and solutions to use:

• You can also use the following tools within CI/CD to control compliance with Pod Security Standards:

  • Kyverno CLI
  • The gator CLI

• Kubesec

9.12 Audit log collection is set up for incident investigation9.12 Audit log collection is set up for incident investigation

Events available to the user in the Managed Service for Kubernetes service can be classified as levels:

• Kubernetes API events (Kubernetes audit logging)
• Kubernetes node events
• Kubernetes pod events
• Kubernetes metrics
• Kubernetes flow logs

For more information about setting up audit event logging at various levels, see Collecting, monitoring, and analyzing Managed Service for Kubernetes audit logs.

In Managed Service for Kubernetes, you can audit the current role model used in the service. To do this, open the Kubernetes cluster page in the management console, and go to the Access management tab.

You can also use:

• KubiScan.
• Krane.
• Yandex Audit Trails events.