Security Kubernetes

The section includes recommendations for Yandex Cloud users on security settings in Yandex Managed Service for Kubernetes.

ResponsibilityResponsibility

The user is responsible for all actions made inside the Kubernetes node. The user is responsible for the security of the nodes and their proper setup in accordance with PCI DSS requirements and other security standards.

Yandex Cloud is responsible for the Kubernetes API security.

The user is responsible for correctly choosing security settings in Managed Service for Kubernetes, including selecting the channel and the update schedule.

Sensitive dataSensitive data

To comply with PCI DSS or other security standards when using Managed Service for Kubernetes, do not:

• Use sensitive data in names and descriptions of clusters, node groups, namespaces, services, and pods.
• Use sensitive data in Kubernetes node labels and Yandex Cloud service resource labels.
• Use sensitive data in pod manifests.
• Use sensitive data in etcd in clear text.
• Write sensitive data to Managed Service for Kubernetes logs.

Resource modelResource model

Wherever possible, ensure maximum isolation between resources:

• Use a separate organization for each "large-scale" project.
• Use a separate cloud for each development team.
• Use a separate Kubernetes cluster located in a separate folder for each service.
• Use a separate namespace for each microservice.

Your clouds must have no shared resources. Cloud members must have access only to their clouds.

Less strong isolation models are also possible, for example:

• Projects are split between different clouds.
• Development teams are assigned independent folders.
• Services have separate Kubernetes clusters.
• Microservices have independent namespaces.

Managed Service for Kubernetes network securityManaged Service for Kubernetes network security

We do not recommend granting access to the Kubernetes API and node groups from non-trusted networks, e.g., from the internet.
Use firewall protection when needed (for example, security groups). In the section below, you can find links to instructions on how to set up firewall protection in security groups.

SegmentationSegmentation

Cloud levelCloud level

Restrict network access to the Kubernetes API (master) and node groups using instructions for security groups.

When using an ALB as an Ingress Gateway, also complete the following steps:

1. Apply the security group to the ALB.

2. Additionally, apply the security group to the node group:

   • Source type: <security group applied to the ALB>.
   • Destination type: node group.
   • Port range: 30000-32767.

Kubernetes levelKubernetes level

Restrict network access inside Kubernetes using the Network Policy.

You can use two network plugins in Yandex Cloud:

• Calico: a basic plugin.
• Cilium CNI: an advanced plugin that uses advanced network policies applied at the L7 layer (REST/HTTP, gRPC and Kafka).

We recommend using the default deny rules for incoming and outgoing traffic by default with only the relevant traffic allowed.

To generate policies, you can use the Cilium CNI built-in Hubble platform to analyze the traffic manually. Various solutions for automatic generation of network policies are also available on the market.

For useful examples of network policies, see the repo.

A helpful tool to create both basic and advanced network policies is available here.

Setting up incoming network accessSetting up incoming network access

For online endpoints, we recommend allocating an independent Kubernetes cluster or independent node groups (using such mechanisms as: Taints and Tolerations + Node affinity). By doing this, you establish a DMZ so that if your nodes are compromised online, your attack surface is small.

To enable incoming network access to your workloads via HTTP/HTTPS, use the Ingress resource.

There exist at least two variants of an Ingress controller that you can use in Yandex Cloud:

• NGINX Ingress Controller.
• Application Load Balancer Ingress controller.

Benefits of Application Load Balancer Ingress controller:

• Integration with the Yandex Certificate Manager cloud service.
• No need to install a controller to the cluster because everything is deployed on the Application Load Balancer side.

Restricting access to the metadata of VMs in the node groupRestricting access to the metadata of VMs in the node group

For all pods, create a network policy to block network traffic to port 169.254.169.254 or use the default-deny policy from the example. The policy must block workload node group metadata access because these node groups contain sensitive data, such as the token of the service account assigned to the node.

Managed Service for Kubernetes authentication and access managementManaged Service for Kubernetes authentication and access management

The access of IAM accounts to Managed Service for Kubernetes resources is managed at the following levels:

• Managed Service for Kubernetes service roles (access to the Yandex Cloud API): These enable you to control clusters and node groups (for example, create a cluster, create/edit/delete a node group, and so on).

• Service roles to access the Kubernetes API: These let you control cluster resources via the Kubernetes API (for example, perform standard actions with Kubernetes: create, delete, view namespaces, work with pods, deployments, create roles, and so on). Only the basic global roles are available at cluster level: k8s.cluster-api.cluster-admin, k8s.cluster-api.editor, and k8s.cluster-api.viewer.

• Primitive roles: These are global primitive IAM roles that include service roles (for example, the primitive role admin includes both the service administration role and the administrative role to access the Kubernetes API).

• Standard Kubernetes roles: Inside the Kubernetes cluster itself, you can use Kubernetes tools to create both regular roles and cluster roles. Thus you can control IAM accounts in terms of access at the namespace level. To assign IAM roles at the namespace level, you can manually create RoleBinding objects in a relevant namespace stating the cloud user's IAM ID in the subjects name field. Examples:

  apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
  name: iam-user-aje0jndkhkvu04ek #object name RoleBinding
  namespace: micro1-ns
  roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
  subjects:
  - kind: User
  name: aje0jndkq855llvu04ek #cloud user ID
  

For the Managed Service for Kubernetes cluster to run, you need two service accounts: the service account of the cluster and the service account of the node group.

Example of setting up role models and policies in Managed Service for Kubernetes.

Managed Service for Kubernetes secure configurationManaged Service for Kubernetes secure configuration

Secure configurationSecure configuration

In Managed Service for Kubernetes, the user is fully in control of all node group settings, but only partially in control of the master settings. These settings are part of the user's overall cluster security responsibility.

The CIS Kubernetes Benchmark standard is designed to build a secure Kubernetes configuration, including node configurations.

In Yandex Cloud, the Kubernetes node groups are deployed by default with the configuration that complies with CIS Kubernetes Benchmark.

The kube-bench tool enables you to check whether the node group configuration is compliant with CIS Kubernetes Benchmark. The tool officially supports the Yandex Cloud node groups.

Here you can see the examples of launching kube-bench on the nodes.

In addition, kube-bench supports integration with Starboard Operator — another product for kube-bench automatic launching.

Starboard Operator is a free tool that helps you automate scanning of images for vulnerabilities and checking that the configuration complies with CIS Kubernetes Benchmark.

Integration between Starboard and Yandex Container Registry to scan running images

Integrity control (FIM — File integrity monitoring)Integrity control (FIM — File integrity monitoring)

You must control two levels of file integrity in node groups:

• Node OS files, e.g., configuration files.
• Container files, e.g., critical files the user application writes to the volume.

Node OS filesNode OS files

You can use, for example, Osquery as an agent installed on the nodes using DaemonSet and uses specific node folders mounted as a volume into the DaemonSet container (redirected file system).

A comprehensive solution in Osquery and kubequery in K8s.

Container filesContainer files

One of the methods to solve this task:

1. Use readOnlyRootFilesystem in pods.
2. Make sure to mount folders to write the data to as separate volumes: as emptydir or individual disks.

If you mount folders as emptydir, files are stored on the node in the /var/lib/kubelet/pods/PODUID/volumes/kubernetes.ioempty-dir/VOLUMENAME folder. To ensure data integrity, you can monitor this folder by Osquery as node OS files.

In the case of separate disks (not emptydir), you can mount volumes in read mode to the above-mentioned DaemonSet running Osquery.

To control file integrity on the Kubernetes nodes, you can also use the tools listed in Integrity control.

There exist dedicated free solutions for Kubernetes nodes from Google or Argus, including file-integrity-operator.

Data encryption and Managed Service for Kubernetes secret managementData encryption and Managed Service for Kubernetes secret management

At the Kubernetes etcd level, encrypt secrets using an in-built mechanism from Yandex Cloud.

We recommend that you use SecretManager solutions to work with Kubernetes secrets. Yandex Lockbox is such a solution in Yandex Cloud.

Yandex Lockbox was integrated with Kubernetes using the External Secrets open-source project. In Cloud Marketplace, the solution is available in the basic simplified scenario: External Secrets Operator with Yandex Lockbox support.

Useful instructions on working with External Secrets:

• Instructions to work with External Secrets and Yandex Lockbox from the project description.
• Instructions to work with External Secrets and Yandex Lockbox from the Yandex Cloud documentation.

Many methods to differentiate access to secrets using this tool have been described.

The most secure recommended option for encrypting secrets is ESO as a Service (External Secrets Operator as a service). In this case, the global administrator has access to the namespace where ESO is installed, and administrators of individual namespaces create their own SecretStore objects (where they specify IAM authorized access keys for their Lockbox secrets). If this SecretStore object is compromised, the authorized key of only one namespace will be compromised (not all of them, as in the case of Shared ClusterSecretStore).

Encryption in transitEncryption in transit

For in-transit encryption, use TLS interaction between pods. If TLS cannot be used, use the service mesh solutions:

• Istio
• Linkerd

Encryption at restEncryption at rest

If you need to encrypt your stored data, you can use:

• Key Management Service, for encrypting data at the application level (including when you use persistent volumes).
• A custom method of data encryption. However, in this case, protection of the keys and of the key management procedure is the sole responsibility of the user.

Protection against malicious code in Managed Service for KubernetesProtection against malicious code in Managed Service for Kubernetes

There are two levels where you can enable malicious code protection in Kubernetes:

• Container Registry level protection.
• OS-level protection of Kubernetes nodes.

Security scanner in Container Registry.

To protect the containerization host levels, you can use a variety of paid and free solutions from the "Runtime security" and "Antivirus engine" classes. Examples of free solutions:

• Kubernetes ClamAV
• Sysdig Falco (it can also function as an Intrusion Detection System)

Be sure to also use the Kubernetes built-in support for AppArmor and Seccomp.

Analyzing Kubernetes security logs in ELK: audit logs, Policy Engine, Falco.

Managing Managed Service for Kubernetes vulnerabilitiesManaging Managed Service for Kubernetes vulnerabilities

Yandex Cloud within Managed Service for Kubernetes is in charge of vulnerability management and security updates on the master. The user must independently control vulnerabilities on the Kubernetes worker nodes.

Scanning for vulnerabilitiesScanning for vulnerabilities

You can break vulnerability scanning into the following levels:

• Image-level vulnerability scanning.
• Vulnerability scanning of Kubernetes node OSes.

Vulnerability scanning at the image level is detailed in Protection against malicious code in Managed Service for Kubernetes.

Examples of free universal solutions for node OS vulnerability scanning in Kubernetes are given in 9. Security Kubernetes.

There are also paid and free specialized solutions for scanning Kubernetes node OSes and Kubernetes hosts for vulnerabilities, e.g., the free kube-hunter and trivi (scan filesystem).

Security updatesSecurity updates

Managed Service for Kubernetes issues updates in a regular manner. To meet the Information Security standards:

• Select a relevant update channel and enable either automatic installation of updates, or manual installation immediately after publication in the selected channel.
• Double-check that the ad settings meet the Information Security standards.
• Use one of the three latest Kubernetes versions, because updates (including security updates) are only released for these versions.

Backup and recoveryBackup and recovery

Set up backups in Managed Service for Kubernetes by following the guide. When storing your backups in Object Storage, follow recommendations from the Secure configuration for Object Storage.

Security policies in KubernetesSecurity policies in Kubernetes

Requirements listed in Pod Security Standards from Kubernetes let you prevent threats related to Kubernetes objects.

To implement the requirements, you can either use the Kubernetes built-in Pod Security Admission Controller tool or open-source software, e.g., other Admission Controllers: OPA Gatekeeper, Kyverno.

Examples using Kyverno:

• Analyzing Kubernetes security logs in ELK: audit logs, Policy Engine, Falco.
• Example of setting up role models and policies in Managed Service for Kubernetes.

To control compliance with Pod Security Standards, you can also use the following tools within CI/CD:

• Kyverno CLI
• The gator CLI

Or a separate Kubesec tool.

Practices for securely creating and using Docker imagesPractices for securely creating and using Docker images

Use these check lists to meet requirements for secure creation of images:

• Dockerfile best practices
• Kubernetes Security Checklist and Requirements

You can control Dockerfile in your CI/CD pipeline using the Conftest utility.

Runtime protectionRuntime protection

When using minimal images or distroless images without a shell, use ephemeral containers.

Solution running Osquery.

Load sharing between nodesLoad sharing between nodes

Data loads with different security contexts (such as different severities of data processed) must be processed on different Kubernetes nodes. To enable load sharing within a cluster, use different node groups with different settings for node labels and node taints. Be sure to combine both settings.

Collecting, monitoring, and analyzing Managed Service for Kubernetes audit logsCollecting, monitoring, and analyzing Managed Service for Kubernetes audit logs

Events available to the user in the Managed Service for Kubernetes service can be classified as levels:

• Kubernetes API events (Kubernetes Audit logging)
• Kubernetes node events
• Kubernetes pod events
• Kubernetes metrics
• Kubernetes flow logs

Kubernetes API level (Kubernetes audit logging)Kubernetes API level (Kubernetes audit logging)

Audit events are collected from the Kubernetes API level by Cloud Logging.

Analyzing Kubernetes security logs in ELK: audit logs, Policy Engine, Falco.

Kubernetes node levelKubernetes node level

Kubernetes node level events are collected and exported similarly to collecting OS audit logs.

Kubernetes pod levelKubernetes pod level

To learn more about options for collecting and exporting pod-level events in Kubernetes, refer to the Kubernetes official documentation.

Examples of collecting and exporting pod logs:

• Exporting logs to Cloud Logging using Fluent Bit is described in the Managed Service for Kubernetes documentation.
• Exporting pod logs into Elastic or Splunk is described in the Yandex Cloud Security Solution Library.

The Filebeat plugin for transferring logs to Elastic and Fluent Bit with a Cloud Logging plugin is available in Cloud Marketplace.

Kubernetes metricsKubernetes metrics

Monitoring includes a set of metrics to analyze availability of Kubernetes objects and their behavioral anomalies.

For a guide on exporting Monitoring metrics, see the Exporting events to SIEM section.

Kubernetes flow logsKubernetes flow logs

Exporting flow logs to Yandex Object Storage.

Managed Service for Kubernetes role model auditManaged Service for Kubernetes role model audit

In the Managed Service for Kubernetes console, you can audit the current role model used in the service. For this, go to the Access management tab in the service.

You can also use:

• KubiScan
• Krane