9. Security Kubernetes
Introduction
Yandex Managed Service for Kubernetes provides an environment for managing containerized applications in Yandex Cloud infrastructure. Deploy, scale, and manage applications in containers using Kubernetes.
The user is responsible for all actions made inside the Kubernetes node. The user is responsible for the security of the nodes and their proper setup in accordance with PCI DSS requirements and other security standards.
Yandex Cloud is responsible for the Kubernetes API security.
The user is responsible for correctly choosing security settings in Managed Service for Kubernetes, including selecting the channel and the update schedule.
9.1 The use of sensitive data is limited
To comply with PCI DSS or other security standards when using Managed Service for Kubernetes, do not:
- Use sensitive data in names and descriptions of clusters, node groups, namespaces, services, and pods.
- Use sensitive data in Kubernetes node labels and Yandex Cloud service resource labels.
- Use sensitive data in pod manifests.
- Use sensitive data in etcd in clear text.
- Write sensitive data to Managed Service for Kubernetes logs.
- Make sure that names and descriptions of clusters, node groups, namespaces, services, and pods contain no sensitive data.
- Check configuration files for critical data.
- In the management console
, select the folder where your Managed Service for Kubernetes instance is located. - In the list of services, select Managed Service for Kubernetes.
- Make sure that Kubernetes node labels and Yandex Cloud service resource labels contain no sensitive data.
Guides and solutions to use:
Manually edit the names or contents of manifests and other configuration files if they use sensitive data.
9.2 Resources are isolated from each other
Wherever possible, ensure maximum isolation between resources:
- Use a separate organization for each "big" project.
- Use a separate cloud for each development team.
- Use a separate Kubernetes cluster located in a separate folder for each service.
- Use a separate namespace for each microservice.
- Your clouds must have no shared resources. Cloud members must have access only to their clouds.
Less strict isolation models are also possible, e.g., where:
- Projects are deployed in different clouds.
- Development teams use independent folders.
- Services have separate Kubernetes clusters.
- Microservices use different namespaces.
Wherever possible, ensure maximum isolation between resources.
9.3 No access to the Kubernetes API and node groups from untrusted networks
We do not recommend granting access to the Kubernetes API and node groups from non-trusted networks, e.g., from the internet. Use firewall protection where needed (for example, security groups).
Guides and solutions to use:
-
Use network policy configuration tools via the Calico (basic) or Cilium CNI (advanced) plugins in Yandex Cloud. By default, apply the
default deny
rules for incoming and outgoing traffic with only the relevant traffic allowed. -
For online endpoints, allocate an independent Kubernetes cluster or independent node groups (using such mechanisms as Taints and Tolerations
+ Node affinity ). By doing this, you establish a DMZ so that if your nodes are compromised online, your attack surface is small. -
To enable incoming network access to your workloads via HTTP/HTTPS, use the Ingress
resource. There are at least two Ingress controller options that you can use in Yandex Cloud:
9.4 Authentication and access management configured in Managed Service for Kubernetes
For the Kubernetes cluster to run, you need two service accounts: the service account of the cluster and the service account of the node group. The access of IAM accounts to Managed Service for Kubernetes resources is managed at the following levels:
- Managed Service for Kubernetes service roles (access to the Yandex Cloud API): These enable you to control clusters and node groups (for example, create a cluster, create/edit/delete a node group, and so on).
- Service roles to access the Kubernetes API: These let you control cluster resources via the Kubernetes API (for example, perform standard actions with Kubernetes: create, delete, view namespaces, work with pods, deployments, create roles, and so on). Only the basic global roles at the cluster level are available:
k8s.cluster-api.cluster-admin
,k8s.cluster-api.editor
, andk8s.cluster-api.viewer
. - Primitive roles: These are global primitive IAM roles that include service roles (e.g., the primitive
admin
role includes both the service administration role and the administrative role to access the Kubernetes API). - Standard Kubernetes roles: Inside the Kubernetes cluster itself, you can use Kubernetes tools to create both regular roles and cluster roles. Thus you can manage access for IAM accounts at the namespace level. To assign IAM roles at the namespace level, you can manually create RoleBinding objects in a relevant namespace stating the cloud user's IAM ID in the subjects name field.
Make sure the above recommendations are met.
9.5 Managed Service for Kubernetes uses a safe configuration
In Managed Service for Kubernetes, the user is fully in control of all node group settings, but only partially in control of the master settings. The user is responsible for the whole cluster's security.
The CIS Kubernetes Benchmark
- Using the kube-bench
tool, check whether the node group configuration is compliant with CIS Kubernetes Benchmark. The tool officially supports the Yandex Cloud node groups. - Starboard Operator
is a free tool that helps you automate scanning of images for vulnerabilities and checking that the configuration is compliant with CIS Kubernetes Benchmark. Starboard Operator supports integration with kube-bench and is used for its automatic startup.
9.6 Data encryption and Managed Service for Kubernetes secret management are done in ESO as a Service format
At the Kubernetes etcd level, encrypt secrets using an in-built mechanism from Yandex Cloud.
We recommend that you use SecretManager solutions to work with Kubernetes secrets. Yandex Lockbox is such a solution in Yandex Cloud.
Yandex Lockbox was integrated with Kubernetes using the External Secrets
The most secure recommended option for encrypting secrets is ESO as a Service (External Secrets Operator as a service). When using ESO, the global administrator has access to the namespace where ESO is installed, and administrators of individual namespaces create their own SecretStore
Guides and solutions to use:
- Guide on working with External Secrets and Yandex Lockbox from the project description
. - Guide on working with External Secrets and Yandex Lockbox from the Yandex Cloud documentation.
9.7 Docker images are stored in a Container Registry registry configured for regular image scanning
To ensure effective security, we recommend using Container Registry to store Docker images to be deployed in Managed Service for Kubernetes. This allows you to quickly respond to new vulnerabilities in images using built-in recurrent vulnerability scanning.
You should perform vulnerability scanning at least once a week. This will help you detect and eliminate vulnerabilities in images in a timely manner, which significantly reduces risks of unauthorized access to your resources and increases security level of your infrastructure.
Using Container Registry to store images will also provide centralized image version control for simpler updates and security management.
- In the management console
, select the folder the registry with Docker images belongs to. - Select the appropriate registry in Container Registry.
- Go to the Vulnerability scanner tab and click Edit settings.
- Make sure that scheduled Docker image scans are enabled with a frequency of at least once a week.
Guides and solutions to use:
9.8 One of the three latest Kubernetes versions is used with update monitoring
For Kubernetes, both automatic and manual updates are available for clusters and node groups. You can request a manual update of the Kubernetes cluster or its nodes to the latest supported version at any time. Manual updates bypass any configured maintenance windows and maintenance exceptions. Kubernetes issues updates in a regular manner. To meet the Information Security standards:
- Select a relevant update channel and enable either automatic installation of updates, or manual installation immediately after publication in the selected channel.
- Double-check that the ad settings meet the Information Security standards.
- Use one of the three latest Kubernetes versions, because updates (including security updates) are only released for these versions.
To get a list of available versions for a Kubernetes cluster:
- Go to the folder page and select Managed Service for Kubernetes.
- Click the name of the Kubernetes cluster.
- Click Edit in the top-right corner.
- View the list of available versions in the Kubernetes version field under Master configuration.
To get a list of available versions for a Kubernetes node group:
- Go to the folder page and select Managed Service for Kubernetes.
- Click the name of the Kubernetes cluster you need and go to the Nodes manager tab.
- Select the Kubernetes node group from the list and click Edit in the top-right corner.
- Get a list of available versions in the Kubernetes version field.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To get a list of available versions, run the following command:
yc managed-kubernetes list-versions
To get a list of available versions, use the list.
Guides and solutions to use:
9.9 Backup is configured
To ensure continuous operation and data protection, we recommend using backups in Managed Service for Kubernetes. With backups, you can quickly recover the service without experiencing any data or time loss in the wake of a malfunction or accident. Data in Kubernetes clusters is securely stored and replicated within the Yandex Cloud infrastructure. However, you can back up data from Kubernetes cluster node groups at any time and store them in Yandex Object Storage or other types of storage.
Make sure that you have backups of your data from node groups of Kubernetes clusters.
Guides and solutions to use:
You can create backups of Kubernetes cluster node group data using the Velero
When working with Velero that is installed manually, you can use nfs
9.10 Check lists are used for secure creation and use of Docker images
Practices for secure creation and use of Docker images must be in place for protection against potential vulnerabilities, malware, and unauthorized access to data. They ensure image integrity and security compliance while also preventing potential threats coming from its use in the infrastructure. Use these check lists to meet requirements for secure creation of images:
You can control Dockerfile in your CI/CD pipeline using the Conftest
When using minimal images or distroless images without a shell, ephemeral containers
Make sure you have the check lists in place to meet the requirements for secure creation of images.
9.11 The Kubernetes security policy is used
The requirements listed in Kubernetes Pod Security Standards
These requirements allow you to ensure security and reliability of applications in a Kubernetes cluster. To achieve compliance, you can either use the built-in Kubernetes tool called Pod Security Admission Controller
Verify compliance with the Kubernetes Pod Security Standard.
Guides and solutions to use:
-
You can also use the following tools within CI/CD to control compliance with Pod Security Standards:
- Kyverno CLI
- The gator CLI
- Kyverno CLI
9.12 Audit log collection is set up for incident investigation
Events available to the user in the Managed Service for Kubernetes service can be classified as levels:
- Kubernetes API events (Kubernetes audit logging)
- Kubernetes node events
- Kubernetes pod events
- Kubernetes metrics
- Kubernetes flow logs
For more information about setting up audit event logging at various levels, see Collecting, monitoring, and analyzing Managed Service for Kubernetes audit logs.
In Managed Service for Kubernetes, you can audit the current role model used in the service. To do this, open the Kubernetes cluster page in the management console
You can also use:
Make sure that audit logs are being collected.