5. Security Guidance¶
5.1. Introduction¶
Securing Kubernetes requires several layers of security features to provide end to end security for cloud native applications. It is recommended that:
Security testing is fully integrated into the CI/CD pipelines of all parties (e.g., vendors and operators).
Automated security policies are used to flag builds with issues.
Image registries are monitored to automatically block or replace images with known vulnerabilities, while also ensuring policies are used to gate what can be deployed and who can deploy from the registry.
Adopt a layered packaging model which supports separation of concerns during image build.
The following functionalities are recommended for securing Kubernetes platforms:
Image Certification (Scan for vulnerabilities) and Signing
Role-base Access Control
Secret Management
How to overcome the lack of hard Kubernetes Cluster Multi-tenancy
Tenants without hard multi-tenancy requirements (multiple development teams in the same organisation) separated from each other by Namespaces
For strict multi tenancy, a dedicated Kubernetes Cluster per tenant should be used
Integration with other security ecosystem like monitoring and alerting tools
5.1.1. Security Perimeters¶
When applications or workloads run on Kubernetes, there are several layers which come into picture that govern the security. Each of these layers needs to be secured within their perimeters. The various layers that come into picture are:
Container Registry: A container registry is a repository to manage container **images. The access to container registry needs to be secured in order to **prevent unauthorised access or image tampering.
Container Images: Stored instance of a container that holds a set of software needed to run an application. Before loading them to container registry, they need to be secured by performing various checks like vulnerability analysis, scans etc. These images should also be signed from trusted sources.
Containers: A lightweight and portable executable image that contains software and all of its dependencies. The containers need to be prevented from accessing the underlying OS like loading of kernel modules, mounting of directories of underlying OS etc and ensuring that they don’t run in privileged mode.
Pods: A Pod represents a set of running containers on your Cluster. Kubernetes inherently offers pod security policies that define a set of conditions that a pod needs to run with in order to be accepted into the system. These policies help in ensuring the necessary checks for running the pods.
Kubernetes Node: A Kubernetes node in an unsecured boundary can lead to a potential threat to the running workloads. Such a node should be hardened by disabling unused ports, prohibiting root access etc.
Kubernetes Master: A master node in an unsecured boundary can lead to a potential threat to the running workloads. A master may be hardened in terms of security by disabling unused ports, prohibiting root access etc.
Kubernetes Control Plane: The container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. The communication over these APIs needs to be secured via different mechanisms like TLS encryption, API authentication via LDAP etc.
5.2. Principles¶
The following are core principles to consider when securing cloud native applications:
Deploy only secure applications and trusted codes
Only deploy applications from validated and verified images
Only deploy applications from trusted registries
Containers orchestration (Kubernetes) secured with administrative boundaries between tenants
Use Namespaces to establish security boundaries between tenants
Create and define Cluster network policies
Run a Cluster-wide pod security policy
Turn on Audit Logging
Separate sensitive workloads using Namespaces
Secure tenant metadata Access
Segregate container networks using security zoning and network standards
Harden the Host OS running the containers
Use container-aware runtime defence tools
Enable Role-Based Access Control (RBAC)
5.3. Node Hardening¶
5.3.1. Node hardening: Securing Kubernetes hosts¶
When an operating system or application is installed, it comes with default settings. Usually, all ports are open, and all application services are turned on. In other words, freshly installed assets are highly insecure.
Ensure Kubernetes nodes are secure, hardened and configured correctly following well known security frameworks. Security benchmarks, for example, CIS benchmarks are a set of configuration standards and best practices designed to help ‘harden’ the security of their digital assets.
5.3.2. Restrict direct access to nodes¶
Restrict root/administrative access to Kubernetes nodes while avoiding direct access to nodes for operational activities including debugging, troubleshooting, and other tasks.
5.3.3. Vulnerability assessment¶
Vulnerability assessments are a crucial part of IT risk management lifecycles. The mitigation of the vulnerabilities helps in protecting systems and data from unauthorised access and breaches. Implement necessary vulnerability scanner tools (e.g., OpenVAS and many other opensource and commercial tools) to identify threats and flaws within the infrastructure that represents potential risks.
5.3.4. Patch management¶
Patch management is another key aspect of IT risk management lifecycle for security, compliance, uptime, feature enhancements, etc. Implement necessary patch management to ensure your environment is not susceptible to exploitation.
5.4. Securing Kubernetes orchestrator¶
The communication over the Kubernetes control plane APIs needs to be secured via different mechanisms like TLS encryption, API authentication via LDAP etc. A control plane node in an unsecured boundary can lead to a potential threat to the running workloads. It is recommended that a control plane node is hardened in terms of security by disabling unused ports, prohibiting root access etc.
They following are security recommendations for orchestration manager:
Cluster management Network isolation can help protect the master node and control where administrative commands can run. Use network isolation techniques, configure RBAC on the Cluster manager and configure node service accounts following the principle of least privilege.
Ensure that access control is applied to registries requiring unique credentials, to limit who can control the build or add images.
Network access runs over TLS connections.
User roles and access levels are configured to provide segregation of duties.
Do not mix container and non-containers services on the same node
Do not run containers as root
Multi-factor authentication is used for all administrative access.
Harden the configuration by using CIS (Center for Internet Security) benchmarks, which are available for container runtime and Kubernetes
Deploy security products that provide whitelisting, behaviour monitoring and anomaly detection for preventing malicious activity
Avoid privileged container application through policy management to reduce the effects of potential attacks.
Enable integration with other security ecosystem (SIEM)
Isolate environments (Dev /test /Production) from other environments within the Cluster.
Create administrative boundaries between resources using Namespace and avoid using default Namespaces.
Enable Seccomp to ensure that the workloads have restricted actions available within the container application.
Limit discovery by restricting services and users that can access Cluster management metadata on configuration, containers and nodes
5.4.1. Control network access to sensitive ports¶
Kubernetes clusters usually listen on a range of well-defined and distinctive ports which makes it easy to identify the clusters and attack them. Hence, it is highly recommended to configure authentication and authorisation on the cluster and cluster nodes.
The Kubernetes documentation specifies the default ports used in Kubernetes. Make sure that your network blocks access to unnecessary ports and consider limiting access to the Kubernetes API server except from trusted networks.
Master node(s):
Protocol |
Port Range |
Purpose |
|---|---|---|
TCP |
6443 |
Kubernetes API Server |
TCP |
2379-2380 |
etcd server client API |
TCP |
10250 |
Kubelet API |
TCP |
10259 |
kube-scheduler |
TCP |
10257 |
kube-controller-manager |
Worker nodes:
Protocol |
Port Range |
Purpose |
|---|---|---|
TCP |
10250 |
Kubelet API |
TCP |
30000-32767 |
NodePort Services |
5.4.2. Controlling access to the Kubernetes API¶
The Kubernetes platform is controlled using APIs, which are the first items to be secured in order to defend against attackers. Controlling who has access and what actions they are allowed to perform is the primary concern.
5.4.3. Use Transport Layer Security and Service Mesh¶
Communication in the cluster between services should be handled using TLS, encrypting all traffic by default. Kubernetes expects that all API communication in the cluster is encrypted by default with TLS, and the majority of installation methods will allow the necessary certificates to be created and distributed to the cluster components. Note that some components and installation methods may enable local ports over HTTP and administrators should familiarize themselves with the settings of each component to identify potentially unsecured traffic.
Advances in network technology, such as the service mesh, have led to the creation of products like LinkerD and Istio which can enable TLS by default while providing extra telemetry information on transactions between services. The service mesh is a mesh of layer 7 proxies handling service-to-service communications. The service mesh architecture consists of data plane components made up of network proxies paired with each micro-service, and control plane components providing proxies configuration, managing TLS certificates and policies. The two documents, NIST SP 800-204A (Building Secure Microservices-based Applications Using Service-Mesh Architecture) and NIST SP 800-204B (Attribute-based Access Control for Microservices-based Applications Using a Service Mesh) provide guidance to deploy service mesh.
5.4.5. Restrict access to etcd and encrypt contents within etcd¶
etcd is a critical Kubernetes component which stores information on state and secrets, and it should be protected differently from the rest of your cluster. Write access to the API server’s etcd is equivalent to gaining root on the entire cluster, and even read access can be used to escalate privileges fairly easily.
The Kubernetes scheduler will search etcd for pod definitions that do not have a node. It then sends the pods it finds to an available kubelet for scheduling. Validation for submitted pods is performed by the API server before it writes them to etcd, so malicious users writing directly to etcd can bypass many security mechanisms e.g., PodSecurityPolicies.
Administrators should always use strong credentials from the API servers to their etcd server, such as mutual auth via TLS client certificates, and it is often recommended to isolate the etcd servers behind a firewall that only the API servers may access.
5.4.6. Controlling access to the Kubelet¶
Kubelets expose HTTPS endpoints which grant powerful control over the node and containers. By default Kubelets allow unauthenticated access to this API. Production clusters should enable Kubelet authentication and authorization
5.4.7. Securing Kubernetes Dashboard¶
The Kubernetes dashboard is a webapp for managing your cluster. It is not a part of the Kubernetes cluster itself, it has to be installed by the owners of the cluster; a number of tutorials show how to do this. Unfortunately, most of them create a service account with very high privileges. This caused Tesla and some others to be hacked via such a poorly configured Kubernetes dashboard (Reference: Tesla cloud resources are hacked to run cryptocurrency-mining malware).
To prevent attacks via the dashboard, you should follow some best practices:
Do not expose the dashboard without additional authentication to the public. There is no need to access such a powerful tool from outside your LAN
Turn on RBAC, so you can limit the service account the dashboard uses
Review the privileges granted to the service account of the dashboard privileges, and remove disable any additional privileges assigned.
Grant permissions per user, so each user can only access what they are supposed to access
If using network policies, block requests to the dashboard even from internal pods (this will not affect the proxy tunnel via kubectl proxy)
Before version 1.8, the dashboard had a service account with full privileges, so check that there is no role binding for cluster-admin left.
Deploy the dashboard with an authenticating reverse proxy, with multi-factor authentication enabled. This can be done with either embeded OpenID Connect (OIDC) id_tokens or using Kubernetes Impersonation. This allows the use of the dashboard with the user’s credentials instead of using a privileged ServiceAccount. This method can be used on both on-prem and managed cloud clusters.
5.5. Use Namespaces to Establish Security Boundaries¶
Namespaces in Kubernetes is the first level of isolation between components. It is easier to apply security controls (Network Policies, Pod policies, etc.) to different types of workloads when deployed in separate Namespaces.
5.6. Separate Sensitive Workload¶
To limit the potential impact of a compromise, it is recommended to run sensitive workloads on a dedicated set of nodes. This approach reduces the risk of a sensitive application being accessed through a less-secure application that shares a container runtime or host.
The separation can be achieved by using node pools and Kubernetes Namespaces.
5.7. Create and Define Network Policies¶
Network Policies allow Kubernetes managers to control network access into and out of the cloud native applications. It is recommended to have a well defined ingress and egress policy for cloud native applications. It is also important to modify the default network policies, such as blocking or allowing traffic from other Namespaces or Clusters while ensuring the Namespaces/Clusters are running with policy support enabled.
5.8. Run latest Version¶
As new security features and patches are added in every quarterly update, it is important to take advantage of these fixes and patches.
It is recommended to run the latest release with its most recent patches.
5.9. Secure Platform Metadata¶
Kubernetes metadata contain sensitive information including kubelet admin credentials. It is recommended to secure them using encryption to avoid this being stolen and use to for escalated privileges in the the Cluster.
Limit discovery by restricting services and users that can access Cluster management metadata on configuration, container application, and nodes
Ensure all metadata is encrypted and network access runs over TLS connections
5.10. Enable Logging and Monitoring¶
Logging, monitoring, alerting and log aggregation are essential for Kubernetes. Enable and monitor audit logs for anomalous or unwanted API calls, especially any authorisation failure.
5.11. Run-Time Security¶
The following are recommended best practices for container run-time:
Integrate run-time processes to Security Information and Event Monitoring (SIEM)
Use container-aware run-time defence tools
Ensure all running cloud native applications are from secure and verified images
Cloud native applications are not run with root privileges
Ensure sensitive workloads are properly segmented by Namespaces or Cluster to mitigate the scope of compromise.
5.12. Secrets Management¶
It is recommended that the principle of least privilege is applied to secret management in Kubernetes:
Ensure that the cloud native applications can only read the secrets that these applications need
Have different set of secrets for different environments(like production, development, and testing)
Secret values protect sensitive data, it is recommended to protect them from unauthorised access. Ideally, by being protected at rest and in transit. Encryption in transit is achieved by encrypting the traffic between the Kubernetes control-plane components and worker nodes using TLS.
It is recommended that Secrets are not be stored in scripts or code but provided dynamically at runtime as needed. Keep any sensitive data, including SSH keys, API access keys, and database credentials, in a secure data repository such as a key manager or vault. Only pull credentials on demand and over secure channels to ensure sensitive data is not written to disk unprotected. The key manager or vault encryption keys should be backed by a FIPS 140-2 Hardware Security Module. It is also important to implement the following:
Check there are no hard-coded passwords, keys, and other sensitive items in the container application.
Where possible use security tools to automate scanning for hard-coded passwords, keys, and other sensitive items in the container application
5.13. Trusted Registry¶
Ensure that the container registry only accepts container images from trusted sources that have tested and validated the images. Where images are provided by third parties, define and follow a formal process to validate compliance with security requirements. Also ensure that access control is applied to registries requiring unique credentials, to limit who can control the build or add images.
It is strongly recommended that network access to the registry is secured using TLS, SSL or VPN connections to ensure trust.
Ensure container applications are validated to assess their use and applicability as well as scanned for viruses and vulnerabilities. Only deploy container application from images that are signed with a trusted key
Ensure the latest certified container application is always selected by versioning images
Trusted repository and registry services should reject containers that are not properly signed
Use approved registries for images loaded into production
Where possible, use third-party products to validate container content both before and after deployment
Ensure stale images are removed from the registry. Remove unsafe, vulnerable images (e.g. containers should no longer be used based on time triggers and labels associated with images).
5.14. Isolation¶
5.14.1. VM vs. Container Isolation¶
Sometimes container isolation is compared directly with VM based isolation, with the conclusion “there are issues with container isolation, it is not as good as VM isolation”. Such 1:1 comparison is not reasonable because VM and container based isolation are fundamentally different:
VMs: hard isolation, in the layers underlying the application SW
Containers: isolation by SW based mechanisms available in OS, Docker and Kubernetes. A container workload is just a set of Linux processes. It is possible to configure SW based additional isolation for container workloads, for example by kernel namespaces.
The primary isolation mechanism in Kubernetes environment should be VM or physical machine based. This implies that multiple cloud native applications should not be deployed together in the same Kubernetes Cluster - unless these applications have been planned and verified to co-exist. Thus, the default is to allocate one Namespace per Cloud Native Network Function (CNF).
5.14.2. Container Isolation in Kubernetes Cluster¶
5.14.2.1. Namespaces¶
Kubernetes Namespaces should be used to provide resource isolation within a Kubernetes Cluster. They should not be used to isolate different steps in the deployment process like Development, Production, or Testing. The most reliable separation is achieved by deploying sensitive workloads into dedicated Clusters.