Zero Trust Security for Kubernetes with a Service Mesh

Earlier this year, The White House issued an Executive Order on Improving the Nation’s Cyber Security, which laid the groundwork for creating a zero trust architecture for federal agencies. The National Security Agency (NSA) also recently published a Kubernetes hardening guide that highlights best practices for key zero trust security principles in Kubernetes.

These documents emphasize the need for organizations and developers to rethink the way they secure their applications and infrastructure. Given how security needs are changing with the adoption of cloud infrastructure, it is imperative to understand how to apply these principles across your run times, clouds, platforms and specifically Kubernetes clusters.

One key approach the NSA guide recommends is to use a service mesh in Kubernetes environments to authorize, authenticate and encrypt service-to-service communications. Let’s take a closer look at how an open source service mesh like HashiCorp Consul on Kubernetes can help strengthen your security posture and promote zero trust networking.

»Zero Trust Security

Before we jump into how a zero trust approach applies to Kubernetes, we need some context on zero trust principles and why they’re becoming more important.

HashiCorp Co-Founder and CTO Armon Dadgar explains zero trust security in a whiteboard session.

Securing infrastructure, data and access becomes increasingly complex and difficult across multiple clouds and on-premises data centers. As organizations move to multi-cloud and hybrid infrastructures, the measures they took to secure their private data centers start to become obsolete. IP-based identity and perimeter-based access are no longer relevant in a world of ephemeral IP addresses and a constantly changing — and often remote — workforce that needs constant access to shared resources.

This shift requires a different approach to security, one that trusts nothing — not even your own services and users — but instead authenticates and authorizes everything before allowing access. Critically, the move to zero trust is not binary; it’s an ongoing approach that requires a fundamental shift to your architecture. Fortunately, these three best-practice zero trust principles can help point the way:

1. Deliver identity-based service-to-service access and communication: Services should be based on service identity. Service identity, not IP addresses, should be used for authorization. And services should mutually authenticate their identities when establishing connections.
2. Include secret and certificate management and hardened Kubernetes encryption: Secrets should be encrypted, time bound and able to work with a global service identity that enables data encryption in transit. Access credentials should be time-bound, requiring the user or application to refresh their credentials at defined intervals.
3. Enable observability with audits and logging: All access attempts should be audited and logged.

»Applying Zero Trust Principles to Kubernetes

Now that we have established that network perimeter based security isn’t enough in dynamic environments, we need less reliance on network perimeter controls. What steps can organizations take to implement zero trust principles in Kubernetes environments?

»Deliver Identity-Based Service-to-Service Access and Communication

Every Kubernetes cluster provides a flat network where every container or service can talk to another without any restrictions. Kubernetes considers the container network or applications running on it as trusted, not requiring authentication. For example, if a database service and logging service are running on the same Kubernetes cluster, by default they can access each other at a network level.

Users can create policies in Kubernetes to apply default rules to deny both ingress and egress traffic in the cluster. However, even with those restrictions, you still need service-to-service authentication and authorization to ensure that only required resources are made available to services. You can solve this problem by introducing a service mesh, which allows you to assign service identities to each service running on the Kubernetes cluster. Based on service identity, the mesh can authenticate service identities using mTLS, and service-access requests can be authorized or blocked using intentions, which allow operators to define service-to-service communication permissions by service name. With a service mesh in place, the logging service can access the database service only if both services can mutually authenticate their credentials.

»Include Secret and Certificate Management and Hardened Kubernetes Encryption

Using credentials for the Kubernetes control plane, either for identity or for managing secrets, expands the attack surface, is hard to maintain and does not follow the principles listed above. Kubernetes secrets have several weaknesses in a zero trust security architecture:

• By default, secrets are base-64 encoded, not encrypted.
• Since secrets do not expire (they’re not time-bound) they can expose you to risk.
• Kubernetes can manage resources, such as secrets, only within a cluster boundary. If you have sets of clusters, resources used in multiple clusters must be managed separately.

A service mesh with a secrets broker can address this challenge. Consul on Kubernetes, for example, has an integration with HashiCorp Vault — a centralized secrets management solution that closes the gaps in Kubernetes secrets. The goal is to ensure that secrets are encrypted at rest with centralized access control and auditing. The workflow should support both single Kubernetes clusters and federated multi-cluster deployments. In addition, certificate auto-rotation can help operators reduce the time-to-live (TTL) values of their TLS certificates, strengthening their security posture.

»Enable Observability with Audits and Logging

To improve Kubernetes security, It’s important to understand what’s happening inside the cluster to see what service-access requests were made. Audit logs let the audit team inspect event data to see which credentials have been used, what actions have taken place, and the timestamps associated with these transactions. This provides greater insight and accountability for security teams.

A service mesh deploys sidecar proxies capable of emitting metrics with each of the services running in the cluster and keeps the records for all the service-to-service communication and access requests. Ideally, the service mesh would integrate with open source monitoring tools such as Prometheus and Grafana to make it easier to analyze service-networking patterns and bolster security.

»Conclusion

As service deployment environments get more complex with multiple Kubernetes clusters, multiple runtimes, multi-cloud and on-premises deployments, and all of their various interconnections, following zero trust security principles becomes a necessity.

A service mesh like HashiCorp Consul can be an essential part of this process, providing a management layer that enforces zero trust principles by providing identity-based service networking that mutually authenticates and authorizes any access. Consul makes it easy to enforce granular security policies in Kubernetes and across multiple environments without having to code all the security parameters into the application itself, which results in a fast, easy-to-manage onramp to a more zero trust security posture.

A version of this blog post was originally published on The New Stack.