How to keep a secret secret within Kubernetes?



While there are significant efforts to improve Kubernetes component layers, the state of Secret Management is not receiving much interest. Using etcd to store API object definition & states, Kubernetes secrets are encoded in Base64 and shipped into the key-value store database. Even if the filesystems on which etcd runs are encrypted, the secrets are still not.

To solve this unsecure design, both commercial and open source solutions exist leveraging different approaches to handle secrets calling for a different set of tools or practices instead of leveraging the native Kubernetes way to manage secrets. This leads to training and maintaining for a set of niche skills and tools increasing the cost and complexity of Kubernetes day 2 operations.

With Trousseau, an open-source project sponsored by Ondat*, any user/workload can leverage the native Kubernetes way to store and access secrets in a safe way by using the API of any Key Management Service (KMS) providers through the Kubernetes KMS plugin framework.*

Deploying a complex application using containers on Kubernetes is easier than ever compare to any previous experience using bare metal or virtual machines within on-premises or cloud deployment. However, this statement is only true for Development but not so much when reaching Acceptance and Production due to security policies to comply with for business-critical applications.

One of the major challenges is authentication, a very large topic that includes networking, storage, identity management, and secret management. The latter, Secret management, provides Kubernetes with end-to-end handling of credentials between containers and containers with outside services like a front-end accessing a database, either running or not on Kubernetes. Obviously, no one wants the production database credentials to be exposed to the external world.

From a design perspective, the state of Secret Management is not receiving much interest. The Kubernetes project team created and maintains a Secret Management framework storing any type in the local etcd running on the control plane, an insecure base64 encoded approach which, despite having at best both TLS enabled and an at-rest encrypted filesystem, will not protect our precious credentials. Let's explore how to solve the Kubernetes secret management together.

The Kubernetes anti-patterns

To solve this insecure design, both open-source and commercial solutions exist. Some are more intrusive requiring a full deployment of Key Management Services, others are more experimental relying on Git to store the actual encrypted secrets, both require a new set of practices leading to an additional gap within the organization's skill matrix.

While some of the solutions might fit better an Application team and another the Infrastructure team, none will reach consensus. In fact, Application teams might be much more open to exploring new pathways like Sealed Secret using Git repositories to improve the CI/CD processes while Infrastructure teams would rather leverage an existing corporate Key Management Service like CyberARK or Hashicorp Vault optimizing their existing investment.

One of the major downsides of using the Application team's non-native Kubernetes pattern is that it will require training, practice, and support both the Development, Application, Infrastructure, and Security teams on a set of niche tools, skills, and unknown practices. Presented like that, major red flags are flying high!

Leveraging existing investment would be the most welcome path by any company from a skill and cost-efficiency standpoint. Sadly, there is a major downside of using such enterprise options due to the unique opportunity to upsell the container piece as a separate subscription or licensing for Kubernetes zeroing the cost efficiency benefit.

The Kubernetes patterns

Interestingly enough, while attempting to address the challenges of Secret Management for Kubernetes, almost none of these solutions are actually leveraging the Kubernetes Secret Management framework designed to address the application needs out of simple and native APIs.

It is well known that secrets encoded in base64 within etcd are not safe! The reason why there is a native Kubernetes pattern addressing this specific need within the Secret Management framework; the Kubernetes KMS provider plugin for data encryption. Without the need to introduce new practices, tooling, skills, and investment, the KMS encryption provider takes care of encrypting the data within etcd using an envelope encryption scheme with a data encryption key generated for each encryption and stored within a remote KMS.

Secret or not, but don't clear text!

When it comes to business-critical applications, no organization can settle on security. In the meantime, the go-to-market defines the leaders of a market and the speed of innovation is a crucial factor for success. Using native Kubernetes API and framework provides a simple, controllable, and auditable approach within the need to invest in an extra layer of tooling and skills.

With Trousseau, an open-source project sponsored by Ondat, any user/workload can leverage the native Kubernetes way to store and access secrets in a safe way by using the API of any Key Management Service (KMS) providers through the Kubernetes KMS plugin framework.

Join the Trousseau effort today by testing, contributing, and even sponsoring the project's effort to provide a Kubernetes native secure secret management!

Explore Trousseau with a Hands-in lab:
Keep Your Secrets and Persistent Volumes Safe With Trousseau and Vault

written by:
Romuald Vandepoel
Romuald Vandepoel is a technologist, facilitator, advocate, and open source contributor responsible for modernizing and transforming customer and partner organizations at Red Hat, Inc. He embraces his role as a trusted advisor within architecture boards and CTO offices providing guidance on culture, process and technology adoption strategies. As a transformational leader, he supports transitions from IT legacy siloed monoliths to ecosystems of Business-Unit-as-a-Service to accelerate the organization's innovation and growth. Romuald participated as moderator and speaker for both in-person and virtual events for Red Hat, FOSDEM, , DevNetwork, Tech Field Day, and CNCF.  He also maintains and contributes to two open source projects, Trousseau and Discoblocks, and is a member of the Ondat Advisory Board where he provides guidance on open source strategies. Romuald is based both in Eindhoven and San Jose and has held diverse roles in tech companies including Tyco, SonicWALL, NetApp, and Red Hat.

Register for our SaaS Platform

Learn how Ondat can help you scale persistent workloads on Kubernetes.