Policy-as-code and GitOps for Stateful Applications

Policy-as-code and GitOps for Stateful Applications


On the 24th of March, we hosted our first DoK London (Data on Kubernetes) community event in person! It has been a real bliss to revert back to that model! It reminded me how much I enjoy sharing experiences with the community and feeling emotions when interacting with fellow human beings in the same room, finally! I have to say, the extra bonus was that it helped me lower my level of anxiety a little bit. Given the current situation in the world, between a pandemic turning into an endemic and a war at the door of Europe, it’s not an easy task to focus on work and tech updates on a daily basis. For me, engaging in face-to-face conversations with peers is a good way to be back on track and focus on an ecosystem I can influence or just zero in on projects and activities I can eventually control (at least to some extent 😅 ).

We used the same format as the first event (this one was virtual though), with a hands-on lab followed by a panel discussion with experts in their domains. In the first part, we delved into a combination of GitOps and Policy-as-Code, or in the case of Kubernetes, I prefer to call it Policy-as-YAML. I picked these very topics for a couple of reasons, but first, let’s talk about the background.

If you are new to Kubernetes, but not to Software Development or Cloud technologies, you have certainly worked with some kind of Version Control System. And if you have been long enough in this industry, Git or Github is certainly not the first you have used, but maybe CVS (not the pharmacy) or Apache Subversion (back in the 90’s and 2000’s, respectively). More recently, DevOps practices have converged into using these tools as a single source of truth, not only for the code, but also to store workflows that describe how to build the infrastructure and platform components required to transform the code into a final artifact. That opinionated version of the application can then be used, shipped, or stored in various formats and destination platforms, such as cloud hyperscalers. Cloud providers commonly provide version control as-a-service and it is easy to leverage their pipelines to build your application.

However, there’s an ultimate level of automation that allows you to manage the lifecycle of your containerized applications in the safest and most scalable way, using GitOps principles. People often confuse GitOps with Infrastructure-as-Code. Obviously using a version control system is the foundation of GitOps, but it is the tree hiding the forest. GitOps relies on the following extra features:

  • The software repository must contain Kubernetes manifests. They are used to deploy the application artifacts to destination Kubernetes clusters.
  • Applications are deployed and updated in the cluster using a reconciliation loop that is monitoring remote repositories. Upon new changes, the watch process makes sure they are detected and that synchronization with the current state of the cluster is triggered. Platform admins don’t use kubectl to make changes to applications in the cluster. The reconciliation loop is usually provided by a GitOps Continuous Delivery (CD) solution, making use of Kubernetes Customer Resource Definitions (CRD) and the Operator pattern.

If GitOps principles are then applied to the lifecycle management of stateful applications, a couple of other parameters need to be taken into account. Stateful applications are typically critical applications such as databases or message queues. They need extra care and are often managed using Operators, which leverage Custom Resources, resources that are not native to Kubernetes as they use API extensions. By design, in Kubernetes, there is no standard to express stateful applications properties such as the size of the database, user names and associated permissions, backup schedules, etc. Consequently, you may experience sprawl of objects with unpredictable structures. In turn, this entropy leads to more complexity when it comes to deploying control mechanisms to enforce compliance and company-wide guidelines for these critical applications. Because business-critical applications are by definition what makes the business profitable, additional constraints and guardrails must be applied to guarantee service continuity. Without such policies, catastrophic situations can occur, not only leading to service disruption but also eventual loss of data!

So what can we do to mitigate this issue? Well, enforce these policies before artifacts get deployed into the production cluster. First, make sure to shift your data left. This will enable early testing under similar conditions as production, reducing the number of bugs and issues that can arise after application rollout. Second, enrich your GitOps CD pipeline with a CI pipeline that makes use of a policy engine. Preferably, the solution should not introduce a new language, but rather build on the existing Kubernetes and Cloud-Native ecosystem. A good example is Kyverno, a policy engine specifically designed for Kubernetes, that provides both in-cluster mutation and validation webhook capabilities, as well as a CLI tool to detect policy drift as part of a more traditional CI pipeline. Policies are expressed using YAML and can be decorated with various expressions and pattern-matching formats. Version control systems can then be used to store and audit policies in the same way application artifacts are managed in the context of GitOps. You can see more details on how this can be achieved by watching the recording below. You can also follow along with the Instruqt lab available here.

Historically, Kubernetes has been labeled as the defacto cloud operating system for orchestrating stateless workloads. Over time, it organically added features to meet the requirements of stateful applications. Kubernetes adoption will be substantially boosted the instant enterprises consider the platform as a safe option to run their business-critical applications. This moment is happening right now: we already know that Kubernetes provides the right level of distribution and availability to run containerized applications at scale. But compliance and distributed storage capabilities are not addressed by default, especially when it comes to stateful applications. Kyverno and Ondat are good examples of additional tools that allow your stateful workloads to call Kubernetes home! We cover all these aspects in the lab, so don’t hesitate to share it with others and spread the love for stateful applications running in Kubernetes!


written by:
Nic Vermandé
Nic is an experienced hands-on technologist, evangelist and product owner who has been working in the fields of Cloud-Native technologies, Open Source Software, Virtualization and Datacenter networking for the past 17 years. Passionate about enabling users and building cool tech solving real-life problems, you’ll often see him speaking at global tech conferences and online events, spreading the word and walking the walk with customers.

Register for our SaaS Platform

Learn how Ondat can help you scale persistent workloads on Kubernetes.