Deploy and Configure Kustomize and MongoDB for Kubernetes Stateful Applications

Part 2: Let’s Kustomize, install the MongoDB operator, and Ondat in our CI/CD pipeline

This is the second article in a series. The first one is available here, where we present the tools allowing us to create a Kubernetes-friendly and automated environment for developing a web application based on Flask, MongoDB, Pymongo, and the Marvel API. In this second part, we deploy and configure Kustomize, MongoDB, and Ondat.

Install and Configure Kustomize

As mentioned in the previous article, there are 2 ways to use Kustomize. You can natively use kubectl -k or install the binary. As for the latter, you can use the following command that automatically detects your OS and install Kustomize:

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

However, this script doesn’t work for ARM-based architectures. Alternatively, navigate to the release page, pick the correct binary for your distribution, and move it into your $PATH.

To test that Kustomize is working as expected, you can download the application manifests from https://github.com/vfiftyfive/CFD12-Demo-Manifests and generate the dev manifests. For this, run the following commands:

$ git clone https://github.com/vfiftyfive/CFD12-Demo-Manifests && cd CFD12-Demo-Manifests
Cloning into 'CFD12-Demo-Manifests'...
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (164/164), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 164 (delta 80), reused 136 (delta 52), pack-reused 0
Receiving objects: 100% (164/164), 15.02 KiB | 3.75 MiB/s, done.
Resolving deltas: 100% (80/80), done.

$ kustomize build overlay/dev 
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ondat
parameters:
  csi.storage.k8s.io/controller-expand-secret-name: csi-controller-expand-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
  csi.storage.k8s.io/controller-publish-secret-name: csi-controller-publish-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-publish-secret-name: csi-node-publish-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
  csi.storage.k8s.io/provisioner-secret-name: csi-provisioner-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  storageos.com/replicas: "1"
provisioner: csi.storageos.com...

We truncated the output, but the command produces all the manifests required to deploy the application into the dev cluster. You can also notice there’s an additional file named kustomization.yaml within the base and dev folders. These files are required for Kustomize to know which manifests to render and how. The customization file located in the base folder contains the following code:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ondat_sc.yaml
- mongo_sts.yaml
- marvel_deploy.yaml
- job.yaml
- marvel_svc.yaml

Once Kustomize looks into the base folder, it applies customization to all its children YAML resources described in that file under resources.

The dev folder also contains a kustomization.yaml file, which details the specific changes needed to render the final version of the manifests.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patchesStrategicMerge:
- ondat_sc.yaml
- marvel_deploy.yaml
- mongodbcommunity_cr.yaml
- job.yaml
resources:
- ../../base
configMapGenerator:
- name: mongo-config
  literals:
  - MONGO_SEED0=mongodb-0.mongodb.default.svc.cluster.local
  - MONGO_SEED1=mongodb-1.mongodb.default.svc.cluster.local
  - MONGO_SEED2=mongodb-2.mongodb.default.svc.cluster.local
  - OFFSET=600
  - MONGO_USERNAME=admin
secretGenerator:
- name: admin-password
  literals:
  - password=mongo
configurations:
- name_reference.yaml

Let’s take a look at the different sections:

• pacthesStrategicMerge lists the files Kustomize should look at when building the target manifests. These files are located under the dev folder and describe the amendments applied to the manifests located in the base folder. These files are accessible within the git repo, and here is a summary of the changes they describe:
• ondat.sc.yaml: Create 1 replica for all volumes using that StorageClass, and enable encryption.
• marvel_deploy.yaml: Set the number of replicas to 2, inject the MongoDB password from the Kustomize Secretgenerator, and add the environment variables from the configMap generator.
• mongodbcommunity_cr.yaml: Set the MongoDB version to 5.0.5, set up the admin user, configure the Kubernetes StatefulSet with a volumeClaimTemplate using the Ondat StorageClass, and set the data and logs volume size.
• job.yaml: Inject the MongoDB password from the Kustomize Secretgenerator and environment variables from the configMap generator.
  
  

In the remaining sections, we have:

• resources , which defines the location of the base manifests.
• configMapGenerator , which generates a configMap with a unique name every time Kustomize runs. You can use literals or files to define your variables.
• secretGenerator , which generates Secrets with a unique name every time Kustomize runs. You can use literals or files to define your secrets.
• configurations , which specifies custom objects path that Kustomize uses to modify or insert particular values. For example, in our scenario, Kustomize dynamically configures the reference to a secret in the MongoDB custom resource. As it is a custom resource, Kustomize doesn’t know where to find the equivalent of a secret name parameter. Instead, we define it in the file name_reference.yaml:
  
  

nameReference:
- kind: Secret
  fieldSpecs:
  - kind: MongoDBCommunity
    path: spec/users/passwordSecretRef/name

As a result, a unique Secret name composed of the initial Secret name and a random string will be added at this location every time Kustomize is invoked. Kustomize will also replace all secrets within native objects YAML definition in relevant resources. Those are the resources you have defined in the kustomization.yaml file.

Install and Configure the MongoDB Operator

For this article, we have used the MongoDB Community Operator since it is free and easy to use. However, for production environments, we recommend using an Enterprise version of the Operator or being ready to support it yourself and get help from the community. There are multiple options available, such as the MongoDB Enterprise Operator, the Percona Operator, or the KubeDB Operator.

Understanding the Operator

Some of you may not be familiar with Kubernetes Operators or why we need one to install a database cluster altogether. So let’s focus a little bit on this aspect. A Kubernetes Operator is fundamentally made up of 2 parts:

• A piece of code provided as a container that continuously monitors specific objects in the Kubernetes API and performs actions as a result of this active monitoring. It is called a custom controller.
• The monitored custom resources. A custom resource is a Kubernetes object that is not native. The custom resource schema is defined within the Custom Resource Definiton (CRD), provided as a YAML file by the user, and sent to the Kubernetes API. Every object created with that schema is further saved in the Kubernetes etcd store. You can compare the CRD to a class in OOP and the custom resource to an instance of that class. It extends the existing Kubernetes API.
  
  

The Kubernetes Operator’s job is to perform automated actions and interact with the cluster or systems outside the cluster like a human operator. It is not limited to deploying components only. It can perform CRUD operations in reaction to Kubernetes events, which are dynamic by nature. It utilizes events data as input to workflows. A common use case for Operators is to automate the deployment of software solutions within Kubernetes. So in the case of a database, the Kubernetes Operator will install the database (cluster or standalone) once the corresponding database resource has been ingested by the Kubernetes API, typically in the form of a YAML manifest that describes the configuration of the database as a custom resource.

Also, remember the database is deployed as a StatefulSet. When you scale the StatefulSet, Kubernetes doesn’t automagically scale the database cluster. It just deploys new containers with the database image. There is some extra work needed to configure it. It is performed by the Kubernetes Operator. The same is true when you scale down the StatefulSet.

In the case of the MongoDB Operator, the added CRD produces an object of kind mongodbcommunity . The custom resource encapsulates all the information required to deploy and maintain a MongoDB database. First, you need to install the Operator and create the custom resource. It can be achieved during the automated deployment of your Kubernetes cluster or manually once the cluster has been installed. You can find all the required files in this repo. We’re going to perform a couple of steps to install the Operator:

• Install the CRD
• Create the Operator configuration manifest. You can refer to the MongoDB Community Operator documentation and examples. You can also find the configuration we have used in the git repo. The Operator configuration manifest is manager.yaml.

You need to configure the scope of the Operator first. It defines which namespaces the Operator monitors: the Namespace where the Operator is installed, a specific namespace, or all namespaces. In our example, the manager.yaml file has the following configuration:

env:
- name: WATCH_NAMESPACE
  value: "*"

It tells the Operator to monitor all namespaces for MongoDB custom resources operations.

• Create cluster-wide roles and role-bindings. The ClusterRoleBindingservice account namespace must be modified with the name of the namespace you want to use (in bold in the text below). It is located in the file clusterwide/role_binding.yaml



kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mongodb-kubernetes-operator
subjects:
- kind: ServiceAccount
  namespace: mongo-operator
  name: mongodb-kubernetes-operator
roleRef:
  kind: ClusterRole
  name: mongodb-kubernetes-operator
  apiGroup: rbac.authorization.k8s.io

• For each namespace that you wish the Operator to watch, you need to deploy a Role, RoleBindingand ServiceAccount in that namespace
• The last step is creating and deploying the database configuration manifest (the custom resource), which is detailed later.
  
  

Deploy the Operator

Execute the following commands to install and configure the Operator:

#Clone the git repository
$ git clone https://github.com/vfiftyfive/mongodb-community-operator-manifests && cd mongodb-community-operator-manifests
Cloning into 'mongodb-community-operator-manifests'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 17 (delta 5), reused 15 (delta 3), pack-reused 0
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (5/5), done.

#Install the CRD
$ kubectl apply -f mongo-crd.yaml

#Deploy the clusterwide RBAC resources
$ kubectl apply -f clusterwide/
clusterrole.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
clusterrolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created

#Create a namespace for the operator
$ kubectl create ns mongo-operator
namespace/mongo-operator created

#Deploy namespace RBAC resources in the operator namespace.
$ kubectl apply -k rbac/ -n mongo-operator
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created

#Deploy namespace RBAC resources in the default namespace (where the app will be deployed)
$ kubectl apply -k rbac/ 
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created

#Deploy the Operator
$ k apply -f manager.yaml -n mongo-operator
deployment.apps/mongodb-kubernetes-operator created

#Check the Operator has correctly been deployed
$ kubectl get po -n mongo-operator
NAME                                           READY   STATUS ...
mongodb-kubernetes-operator-6d46dd4b74-ztx9c   1/1     Running ...

The Operator is now ready to deploy a new MongoDB database as soon as it detects that a new custom resource has been added into the Kubernetes API.

The next step is to install the distributed storage layer, Ondat (formerly StorageOS).

Install Ondat

Ondat provides a data-mesh acting as a distributed persistent storage layer that provides premium features that are not included by default in Kubernetes. This includes replication, encryption, performance optimization, intelligent volume placement, etc, all managed as part of native Kubernetes labels and annotations.

It can be deployed with a single line through a kubectl plugin (or alternatively using a helm chart available here):

kubectl storageos install --include-etcd

But first, let’s install the plugin by executing the following command:

curl -sSLo kubectl-storageos.tar.gz \
    https://github.com/storageos/kubectl-storageos/releases/download/v1.0.0/kubectl-storageos_1.0.0_linux_amd64.tar.gz \
    && tar -xf kubectl-storageos.tar.gz \
    && chmod +x kubectl-storageos \
    && sudo mv kubectl-storageos /usr/local/bin/ \
    && rm kubectl-storageos.tar.gz

As there are some prereqs for your Kubernetes hosts’ kernel modules, you should first run the preflight checks to see if your cluster is compatible by running the following command:

kubectl storageos preflight

(Quick tip: if you are using GKE, use the ubuntu_containerd image, it already includes the linux-modules-extra-x.y.z package)

The command will output a comprehensive report like the one below. The screenshot depicts an example with no NVMe drives present in our 3-node cluster. You can safely ignore this warning if you’re not running performance-intensive workloads.

kubeclt storageos install --include-etcd

$ kubectl get pods -n storageos | grep node                                                                                                                                                 
storageos-node-8chxx                     3/3     Running          
storageos-node-gsdzt                     3/3     Running          
storageos-node-sq2ds                     3/3     Running