Part 2: Let’s Kustomize, install the MongoDB operator, and Ondat in our CI/CD pipeline
This is the second article in a series. The first one is available here, where we present the tools allowing us to create a Kubernetes-friendly and automated environment for developing a web application based on Flask, MongoDB, Pymongo, and the Marvel API. In this second part, we deploy and configure Kustomize, MongoDB, and Ondat.
Install and Configure Kustomize
As mentioned in the previous article, there are 2 ways to use Kustomize. You can natively use kubectl -k
or install the binary. As for the latter, you can use the following command that automatically detects your OS and install Kustomize:
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
However, this script doesn’t work for ARM-based architectures. Alternatively, navigate to the release page, pick the correct binary for your distribution, and move it into your $PATH
.
To test that Kustomize is working as expected, you can download the application manifests from https://github.com/vfiftyfive/CFD12-Demo-Manifests and generate the dev manifests. For this, run the following commands:
$ git clone https://github.com/vfiftyfive/CFD12-Demo-Manifests && cd CFD12-Demo-Manifests
Cloning into 'CFD12-Demo-Manifests'...
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (164/164), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 164 (delta 80), reused 136 (delta 52), pack-reused 0
Receiving objects: 100% (164/164), 15.02 KiB | 3.75 MiB/s, done.
Resolving deltas: 100% (80/80), done.
$ kustomize build overlay/dev
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ondat
parameters:
csi.storage.k8s.io/controller-expand-secret-name: csi-controller-expand-secret
csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
csi.storage.k8s.io/controller-publish-secret-name: csi-controller-publish-secret
csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/node-publish-secret-name: csi-node-publish-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
csi.storage.k8s.io/provisioner-secret-name: csi-provisioner-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
storageos.com/replicas: "1"
provisioner: csi.storageos.com...
We truncated the output, but the command produces all the manifests required to deploy the application into the dev cluster. You can also notice there’s an additional file named kustomization.yaml
within the base
and dev
folders. These files are required for Kustomize to know which manifests to render and how. The customization file located in the base
folder contains the following code:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ondat_sc.yaml
- mongo_sts.yaml
- marvel_deploy.yaml
- job.yaml
- marvel_svc.yaml
Once Kustomize looks into the base
folder, it applies customization to all its children YAML resources described in that file under resources.
The dev folder also contains a kustomization.yaml
file, which details the specific changes needed to render the final version of the manifests.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patchesStrategicMerge:
- ondat_sc.yaml
- marvel_deploy.yaml
- mongodbcommunity_cr.yaml
- job.yaml
resources:
- ../../base
configMapGenerator:
- name: mongo-config
literals:
- MONGO_SEED0=mongodb-0.mongodb.default.svc.cluster.local
- MONGO_SEED1=mongodb-1.mongodb.default.svc.cluster.local
- MONGO_SEED2=mongodb-2.mongodb.default.svc.cluster.local
- OFFSET=600
- MONGO_USERNAME=admin
secretGenerator:
- name: admin-password
literals:
- password=mongo
configurations:
- name_reference.yaml
Let’s take a look at the different sections:
pacthesStrategicMerge
lists the files Kustomize should look at when building the target manifests. These files are located under the dev folder and describe the amendments applied to the manifests located in the base folder. These files are accessible within the git repo, and here is a summary of the changes they describe:ondat.sc.yaml
: Create 1 replica for all volumes using thatStorageClass
, and enable encryption.marvel_deploy.yaml
: Set the number of replicas to 2, inject the MongoDB password from the KustomizeSecret
generator, and add the environment variables from theconfigMap
generator.mongodbcommunity_cr.yaml
: Set the MongoDB version to 5.0.5, set up the admin user, configure the KubernetesStatefulSet
with avolumeClaimTemplate
using the OndatStorageClass
, and set the data and logs volume size.job.yaml
: Inject the MongoDB password from the KustomizeSecret
generator and environment variables from theconfigMap
generator.
In the remaining sections, we have:
resources
, which defines the location of the base manifests.configMapGenerator
, which generates aconfigMap
with a unique name every time Kustomize runs. You can use literals or files to define your variables.secretGenerator
, which generatesSecrets
with a unique name every time Kustomize runs. You can use literals or files to define your secrets.configurations
, which specifies custom objects path that Kustomize uses to modify or insert particular values. For example, in our scenario, Kustomize dynamically configures the reference to a secret in the MongoDB custom resource. As it is a custom resource, Kustomize doesn’t know where to find the equivalent of a secret name parameter. Instead, we define it in the filename_reference.yaml
:
nameReference:
- kind: Secret
fieldSpecs:
- kind: MongoDBCommunity
path: spec/users/passwordSecretRef/name
As a result, a unique Secret
name composed of the initial Secret
name and a random string will be added at this location every time Kustomize is invoked. Kustomize will also replace all secrets within native objects YAML definition in relevant resources. Those are the resources you have defined in the kustomization.yaml
file.
Install and Configure the MongoDB Operator
For this article, we have used the MongoDB Community Operator since it is free and easy to use. However, for production environments, we recommend using an Enterprise version of the Operator or being ready to support it yourself and get help from the community. There are multiple options available, such as the MongoDB Enterprise Operator, the Percona Operator, or the KubeDB Operator.
Understanding the Operator
Some of you may not be familiar with Kubernetes Operators or why we need one to install a database cluster altogether. So let’s focus a little bit on this aspect. A Kubernetes Operator is fundamentally made up of 2 parts:
- A piece of code provided as a container that continuously monitors specific objects in the Kubernetes API and performs actions as a result of this active monitoring. It is called a custom controller.
- The monitored custom resources. A custom resource is a Kubernetes object that is not native. The custom resource schema is defined within the Custom Resource Definiton (CRD), provided as a YAML file by the user, and sent to the Kubernetes API. Every object created with that schema is further saved in the Kubernetes etcd store. You can compare the CRD to a class in OOP and the custom resource to an instance of that class. It extends the existing Kubernetes API.
The Kubernetes Operator’s job is to perform automated actions and interact with the cluster or systems outside the cluster like a human operator. It is not limited to deploying components only. It can perform CRUD operations in reaction to Kubernetes events, which are dynamic by nature. It utilizes events data as input to workflows. A common use case for Operators is to automate the deployment of software solutions within Kubernetes. So in the case of a database, the Kubernetes Operator will install the database (cluster or standalone) once the corresponding database resource has been ingested by the Kubernetes API, typically in the form of a YAML manifest that describes the configuration of the database as a custom resource.
Also, remember the database is deployed as a StatefulSet
. When you scale the StatefulSet
, Kubernetes doesn’t automagically scale the database cluster. It just deploys new containers with the database image. There is some extra work needed to configure it. It is performed by the Kubernetes Operator. The same is true when you scale down the StatefulSet
.
In the case of the MongoDB Operator, the added CRD produces an object of kind mongodbcommunity
. The custom resource encapsulates all the information required to deploy and maintain a MongoDB database. First, you need to install the Operator and create the custom resource. It can be achieved during the automated deployment of your Kubernetes cluster or manually once the cluster has been installed. You can find all the required files in this repo. We’re going to perform a couple of steps to install the Operator:
- Install the CRD
- Create the Operator configuration manifest. You can refer to the MongoDB Community Operator documentation and examples. You can also find the configuration we have used in the git repo. The Operator configuration manifest is
manager.yaml
.
You need to configure the scope of the Operator first. It defines which namespaces the Operator monitors: the Namespace
where the Operator is installed, a specific namespace, or all namespaces. In our example, the manager.yaml
file has the following configuration:
env:
- name: WATCH_NAMESPACE
value: "*"
It tells the Operator to monitor all namespaces for MongoDB custom resources operations.
- Create cluster-wide roles and role-bindings. The
ClusterRoleBinding
service account namespace must be modified with the name of the namespace you want to use (in bold in the text below). It is located in the fileclusterwide/role_binding.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-kubernetes-operator
subjects:
- kind: ServiceAccount
namespace: mongo-operator
name: mongodb-kubernetes-operator
roleRef:
kind: ClusterRole
name: mongodb-kubernetes-operator
apiGroup: rbac.authorization.k8s.io
- For each namespace that you wish the Operator to watch, you need to deploy a
Role
,RoleBinding
andServiceAccount
in that namespace - The last step is creating and deploying the database configuration manifest (the custom resource), which is detailed later.
Deploy the Operator
Execute the following commands to install and configure the Operator:
#Clone the git repository
$ git clone https://github.com/vfiftyfive/mongodb-community-operator-manifests && cd mongodb-community-operator-manifests
Cloning into 'mongodb-community-operator-manifests'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 17 (delta 5), reused 15 (delta 3), pack-reused 0
Receiving objects: 100% (17/17), done.
Resolving deltas: 100% (5/5), done.
#Install the CRD
$ kubectl apply -f mongo-crd.yaml
#Deploy the clusterwide RBAC resources
$ kubectl apply -f clusterwide/
clusterrole.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
clusterrolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
#Create a namespace for the operator
$ kubectl create ns mongo-operator
namespace/mongo-operator created
#Deploy namespace RBAC resources in the operator namespace.
$ kubectl apply -k rbac/ -n mongo-operator
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
#Deploy namespace RBAC resources in the default namespace (where the app will be deployed)
$ kubectl apply -k rbac/
serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
#Deploy the Operator
$ k apply -f manager.yaml -n mongo-operator
deployment.apps/mongodb-kubernetes-operator created
#Check the Operator has correctly been deployed
$ kubectl get po -n mongo-operator
NAME READY STATUS ...
mongodb-kubernetes-operator-6d46dd4b74-ztx9c 1/1 Running ...
The Operator is now ready to deploy a new MongoDB database as soon as it detects that a new custom resource has been added into the Kubernetes API.
The next step is to install the distributed storage layer, Ondat (formerly StorageOS).
Install Ondat
Ondat provides a data-mesh acting as a distributed persistent storage layer that provides premium features that are not included by default in Kubernetes. This includes replication, encryption, performance optimization, intelligent volume placement, etc, all managed as part of native Kubernetes labels and annotations.
It can be deployed with a single line through a kubectl
plugin (or alternatively using a helm chart available here):
kubectl storageos install --include-etcd
But first, let’s install the plugin by executing the following command:
curl -sSLo kubectl-storageos.tar.gz \
https://github.com/storageos/kubectl-storageos/releases/download/v1.0.0/kubectl-storageos_1.0.0_linux_amd64.tar.gz \
&& tar -xf kubectl-storageos.tar.gz \
&& chmod +x kubectl-storageos \
&& sudo mv kubectl-storageos /usr/local/bin/ \
&& rm kubectl-storageos.tar.gz
As there are some prereqs for your Kubernetes hosts’ kernel modules, you should first run the preflight checks to see if your cluster is compatible by running the following command:
kubectl storageos preflight
(Quick tip: if you are using GKE, use the ubuntu_containerd image, it already includes the linux-modules-extra-x.y.z package)
The command will output a comprehensive report like the one below. The screenshot depicts an example with no NVMe drives present in our 3-node cluster. You can safely ignore this warning if you’re not running performance-intensive workloads.
You can then use the plugin with the install sub-command:
kubeclt storageos install --include-etcd
There are many options to this command. If you consider a production deployment, you should take a look at them and get familiar with best practices. As this is not the topic of this article, I’m just going to point you in the right direction here!
To check the deployment is complete, make sure the DaemonSet
is up and running by running the following command:
$ kubectl get pods -n storageos | grep node
storageos-node-8chxx 3/3 Running
storageos-node-gsdzt 3/3 Running
storageos-node-sq2ds 3/3 Running
All containers must be up and running. You should also notice a new StorageClass
named “storageos”. But remember that Kustomize is going to create a new StorageClass
, so we’re not going to use that one.
We have now deployed and configured Kustomize, MongoDB, and Ondat. This is already a lot! In the next part, we’ll install and configure Skaffold and build the pipeline to continuously develop, deploy and test our application. We kept the best for the end, so don’t miss the next article!