Using the RabbitMQ Kubernetes Operator with Persistent Data

RabbitMQ & Kubernetes

Although RabbitMQ is easy to deploy on traditional infrastructure, the initial transition to Kubernetes originally brought a number of challenges. For those who have tried this in the past, you’ll be aware that the process wasn’t for the faint hearted!

The successful launch of a cluster required a diverse understanding of Kubernetes including the configuration of RBAC, ConfigMaps, Secrets and Lengthy YAML declarations to achieve a fully working deployment.

Fortunately, the RabbitMQ team addressed these challenges for Kubernetes with the development of the RabbitMQ Operator, a quick and convenient way of deploying RabbitMQ clusters on Kubernetes. The operator also provides integration hooks for persistent storage allowing CSI data layers like Ondat, to integrate into the deployment process.

Installing RabbitMQ on Kubernetes using the RabbitMQ Operator

In this example, we show the end to end setup of RabbitMQ via the RabbitMQ Kubernetes Operator with Persistent Data from Ondat, using a Kubernetes cluster with StorageOS installed as per our Installation & Setup Guide

1. Configuring a Production StorageClass

We start by configuring a StorageOS StorageClass called production to provide 3 copies of Data, 1 Primary, and 2 Replicas.

StorageClasses provide a convenient means of providing tiering, features and multi-tenancy in a Kubernetes environment. For more information see StorageOS StorageClasses – Tiering and Features –

✨ Creating StorageClass production - kubectl apply -f- <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: production
 labels:
 app: storageos
provisioner: csi.storageos.com # CSI Driver
allowVolumeExpansion: true
parameters:
 storageos.com/replicas: "2" # 3 copies of Data, 1 Primary, 2 Replicas
 csi.storage.k8s.io/controller-expand-secret-name: csi-controller-expand-secret csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
 csi.storage.k8s.io/controller-publish-secret-name: csi-controller-publish-secret csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
 csi.storage.k8s.io/fstype: ext4
 csi.storage.k8s.io/node-publish-secret-name: csi-node-publish-secret
 csi.storage.k8s.io/node-publish-secret-namespace: kube-system
 csi.storage.k8s.io/provisioner-secret-name: csi-provisioner-secret
 csi.storage.k8s.io/provisioner-secret-namespace: kube-system
EOF
storageclass.storage.k8s.io/production created

2. Installing the RabbitMQ Operator

Installing the RabbitMQ Kubernetes Operator is straight forward with the option to install the operator directly from the RabbitMQ Cluster Operator repository –

✨ Deploying RabbingMQ Cluster Operator - kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml namespace/rabbitmq-system created
customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created

3. Verify the availability of the RabbitMQ CRD

After installing, you’ll be able to verify the installation by checking for the RabbitMQ Custom Resource Definition –

🔍 Checking CRD for RabbitMQ extension - kubectl get customresourcedefinitions | grep rabbit rabbitmqclusters.rabbitmq.com           2021-07-07T16:35:05Z

4. Define a RabbitMQ Cluster

The RabbitmqCluster Operator is used to define the specifications of the RabbitMQ cluster and in this example, we’re defining 3 replicas. For data persistence, we are specifying the StorageClass of production provided by Ondat, that we created in step 1 –

✨ Creating RabbitMQ Cluster with Persistent Storage - kubectl apply -f- <<EOF
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
 name: rabbitmqcluster
spec:
 replicas: 3
 persistence:
 storageClassName: production
 storage: 5Gi
 resources:
 requests:
 cpu: 100m
 memory: 2Gi
 limits:
 cpu: 100m
 memory: 2Gi
EOF
rabbitmqcluster.rabbitmq.com/rabbitmqcluster created

5. Monitor Progress

The RabbitMQ operator will initialise the creation of pods, seen here as pod/rabbitmqcluster-server-[0-2], services for the user interface, internal networking and a statefulset for management of the application –

🔍 Check Progress - kubectl get all -A | grep -i rabbit
default              pod/rabbitmqcluster-server-0                      0/1     Running   0          3m
default              pod/rabbitmqcluster-server-1                      0/1     Running   0          3m
default              pod/rabbitmqcluster-server-2                      0/1     Running   0          3m
rabbitmq-system      pod/rabbitmq-cluster-operator-5b4b795998-4424m    1/1     Running   0          4m36s
default              service/rabbitmqcluster                      ClusterIP   10.101.59.229    <none>        15692/TCP,5672/TCP,15672/TCP   3m
default              service/rabbitmqcluster-nodes                ClusterIP   None             <none>        4369/TCP,25672/TCP             3m
rabbitmq-system      deployment.apps/rabbitmq-cluster-operator    1/1     1            1           4m36s
rabbitmq-system      replicaset.apps/rabbitmq-cluster-operator-5b4b795998    1         1         1       4m36s
default     statefulset.apps/rabbitmqcluster-server   0/3     3m
default     rabbitmqcluster.rabbitmq.com/rabbitmqcluster   False              Unknown            3m1s

After a short period, the application will transition to a Running state with 3/3 showing for the statefulset –

🔍 Check Progress - kubectl get all -A | grep -i rabbit
default              pod/rabbitmqcluster-server-0                      1/1     Running   0          6m41s
default              pod/rabbitmqcluster-server-1                      1/1     Running   0          6m41s
default              pod/rabbitmqcluster-server-2                      1/1     Running   0          6m41s
rabbitmq-system      pod/rabbitmq-cluster-operator-5b4b795998-4424m    1/1     Running   0          8m17s
default              service/rabbitmqcluster                      ClusterIP   10.101.59.229            15692/TCP,5672/TCP,15672/TCP   6m41s
default              service/rabbitmqcluster-nodes                ClusterIP   None                     4369/TCP,25672/TCP             6m41s
rabbitmq-system      deployment.apps/rabbitmq-cluster-operator    1/1     1            1           8m17s
rabbitmq-system      replicaset.apps/rabbitmq-cluster-operator-5b4b795998    1         1         1       8m17s
default     statefulset.apps/rabbitmqcluster-server   3/3     6m41s
default     rabbitmqcluster.rabbitmq.com/rabbitmqcluster   True               True               6m42s

6. RabbitMQ Credentials

The RabbitMQ username and password are configured as Kubernetes secrets during setup and configuration, access both the username and password as follows –

🔍 Capturing RabbitMQ Username - kubectl get secret rabbitmqcluster-default-user -o jsonpath='{.data.username}' | base64 --decode Kyf2i5kxMktl00U9ANls_yrjPOeO9I2c
🔍 Capturing RabbitMQ Password - kubectl get secret rabbitmqcluster-default-user -o jsonpath='{.data.password}' | base64 --decode LtFV4PtfxsLO4cU0gA5-8stX-oLpLrNw

7. Access the RabbitMQ UI

With login credentials, we can access the RabbitMQ UI using the convenient kubectl port-forwarding functionality, forwarding the UI to our local system. In this example we’re using a host called k8s-1 –

✨ Port Forwarding svc/rabbitmqcluster:15672 via kubectl - kubectl port-forward --address 0.0.0.0 svc/rabbitmqcluster 15672 ✨ Access via - http://k8s-1:15672 - Press Ctrl-C to exit
Forwarding from 0.0.0.0:15672 -> 15672

Browse to the target for the RabbitMQ User Interface and log in using the credentials from step 6 –

After logging in, you’ll be presented with an overview of the cluster, configured as expected with 3 nodes –

8. Scale the RabbitMQ Cluster

With native Kubernetes integration, the RabbitMQ cluster can be scaled via the statefulset –

✨ Scaling RabbitMQ Cluster to 4 Nodes - kubectl scale statefulset.apps/rabbitmqcluster-server --replicas 4 statefulset.apps/rabbitmqcluster-server scaled

If the UI is kept in view, you’ll see the cluster view dynamically update, as the new node comes online

9. View the RabbitMQ Persistent Storage

Finally, behind the scenes Ondat is providing highly available persistent data for all nodes

🔍 Show PV - kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                          STORAGECLASS   REASON   AGE
pvc-0f365cf3-d3f6-475e-8c76-79f066cb5c58   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-1   production              18h
pvc-2bc95d1a-6041-4006-8bf3-ec83cd3b9d85   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-0   production              18h
pvc-2d8477f7-0d30-4799-b562-d224362d588e   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-2   production              18h
pvc-d8091522-d7b7-4f9c-81c6-d6b5cc9695f9   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-3   production              18h

🔍 Show PVC - kubectl get pvc
NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistence-rabbitmqcluster-server-0   Bound    pvc-2bc95d1a-6041-4006-8bf3-ec83cd3b9d85   5Gi        RWO            production     18h
persistence-rabbitmqcluster-server-1   Bound    pvc-0f365cf3-d3f6-475e-8c76-79f066cb5c58   5Gi        RWO            production     18h
persistence-rabbitmqcluster-server-2   Bound    pvc-2d8477f7-0d30-4799-b562-d224362d588e   5Gi        RWO            production     18h
persistence-rabbitmqcluster-server-3   Bound    pvc-d8091522-d7b7-4f9c-81c6-d6b5cc9695f9   5Gi        RWO            production     18h

We hope that you have found this demonstration and viewpoint of both RabbitMQ and Ondat useful.  More information on RabbitMQ can be found at rabbitmq and should you wish to try Ondat, you can test drive it!