Using the RabbitMQ Kubernetes Operator with Persistent Data



RabbitMQ & Kubernetes

Although RabbitMQ is easy to deploy on traditional infrastructure, the initial transition to Kubernetes originally brought a number of challenges. For those who have tried this in the past, you’ll be aware that the process wasn’t for the faint hearted!

The successful launch of a cluster required a diverse understanding of Kubernetes including the configuration of RBAC, ConfigMaps, Secrets and Lengthy YAML declarations to achieve a fully working deployment.

Fortunately, the RabbitMQ team addressed these challenges for Kubernetes with the development of the RabbitMQ Operator, a quick and convenient way of deploying RabbitMQ clusters on Kubernetes. The operator also provides integration hooks for persistent storage allowing CSI data layers like Ondat, to integrate into the deployment process.

Installing RabbitMQ on Kubernetes using the RabbitMQ Operator

In this example, we show the end to end setup of RabbitMQ via the RabbitMQ Kubernetes Operator with Persistent Data from Ondat, using a Kubernetes cluster with StorageOS installed as per our Installation & Setup Guide

1. Configuring a Production StorageClass

We start by configuring a StorageOS StorageClass called production to provide 3 copies of Data, 1 Primary, and 2 Replicas.

StorageClasses provide a convenient means of providing tiering, features and multi-tenancy in a Kubernetes environment. For more information see StorageOS StorageClasses – Tiering and Features

Creating StorageClass production - kubectl apply -f- <<EOF
kind: StorageClass
 name: production
 app: storageos
provisioner: # CSI Driver
allowVolumeExpansion: true
parameters: "2" # 3 copies of Data, 1 Primary, 2 Replicas csi-controller-expand-secret kube-system csi-controller-publish-secret kube-system ext4 csi-node-publish-secret kube-system csi-provisioner-secret kube-system
EOF created

2. Installing the RabbitMQ Operator

Installing the RabbitMQ Kubernetes Operator is straight forward with the option to install the operator directly from the RabbitMQ Cluster Operator repository –

Deploying RabbingMQ Cluster Operator - kubectl apply -f namespace/rabbitmq-system created created
serviceaccount/rabbitmq-cluster-operator created created created created created
deployment.apps/rabbitmq-cluster-operator created

3. Verify the availability of the RabbitMQ CRD

After installing, you’ll be able to verify the installation by checking for the RabbitMQ Custom Resource Definition –

🔍 Checking CRD for RabbitMQ extension - kubectl get customresourcedefinitions | grep rabbit           2021-07-07T16:35:05Z

4. Define a RabbitMQ Cluster

The RabbitmqCluster Operator is used to define the specifications of the RabbitMQ cluster and in this example, we’re defining 3 replicas. For data persistence, we are specifying the StorageClass of production provided by Ondat, that we created in step 1 –

Creating RabbitMQ Cluster with Persistent Storage - kubectl apply -f- <<EOF
kind: RabbitmqCluster
 name: rabbitmqcluster
 replicas: 3
 storageClassName: production
 storage: 5Gi
 cpu: 100m
 memory: 2Gi
 cpu: 100m
 memory: 2Gi
EOF created

5. Monitor Progress

The RabbitMQ operator will initialise the creation of pods, seen here as pod/rabbitmqcluster-server-[0-2], services for the user interface, internal networking and a statefulset for management of the application –

🔍 Check Progress - kubectl get all -A | grep -i rabbit
default              pod/rabbitmqcluster-server-0                      0/1     Running   0          3m
default              pod/rabbitmqcluster-server-1                      0/1     Running   0          3m
default              pod/rabbitmqcluster-server-2                      0/1     Running   0          3m
rabbitmq-system      pod/rabbitmq-cluster-operator-5b4b795998-4424m    1/1     Running   0          4m36s
default              service/rabbitmqcluster                      ClusterIP    <none>        15692/TCP,5672/TCP,15672/TCP   3m
default              service/rabbitmqcluster-nodes                ClusterIP   None             <none>        4369/TCP,25672/TCP             3m
rabbitmq-system      deployment.apps/rabbitmq-cluster-operator    1/1     1            1           4m36s
rabbitmq-system      replicaset.apps/rabbitmq-cluster-operator-5b4b795998    1         1         1       4m36s
default     statefulset.apps/rabbitmqcluster-server   0/3     3m
default   False              Unknown            3m1s

After a short period, the application will transition to a Running state with 3/3 showing for the statefulset –

🔍 Check Progress - kubectl get all -A | grep -i rabbit
default              pod/rabbitmqcluster-server-0                      1/1     Running   0          6m41s
default              pod/rabbitmqcluster-server-1                      1/1     Running   0          6m41s
default              pod/rabbitmqcluster-server-2                      1/1     Running   0          6m41s
rabbitmq-system      pod/rabbitmq-cluster-operator-5b4b795998-4424m    1/1     Running   0          8m17s
default              service/rabbitmqcluster                      ClusterIP            15692/TCP,5672/TCP,15672/TCP   6m41s
default              service/rabbitmqcluster-nodes                ClusterIP   None                     4369/TCP,25672/TCP             6m41s
rabbitmq-system      deployment.apps/rabbitmq-cluster-operator    1/1     1            1           8m17s
rabbitmq-system      replicaset.apps/rabbitmq-cluster-operator-5b4b795998    1         1         1       8m17s
default     statefulset.apps/rabbitmqcluster-server   3/3     6m41s
default   True               True               6m42s

6. RabbitMQ Credentials

The RabbitMQ username and password are configured as Kubernetes secrets during setup and configuration, access both the username and password as follows –

🔍 Capturing RabbitMQ Username - kubectl get secret rabbitmqcluster-default-user -o jsonpath='{.data.username}' | base64 --decode Kyf2i5kxMktl00U9ANls_yrjPOeO9I2c
🔍 Capturing RabbitMQ Password - kubectl get secret rabbitmqcluster-default-user -o jsonpath='{.data.password}' | base64 --decode LtFV4PtfxsLO4cU0gA5-8stX-oLpLrNw

7. Access the RabbitMQ UI

With login credentials, we can access the RabbitMQ UI using the convenient kubectl port-forwarding functionality, forwarding the UI to our local system. In this example we’re using a host called k8s-1 –

Port Forwarding svc/rabbitmqcluster:15672 via kubectl - kubectl port-forward --address svc/rabbitmqcluster 15672 Access via - http://k8s-1:15672 - Press Ctrl-C to exit
Forwarding from -> 15672

Browse to the target for the RabbitMQ User Interface and log in using the credentials from step 6 –

After logging in, you’ll be presented with an overview of the cluster, configured as expected with 3 nodes –

8. Scale the RabbitMQ Cluster

With native Kubernetes integration, the RabbitMQ cluster can be scaled via the statefulset –

Scaling RabbitMQ Cluster to 4 Nodes - kubectl scale statefulset.apps/rabbitmqcluster-server --replicas 4 statefulset.apps/rabbitmqcluster-server scaled

If the UI is kept in view, you’ll see the cluster view dynamically update, as the new node comes online

9. View the RabbitMQ Persistent Storage

Finally, behind the scenes Ondat is providing highly available persistent data for all nodes

🔍 Show PV - kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                          STORAGECLASS   REASON   AGE
pvc-0f365cf3-d3f6-475e-8c76-79f066cb5c58   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-1   production              18h
pvc-2bc95d1a-6041-4006-8bf3-ec83cd3b9d85   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-0   production              18h
pvc-2d8477f7-0d30-4799-b562-d224362d588e   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-2   production              18h
pvc-d8091522-d7b7-4f9c-81c6-d6b5cc9695f9   5Gi        RWO            Delete           Bound    default/persistence-rabbitmqcluster-server-3   production              18h

🔍 Show PVC - kubectl get pvc
NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistence-rabbitmqcluster-server-0   Bound    pvc-2bc95d1a-6041-4006-8bf3-ec83cd3b9d85   5Gi        RWO            production     18h
persistence-rabbitmqcluster-server-1   Bound    pvc-0f365cf3-d3f6-475e-8c76-79f066cb5c58   5Gi        RWO            production     18h
persistence-rabbitmqcluster-server-2   Bound    pvc-2d8477f7-0d30-4799-b562-d224362d588e   5Gi        RWO            production     18h
persistence-rabbitmqcluster-server-3   Bound    pvc-d8091522-d7b7-4f9c-81c6-d6b5cc9695f9   5Gi        RWO            production     18h

We hope that you have found this demonstration and viewpoint of both RabbitMQ and Ondat useful.  More information on RabbitMQ can be found at rabbitmq and should you wish to try Ondat, you can test drive it

written by:
James Spurin