Go back to Blog articles
RabbitMQ & Kubernetes
Although RabbitMQ is easy to deploy on traditional infrastructure, the initial transition to Kubernetes originally brought a number of challenges. For those who have tried this in the past, you’ll be aware that the process wasn’t for the faint hearted!
The successful launch of a cluster required a diverse understanding of Kubernetes including the configuration of RBAC, ConfigMaps, Secrets and Lengthy YAML declarations to achieve a fully working deployment.
Fortunately, the RabbitMQ team addressed these challenges for Kubernetes with the development of the RabbitMQ Operator, a quick and convenient way of deploying RabbitMQ clusters on Kubernetes. The operator also provides integration hooks for persistent storage allowing CSI data layers like Ondat, to integrate into the deployment process.
Installing RabbitMQ on Kubernetes using the RabbitMQ Operator
In this example, we show the end to end setup of RabbitMQ via the RabbitMQ Kubernetes Operator with Persistent Data from Ondat, using a Kubernetes cluster with StorageOS installed as per our Installation & Setup Guide
1. Configuring a Production StorageClass
We start by configuring a StorageOS StorageClass called production to provide 3 copies of Data, 1 Primary, and 2 Replicas.
StorageClasses provide a convenient means of providing tiering, features and multi-tenancy in a Kubernetes environment. For more information see StorageOS StorageClasses – Tiering and Features –
✨ Creating StorageClass production - kubectl apply -f- <<EOF apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: production labels: app: storageos provisioner: csi.storageos.com # CSI Driver allowVolumeExpansion: true parameters: storageos.com/replicas: "2" # 3 copies of Data, 1 Primary, 2 Replicas csi.storage.k8s.io/controller-expand-secret-name: csi-controller-expand-secret csi.storage.k8s.io/controller-expand-secret-namespace: kube-system csi.storage.k8s.io/controller-publish-secret-name: csi-controller-publish-secret csi.storage.k8s.io/controller-publish-secret-namespace: kube-system csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/node-publish-secret-name: csi-node-publish-secret csi.storage.k8s.io/node-publish-secret-namespace: kube-system csi.storage.k8s.io/provisioner-secret-name: csi-provisioner-secret csi.storage.k8s.io/provisioner-secret-namespace: kube-system EOF storageclass.storage.k8s.io/production created
2. Installing the RabbitMQ Operator
Installing the RabbitMQ Kubernetes Operator is straight forward with the option to install the operator directly from the RabbitMQ Cluster Operator repository –
✨ Deploying RabbingMQ Cluster Operator - kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml namespace/rabbitmq-system created
customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created
3. Verify the availability of the RabbitMQ CRD
After installing, you’ll be able to verify the installation by checking for the RabbitMQ Custom Resource Definition –
🔍 Checking CRD for RabbitMQ extension - kubectl get customresourcedefinitions | grep rabbit rabbitmqclusters.rabbitmq.com 2021-07-07T16:35:05Z
4. Define a RabbitMQ Cluster
The RabbitmqCluster Operator is used to define the specifications of the RabbitMQ cluster and in this example, we’re defining 3 replicas. For data persistence, we are specifying the StorageClass of production provided by Ondat, that we created in step 1 –
✨ Creating RabbitMQ Cluster with Persistent Storage - kubectl apply -f- <<EOF apiVersion: rabbitmq.com/v1beta1 kind: RabbitmqCluster metadata: name: rabbitmqcluster spec: replicas: 3 persistence: storageClassName: production storage: 5Gi resources: requests: cpu: 100m memory: 2Gi limits: cpu: 100m memory: 2Gi EOF rabbitmqcluster.rabbitmq.com/rabbitmqcluster created
5. Monitor Progress
The RabbitMQ operator will initialise the creation of pods, seen here as pod/rabbitmqcluster-server-[0-2], services for the user interface, internal networking and a statefulset for management of the application –
🔍 Check Progress - kubectl get all -A | grep -i rabbit
default pod/rabbitmqcluster-server-0 0/1 Running 0 3m
default pod/rabbitmqcluster-server-1 0/1 Running 0 3m
default pod/rabbitmqcluster-server-2 0/1 Running 0 3m
rabbitmq-system pod/rabbitmq-cluster-operator-5b4b795998-4424m 1/1 Running 0 4m36s
default service/rabbitmqcluster ClusterIP 10.101.59.229 <none> 15692/TCP,5672/TCP,15672/TCP 3m
default service/rabbitmqcluster-nodes ClusterIP None <none> 4369/TCP,25672/TCP 3m
rabbitmq-system deployment.apps/rabbitmq-cluster-operator 1/1 1 1 4m36s
rabbitmq-system replicaset.apps/rabbitmq-cluster-operator-5b4b795998 1 1 1 4m36s
default statefulset.apps/rabbitmqcluster-server 0/3 3m
default rabbitmqcluster.rabbitmq.com/rabbitmqcluster False Unknown 3m1s
After a short period, the application will transition to a Running state with 3/3 showing for the statefulset –
🔍 Check Progress - kubectl get all -A | grep -i rabbit
default pod/rabbitmqcluster-server-0 1/1 Running 0 6m41s
default pod/rabbitmqcluster-server-1 1/1 Running 0 6m41s
default pod/rabbitmqcluster-server-2 1/1 Running 0 6m41s
rabbitmq-system pod/rabbitmq-cluster-operator-5b4b795998-4424m 1/1 Running 0 8m17s
default service/rabbitmqcluster ClusterIP 10.101.59.229 15692/TCP,5672/TCP,15672/TCP 6m41s
default service/rabbitmqcluster-nodes ClusterIP None 4369/TCP,25672/TCP 6m41s
rabbitmq-system deployment.apps/rabbitmq-cluster-operator 1/1 1 1 8m17s
rabbitmq-system replicaset.apps/rabbitmq-cluster-operator-5b4b795998 1 1 1 8m17s
default statefulset.apps/rabbitmqcluster-server 3/3 6m41s
default rabbitmqcluster.rabbitmq.com/rabbitmqcluster True True 6m42s
6. RabbitMQ Credentials
The RabbitMQ username and password are configured as Kubernetes secrets during setup and configuration, access both the username and password as follows –
🔍 Capturing RabbitMQ Username - kubectl get secret rabbitmqcluster-default-user -o jsonpath='{.data.username}' | base64 --decode Kyf2i5kxMktl00U9ANls_yrjPOeO9I2c 🔍 Capturing RabbitMQ Password - kubectl get secret rabbitmqcluster-default-user -o jsonpath='{.data.password}' | base64 --decode LtFV4PtfxsLO4cU0gA5-8stX-oLpLrNw
7. Access the RabbitMQ UI
With login credentials, we can access the RabbitMQ UI using the convenient kubectl port-forwarding functionality, forwarding the UI to our local system. In this example we’re using a host called k8s-1 –
✨ Port Forwarding svc/rabbitmqcluster:15672 via kubectl - kubectl port-forward --address 0.0.0.0 svc/rabbitmqcluster 15672 ✨ Access via - http://k8s-1:15672 - Press Ctrl-C to exit Forwarding from 0.0.0.0:15672 -> 15672
Browse to the target for the RabbitMQ User Interface and log in using the credentials from step 6 –
After logging in, you’ll be presented with an overview of the cluster, configured as expected with 3 nodes –
8. Scale the RabbitMQ Cluster
With native Kubernetes integration, the RabbitMQ cluster can be scaled via the statefulset –
✨ Scaling RabbitMQ Cluster to 4 Nodes - kubectl scale statefulset.apps/rabbitmqcluster-server --replicas 4 statefulset.apps/rabbitmqcluster-server scaled
If the UI is kept in view, you’ll see the cluster view dynamically update, as the new node comes online
9. View the RabbitMQ Persistent Storage
Finally, behind the scenes Ondat is providing highly available persistent data for all nodes
🔍 Show PV - kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-0f365cf3-d3f6-475e-8c76-79f066cb5c58 5Gi RWO Delete Bound default/persistence-rabbitmqcluster-server-1 production 18h pvc-2bc95d1a-6041-4006-8bf3-ec83cd3b9d85 5Gi RWO Delete Bound default/persistence-rabbitmqcluster-server-0 production 18h pvc-2d8477f7-0d30-4799-b562-d224362d588e 5Gi RWO Delete Bound default/persistence-rabbitmqcluster-server-2 production 18h pvc-d8091522-d7b7-4f9c-81c6-d6b5cc9695f9 5Gi RWO Delete Bound default/persistence-rabbitmqcluster-server-3 production 18h 🔍 Show PVC - kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistence-rabbitmqcluster-server-0 Bound pvc-2bc95d1a-6041-4006-8bf3-ec83cd3b9d85 5Gi RWO production 18h persistence-rabbitmqcluster-server-1 Bound pvc-0f365cf3-d3f6-475e-8c76-79f066cb5c58 5Gi RWO production 18h persistence-rabbitmqcluster-server-2 Bound pvc-2d8477f7-0d30-4799-b562-d224362d588e 5Gi RWO production 18h persistence-rabbitmqcluster-server-3 Bound pvc-d8091522-d7b7-4f9c-81c6-d6b5cc9695f9 5Gi RWO production 18h
We hope that you have found this demonstration and viewpoint of both RabbitMQ and Ondat useful. More information on RabbitMQ can be found at rabbitmq and should you wish to try Ondat, you can test drive it!