Go back to Blog articles
Overview
With the increased market use of Kubernetes as a cloud managed service and strict data compliance regulations that include the Payment Card Industry Standard (PCI DSS) and the Health Insurance Portability and Accountability Act (HIPPA), organisations are dependent on encrypted data as a requirement for Kubernetes workloads using persistent storage.
To accommodate these requirements, Ondat has further improved its core Encryption at Rest functionality in V2.4 with Encryption and Decryption handled by the CPU’s accelerated instruction set instruction set, providing accelerated data at rest for both encryption and decryption for AES-256 with AES-XTS encryption.
Ondat, a market leader in the field of cloud native persistent storage provides a Kubernetes native CSI interface, facilitating high performance, data availability and advanced storage features that historically were only available in hardware-based enterprise class storage arrays.
The following example demonstrates the use of Encryption at Rest in Kubernetes with Ondat –
1. Setup and Configuration
For this setup, we have a standard Kubernetes cluster that has StorageOS installed through the one-shot installation process –
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
storageos-k8s1 Ready controlplane,etcd,worker 6d22h v1.17.17
storageos-k8s2 Ready controlplane,etcd,worker 6d22h v1.17.17
storageos-k8s3 Ready controlplane,etcd,worker 6d22h v1.17.172. Setup of an Encrypted and Unencrypted Volume
We setup both an encrypted and unencrypted volume to show the variance in configuration and how data at rest applies to storage within the Kubernetes infrastructure –
✨ Creating an Encrypted and UnEncrypted PVC/PV - kubectl apply -f- <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: encrypted
labels:
storageos.com/encryption: "true"
spec:
storageClassName: fast
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: unencrypted
spec:
storageClassName: fast
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
persistentvolumeclaim/encrypted created
persistentvolumeclaim/unencrypted createdStorageOS simplifies the setup of encrypted volumes with a single change required to the standard workflow. The use of the encryption label as seen above in the setup of the first Persistent Volume Claim (PVC).
This simple change provides an encrypted volume along with the associated encryption keys required for encryption when using Ondat.
3. Viewing Encryption Keys
Viewing the volumes with a kubectl describe shows that the encrypted volume has additional annotations relating to the encryption keys used for encryption at rest –
🔍 Showing PVC Annotation - encrypted - kubectl describe pvc/encrypted
Name: encrypted
Namespace: default
StorageClass: fast
Status: Bound
Volume: pvc-3fd15067-992e-4d93-b816-f686546025d1
Labels: storageos.com/encryption=true
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
storageos.com/encryption-secret-name: storageos-volume-key-2af02d26-4913-4a66-9f4a-0aae9589d8d4
storageos.com/encryption-secret-namespace: default
storageos.com/storageclass: 40579749-5ae8-44ae-8ae2-55a3858d4e15
volume.beta.kubernetes.io/storage-provisioner: csi.storageos.com
Correspondingly, the Kubernetes Secret can be viewed to access the encryption key data specific for this volume-🔍 Showing Encryption Key Secret for PVC encrypted - kubectl get secret storageos-volume-key-2af02d26-4913-4a66-9f4a-
0aae9589d8d4 -o yaml
apiVersion: v1
data:
hmac: 9jqbnLGA9SfG+AhXzgZ9e49d+VrfJa3ki9C8eZOhubA=
iv: JS0Cq1+0C/nnkL7plErbsZx8BHo28ies48sURXm4vAo=
key: 8GkpcbBMQFAGdR4HeiYVsYa86NzFy56xLo/GgObIblt9ryZCGNRMulNv03+ckJP+tEGBEi1Qh1Dd6rokQhoYWw==
vuk: XYZ2qitwuEFhaJPLTl3u9KSmsljNOod0+VO4FiLlWlzSuoFj1blZJOAZ27lhXz8x9PncX//5dJE3JkiO9VThSloSwh12uYWeKsCBlNbyXZ0=
kind: Secret
metadata:
creationTimestamp: "2021-04-30T10:03:35Z"
labels:
app.kubernetes.io/component: storageos-api-manager
app.kubernetes.io/managed-by: storageos-operator
app.kubernetes.io/name: storageos
app.kubernetes.io/part-of: storageos
storageos.com/pvc: encrypted
name: storageos-volume-key-2af02d26-4913-4a66-9f4a-0aae9589d8d4
namespace: default
resourceVersion: "15729433"
selfLink: /api/v1/namespaces/default/secrets/storageos-volume-key-2af02d26-4913-4a66-9f4a-0aae9589d8d4
uid: 4ae921d4-1823-4b79-9261-ca1e075fa3b0
type: OpaqueWhen using encryption with Ondat, encryption & decryption keys are fully under the control and management of the consumer. The use of a native Kubernetes construct for key secret management provides the benefit of native integration with Kubernetes KMS providers, therefore providing further compliance and regulation as desired.
Encryption key access, management and availability is a differentiator between StorageOS and other persistent storage services offered by Cloud Providers as each volume has its own individual key whilst being under full control of the end-user.
4. Demonstrating Encryption at Rest with ‘The Great Gatsby’
To fully appreciate Kubernetes Encryption at Rest and the capabilities provided with StorageOS, we make use of ‘F. Scott Fitgerald’s’ famous book, the ‘The Great Gatsby’, a text widely considered as a literary masterpiece that was recreated as a Hollywood film.
The container image spurin/gatby:latest available on DockerHub (source available via GitHub), performs the task of downloading the text of the book to /data/gatsby.txt upon initialisation.
To demonstrate encryption at rest in Kubernetes, we use two StatefulSets, unencrypted and encrypted, each utilising the spurin/gatsby:latest image, with a persistent data volume mounted as /data. When the container starts, the Great Gatsby text will be saved to the StorageOS persistent volumes.
✨ Creating an UnEncrypted and Encrypted StatefulSet - kubectl apply -f- <<EOF
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: unencrypted
spec:
selector:
matchLabels:
app: unencrypted
serviceName: unencrypted
replicas: 1
template:
metadata:
labels:
app: unencrypted
spec:
volumes:
- name: unencrypted
persistentVolumeClaim:
claimName: unencrypted
containers:
- name: unencrypted
image: spurin/gatsby:latest
imagePullPolicy: Always
volumeMounts:
- mountPath: "/data"
name: unencrypted
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: encrypted
spec:
selector:
matchLabels:
app: encrypted
serviceName: encrypted
replicas: 1
template:
metadata:
labels:
app: encrypted
spec:
volumes:
- name: encrypted
persistentVolumeClaim:
claimName: encrypted
containers:
- name: encrypted
image: spurin/gatsby:latest
imagePullPolicy: Always
volumeMounts:
- mountPath: "/data"
name: encrypted
---
EOF
statefulset.apps/unencrypted created
statefulset.apps/encrypted created5. Verifying Data
From the Kubernetes consumption viewpoint, the use of Encryption at Rest is transparent and can be visualised by executing commands on the unencrypted and encrypted pods –
🔍 Checking Gatsby MD5 UnEncrypted - kubectl exec -it unencrypted-0 -- md5sum /data/gatsby.txt
c71f692044dbb4d89df2444ce8cccb35 /data/gatsby.txt
🔍 Checking Gatsby MD5 Encrypted - kubectl exec -it encrypted-0 -- md5sum /data/gatsby.txt
c71f692044dbb4d89df2444ce8cccb35 /data/gatsby.txt
🔍 Checking Gatsby Text UnEncrypted - kubectl exec -it unencrypted-0 -- grep -B1 'put together' /data/gatsby.txt
"They're a rotten crowd," I shouted across the lawn. "You're worth the
whole damn bunch put together."
🔍 Checking Gatsby Text Encrypted - kubectl exec -it encrypted-0 -- grep -B1 'put together' /data/gatsby.txt
"They're a rotten crowd," I shouted across the lawn. "You're worth the
whole damn bunch put together."In both cases they return the same checksum for the data (c71f692044dbb4d89df2444ce8cccb35), they also both match a famous quote from the book.
To truly see the benefits of Encryption at Rest, we need to look behind the scenes. For this, we’ll require access to the Kubernetes worker nodes where Ondat runs.
6. Data at Rest on the Kubernetes Nodes
In this example our unencrypted data resides on storageos-k8s2 under /var/lib/storageos/data/dev1/vol.201427.*.blob (with these files, relating to the unencrypted /data filesystem). If we check for the same text from the book using the unix strings command, we can see the following from a data at rest viewpoint –
root@storageos-k8s2:~# strings /var/lib/storageos/data/dev1/vol.201427.*.blob | grep -B1 -m 1 "put together"
"They're a rotten crowd," I shouted across the lawn. "You're worth the
whole damn bunch put together."By accessing the Kubernetes worker node, we’re operating at an infrastructure level. What could be deemed as sensitive information is both stored and easily accessible for those with access to the underlying components. Should the Kubernetes node be terminated or the infrastructure recycled without due diligence there is a risk of data leakage.
If we were to execute the same command on the volume with encryption at rest (which resides under /var/lib/storageos/data/dev1/vol. 32831.*.blob), the equivalent data is inaccessible, thus safeguarding from data leakage –
root@storageos-k8s2:~# strings /var/lib/storageos/data/dev1/vol.32831.*.blob | grep -B1 -m 1 "put together"
root@storageos-k8s2:~#Running strings against the entire datastore also demonstrates that encryption is in place –
root@storageos-k8s2:~ # strings /var/lib/storageos/data/dev1/vol.32831.*.blob | head -20
M:d~
p;d|
5SB\9 4_B
T1Ot
#lN]8KX
4Rxc
@'{-
b,'I
p92`
(,^M
ya Z=
>w` _
^TTA
T^ Y
fdPG
:N]j
Q&Zn
0v/ ]
SnYu
Hg}Conclusion
For many organisations, data encryption is the barrier that prevents both the use of Kubernetes and/or Cloud adoption for stateful applications. Ondat solves the problem both of encrypted data, and stateful storage.
