Persistent Storage with OpenEBS on Kubernetes

Today we will explore persistent storage for Cassandra on Kubernetes with OpenEBS.What will be doingWe will be deploying a k3s distribution of kubernetes on Civo, deploy a cassandra cluster, write some data to the cluster and then test the persistence by deleting all the cassandra podsTaken from their Github Page, OpenEBS describes them as the leading open-source container attached storage, built using cloud native architecture, simplifies running stateful applications on kubernetes.Deploy KubernetesCreate a new civo k3s cluster with 3 nodes:$ civo kubernetes create demo-cluster --size=g2.small --nodes=3 --waitAppend the kubernetes config to your kubeconfig file:$ civo kubernetes config demo-cluster --saveMerged config into ~/.kube/configSwitch the context to the new cluster:$ kubectx demo-clusterSwitched to context "demo-cluster".Install OpenEBSDeploy OpenEBS to your Kubernetes cluster by applying this, which can be found from their github page:$ kubectl apply -f https://openebs.github.io/charts/openebs-operator.yamlGive it some time for all the pods to check in and then have a look if all the pods under the openebs namespace is ready:$ kubectl get pods -n openebsNAME READY STATUS RESTARTS AGEopenebs-provisioner-c68bfd6d4-5n7kk 1/1 Running 0 5m52sopenebs-ndm-df25v 1/1 Running 0 5m51sopenebs-snapshot-operator-7ffd685677-h8jpf 2/2 Running 0 5m52sopenebs-admission-server-889d78f96-t64m6 1/1 Running 0 5m50sopenebs-ndm-44k6n 1/1 Running 0 5m51sopenebs-ndm-hpmg8 1/1 Running 0 5m51sopenebs-localpv-provisioner-67bddc8568-shkqr 1/1 Running 0 5m49sopenebs-ndm-operator-5db67cd5bb-mfg5m 1/1 Running 0 5m50smaya-apiserver-7f664b95bb-l87bm 1/1 Running 0 5m53sWe can see a list of storage classes that comes with OpenEBS:$ kubectl get scNAME PROVISIONER AGElocal-path (default) rancher.io/local-path 28mopenebs-jiva-default openebs.io/provisioner-iscsi 5m43sopenebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter 5m41sopenebs-hostpath openebs.io/local 5m41sopenebs-device openebs.io/local 5m41sDeploy CassandraDeploy a Cassandra Service to your Kubernetes Cluster:$ kubectl apply -f https://raw.githubusercontent.com/ruanbekker/blog-assets/master/civo.com-openebs-kubernetes/manifests/cassandra/cassandra-service.yamlThe Cassandra Stateful Set has 3 replicas which is defined in the manifest as replicas: 3. I am also using the Jiva storage engine defined as volume.beta.kubernetes.io/storage-class: openebs-jiva-default which is best suited for running replicated block storage on nodes that make use of ephemeral storage on the Kubernetes worker nodes.Let's deploy Cassandra to your Kubernetes Cluster:$ kubectl apply -f https://raw.githubusercontent.com/ruanbekker/blog-assets/master/civo.com-openebs-kubernetes/manifests/cassandra/cassandra-statefulset.yamlGive it some time for all the pods to check in, then have a look at the pods:$ kubectl get podsNAME READY STATUS RESTARTS AGEpvc-b29a4e4b-20d7-4fd4-83d9-a155ab5c1a93-rep-ffb5b9f9-7nwfl 1/1 Running 0 5m39spvc-b29a4e4b-20d7-4fd4-83d9-a155ab5c1a93-rep-ffb5b9f9-wr4hl 1/1 Running 0 5m39spvc-b29a4e4b-20d7-4fd4-83d9-a155ab5c1a93-ctrl-6f95fd555-ldd86 2/2 Running 0 5m53spvc-b29a4e4b-20d7-4fd4-83d9-a155ab5c1a93-rep-ffb5b9f9-2v75w 1/1 Running 1 5m39scassandra-0 1/1 Running 0 5m53spvc-2fe5a138-6c1d-4bcd-a0e4-e653793abea2-ctrl-bc4769cb4-c8stc 2/2 Running 0 3m40spvc-2fe5a138-6c1d-4bcd-a0e4-e653793abea2-rep-7565997776-w2gqs 1/1 Running 0 3m37spvc-2fe5a138-6c1d-4bcd-a0e4-e653793abea2-rep-7565997776-jk6c2 1/1 Running 0 3m37spvc-2fe5a138-6c1d-4bcd-a0e4-e653793abea2-rep-7565997776-qq9bb 1/1 Running 1 3m37scassandra-1 1/1 Running 0 3m40spvc-a1ff3db9-d2fd-43a8-9095-c3529fe2e456-ctrl-55887dc66f-zjbx8 2/2 Running 0 117spvc-a1ff3db9-d2fd-43a8-9095-c3529fe2e456-rep-7fbc8785b8-cqvdb 1/1 Running 0 114spvc-a1ff3db9-d2fd-43a8-9095-c3529fe2e456-rep-7fbc8785b8-hclgs 1/1 Running 0 114spvc-a1ff3db9-d2fd-43a8-9095-c3529fe2e456-rep-7fbc8785b8-85nqt 1/1 Running 1 114scassandra-2 0/1 Running 0 118sWhen we look at our persistent volumes:$ kubectl get pvNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEpvc-b29a4e4b-20d7-4fd4-83d9-a155ab5c1a93 5G RWO Delete Bound default/cassandra-data-cassandra-0 openebs-jiva-default 12mpvc-2fe5a138-6c1d-4bcd-a0e4-e653793abea2 5G RWO Delete Bound default/cassandra-data-cassandra-1 openebs-jiva-default 10mpvc-a1ff3db9-d2fd-43a8-9095-c3529fe2e456 5G RWO Delete Bound default/cassandra-data-cassandra-2 openebs-jiva-default 8m49sAnd our persistent volume claims:$ kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEcassandra-data-cassandra-0 Bound pvc-b29a4e4b-20d7-4fd4-83d9-a155ab5c1a93 5G RWO openebs-jiva-default 13mcassandra-data-cassandra-1 Bound pvc-2fe5a138-6c1d-4bcd-a0e4-e653793abea2 5G RWO openebs-jiva-default 10mcassandra-data-cassandra-2 Bound pvc-a1ff3db9-d2fd-43a8-9095-c3529fe2e456 5G RWO openebs-jiva-default 9m11sInteract with CassandraView the cluster status:$ kubectl exec cassandra-0 -- nodetool statusDatacenter: dc1-civo-k3s========================Status=Up/Down|/ State=Normal/Leaving/Joining/Moving-- Address Load Tokens Owns (effective) Host ID RackUN 192.168.1.9 83.16 KiB 32 63.4% f3debec3-77c7-438e-bf86-38630fa01c14 rack1-civo-k3sUN 192.168.0.10 65.6 KiB 32 66.1% 3f78ec71-7485-477a-a095-44aa8fade0bd rack1-civo-k3sUN 192.168.2.12 84.73 KiB 32 70.5% 0160da34-5975-43d9-ab89-4c92cfe4ceeb rack1-civo-k3sNow let's connect to our cassandra cluster:$ kubectl exec -it cassandra-0 -- cqlsh cassandra 9042 --cqlversion="3.4.2"Connected to civo-k3s at cassandra:9042.[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]Use HELP for help.cqlsh>Then create a keyspace, table and write some dummy data to our cluster:cqlsh> create keyspace test with replication = {'class': 'SimpleStrategy', 'replication_factor': 2 };cqlsh> use test;cqlsh:test> create table people (id uuid, name varchar, age int, PRIMARY KEY ((id), name));cqlsh:test> insert into people (id, name, age) values(uuid(), 'ruan', 28);cqlsh:test> insert into people (id, name, age) values(uuid(), 'samantha', 25);cqlsh:test> insert into people (id, name, age) values(uuid(), 'stefan', 29);cqlsh:test> insert into people (id, name, age) values(uuid(), 'james', 25);cqlsh:test> insert into people (id, name, age) values(uuid(), 'michelle', 30);cqlsh:test> insert into people (id, name, age) values(uuid(), 'tim', 32);Now let's read the data:cqlsh:test> select * from people; id | name | age--------------------------------------+----------+----- e3a2261d-77d3-4ed6-9daf-51e65ccf618f | tim | 32 594b24ee-e5bc-44cf-87cf-afed48a74df9 | samantha | 25 ca65c207-8dd0-4dc0-99e3-1a0301128921 | michelle | 30 1eeb2b77-57d0-4e10-b5d7-6dc58c43006e | james | 25 3cc12f26-395d-41d1-a954-f80f2c2ff88d | ruan | 28 755d52f2-5a27-4431-8591-7d34bacf7bee | stefan | 29(6 rows)Test Data PersistenceNow, it's time to delete our pods and see if our data is persisted. First we will look at our pods to determine how long they are running and on which node they are running:$ kubectl get pods --selector app=cassandra -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScassandra-0 1/1 Running 0 35m 192.168.0.10 kube-master-75a2 <none> <none>cassandra-1 1/1 Running 0 33m 192.168.1.9 kube-node-5d0a <none> <none>cassandra-2 1/1 Running 0 30m 192.168.2.12 kube-node-aca8 <none> <none>Let's delete all our cassandra pods:$ kubectl delete pod/cassandra-{0..2}pod "cassandra-0" deletedpod "cassandra-1" deletedpod "cassandra-2" deletedAs we can see the first pod is busy starting:$ kubectl get pods --selector app=cassandra -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScassandra-0 0/1 Running 0 37s 192.168.2.13 kube-node-aca8 <none> <none>Give it some time to allow all three pods to check in:$ kubectl get pods --selector app=cassandra -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScassandra-0 1/1 Running 0 4m36s 192.168.2.13 kube-node-aca8 <none> <none>cassandra-1 1/1 Running 0 3m24s 192.168.0.17 kube-master-75a2 <none> <none>cassandra-2 1/1 Running 0 2m3s 192.168.1.12 kube-node-5d0a <none> <none>Right, now that all pods has checked in, let's connect to our cluster and see if the data is still there:$ kubectl exec -it cassandra-0 -- cqlsh cassandra 9042 --cqlversion="3.4.2"Connected to civo-k3s at cassandra:9042.[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]Use HELP for help.cqlsh> use test;cqlsh:test> select * from people; id | name | age--------------------------------------+----------+----- e3a2261d-77d3-4ed6-9daf-51e65ccf618f | tim | 32 594b24ee-e5bc-44cf-87cf-afed48a74df9 | samantha | 25 ca65c207-8dd0-4dc0-99e3-1a0301128921 | michelle | 30 1eeb2b77-57d0-4e10-b5d7-6dc58c43006e | james | 25 3cc12f26-395d-41d1-a954-f80f2c2ff88d | ruan | 28 755d52f2-5a27-4431-8591-7d34bacf7bee | stefan | 29(6 rows)cqlsh:test> exit;And as you can see the data is being persisted.Thank YouI am super impressed with OpenEBS, have a look at their docs and also have a look at their examples on their github page.Let me know what you think. If you liked my content, feel free to visit me at ruan.dev or follow me on twitter at @ruanbekker