GoogleCloudPlatform/gke-stateful-applications-demo

Table of ContentsGoogle Kubernetes Engine Stateful Application DemoIntroductionArchitectureArchitecture DiagramKubernetes Manifests and ControllersStatefulSetsPersistent Storage with Persistent VolumesPod Disruptions BudgetsPod SchedulingDNS and Headless ServicesCassandra ContainerPrerequisitesRun Demo in a Google Cloud ShellSupported Operating SystemsToolsVersionsDeployment StepsValidationUsing CassandaraLaunch a Cassandra containerConnect to the ring with cqlshcqlsh commandCreate a keyspaceDescribe the keyspaceUse the new keyspaceCreate a tableSee the newly created tableInsert data into the tableSelect the data from the tableExitDelete the deploymentTear DownTroubleshootingRelevant MaterialsIntroductionThis proof of concept deploys aKubernetes EngineCluster and then installs an Apache Cassandradatabase running on that cluster.Various scripts are contained within this project that provide push buttoncreation, validation, and deletion of the Cassandra(C*) database and KubernetesEngine cluster.Apache Cassandra was chosen for various reasons. These reasons include thatout of the box Cassandra functions well as a cloud native database. Moreover,this POC continues the work started with the Kubernetes example, and the blogpost: Thousand Instances of Cassandra using Kubernetes Pet Set.When running a database using Kubernetes Engine, as an operator you need to be experienced with Kubernetes Engineand the datasource that you are running. If you are able to use a hostedsolution then it is recommended that you use the hosted solution. On the otherhand, if you are not able to run a datasource in a hosted solution then hostingit is Kubernetes Engine is doable. The challenge is not Kubernetes or Kubernetes Engine, but the challengeis migrating and operating a database in a container based system that can dothings like moving pods/containers between nodes.ArchitectureThe intricacy of the running stateful datastores via Kubernetes Engine or K8s lies withthe various Kubernetes manifests and pod configurations used. A container thathas been customized to run as a pod is used as well. This demo includes multipleconfigured manifests and a specialized container for Cassandra.Many people, including Kelsey Hightower, have talked about how running statefuldatestores inside of K8s is not trivial, and frankly quite complicated. If youcan run a database on a managed service, do it. Let someone else wake up at 2am to fix your database.There are two possibilities when you run a stateful datastore on K8s.You are very experienced K8s user and know exactly what is around the corner.You have no idea what is around the corner, and you are going to learn veryfast.Architecture DiagramThe following diagram represents a Cassandra cluster deployed on Kubernetes Engine.Kubernetes Manifests and ControllersVarious manifests and controllers are utilized to install Cassandra. Thefollowing sections outline the different types used.StatefulSetsStatefulSets is the controller type that is used to install a Cassandra Ring onKubernetes Engine. This controller manages the installation and scaling of a set of Pods andprovides various features:Stable, unique network identifiers.Stable, persistent storage.Ordered, graceful deployment and scaling.Ordered, graceful deletion and termination.Ordered, automated rolling updates.Automated creation of storage volumes.Stable guaranteed storage.Like a Deployment, a StatefulSet manages Pods that are based on an identicalcontainer spec. Unlike a Deployment, a StatefulSet maintains a sticky identityfor each of their Pods. These pods are created from the same spec, but are notinterchangeable: each has a persistent identifier that it maintains across anyrescheduling.Cassandra utilizes all of these features in order to run on Kubernetes Engine.Find more information here.Persistent Storage with Persistent VolumesWith a stateful datasource storage is required.A PersistentVolume (PV) is a piece of storage in the cluster that has beenprovisioned by an administrator, or automatically by a Stateful Set. It is aresource in the cluster just like a node is a cluster resource. PVs are volumeplugins like Volumes, but have a lifecycle independent of any individual podthat uses the PV.A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similarto a pod. Pods consume node resources and PVCs consume PV resources. Pods canrequest specific levels of resources (CPU and Memory). Claims can requestspecific size and access modes (e.g., can be mounted once read/write or manytimes read-only).Find more information here.Pod Disruptions BudgetsAn Application Owner can create a PodDisruptionBudget object (PDB) for eachapplication. A PDB limits the number pods of a replicated application that aredown simultaneously from voluntary disruptions. For example, with Cassandrawe need to ensure that the number of replicas running is never brought belowthe number needed for a quorum.Find more information visit here.Pod SchedulingThis proof of concept utilizes advanced scheduling for pod placement. Bothscheduling anti-affinity rules and taints are used to instruct the K8s schedulerwhere to launch C* pods.Anti-Affinity rules instruct the scheduler to use preferred or required podplacement. Preferred rules provide a weight value that impacts the schedulingalgorithm to not schedule C* pods on the same node. While required rules forcethe scheduler to not schedule C* pods on the same node. Not having multipleCassandra pods on the same node increases fault tolerance, as when you lose anode you only lose one pod. But then you need to have an excess of nodes becausea pod will not reschedule. Preferred does not provide the same level of highavailability, but will allow pods to reschedule on existing nodes if headroomexists.Find more information visit here.Taints prevent pods to be scheduled on a node, unless the pod's manifestcontains the required toleration. The typical use case for this is to targeta specific node pool for say Cassandra pods. For instance, often larger machinetypes are required for C*, and adding taints to those nodes ensure thatCassandra pods will only be scheduled on the nodes.Find more information visit here.DNS and Headless ServicesThe Cassandra ring that is installed as part of this demo is named "cassandra".The name of the service "cassandra" creates the DNS name, and the tld is"default.svc.cluster.local", where "default" is the name of the namespace. Withan application you would connect to at least three of the nodes to provide an HAconnection with a client. The name of the nodes, for example would be:cassandra-0.cassandra.default.svc.cluster.localcassandra-1.cassandra.default.svc.cluster.localcassandra-1.cassandra.default.svc.cluster.localFor more information visit here.Cassandra ContainerWithin this demo we have included a container that is specifically built forrunning on Kubernetes. Features:OSS maintained base container that is used by K8s OSS projectCassandra is installed from a tarballContainer size has been reducedA readiness and liveness probes are built-inPrometheus is installed, but the port is not exposed in the manifest bydefaultdumb-init is used to handle signal processingA logging encoder is provided for JSON loggingDatacenter configurationJMX configurationMultiple ENV variables are exposed for manifest based configurationFor more information about the ENV variables that have been exposed seecontainer/README.md,container/files/run.sh and also themanifest.This container is hosted by Google Professional Services.PrerequisitesA Google Cloud account and project is required for this demo. The project musthave the proper quota to run a Kubernetes Engine cluster with at least3 n1-standard-4 and 3 n1-standard-1 nodes. Additionally, the project must haveat least 30 Compute Engine API CPUs and 12 Compute Engine API In-use IP Addresses.Access to an existing Google Cloud project with the Kubernetes Engine service enabledIf you do not have a Google Cloud account please signup for a free trialhere.How to check your account's quota is documented here: quotas.Run Demo in a Google Cloud ShellClick the button below to run the demo in a Google Cloud Shell.All the tools for the demo are installed. When using Cloud Shell execute the followingcommand in order to setup gcloud cli. When executing this command please setup your regionand zone.gcloud initSupported Operating SystemsThis project will run on macOS, or in a Google Cloud Shell.ToolsWhen not using Cloud Shell, the following tools are required.Google Cloud SDK version >= 204.0.0gcloud clikubectl matching the latest GKE versiondocker (used to build container, you can use hosted container)More recent versions of all the tools may function, please feel free to file anissue if you encounter problems with newer versions.Install Cloud SDKThe Google Cloud SDK is used to interact with your GCP resources.Installation instructions for multiple platforms are available online.Install kubectl CLIThe kubectl CLI is used to interteract with both Kubernetes Engine and kubernetes in general.Installation instructionsfor multiple platforms are available online.Configure AuthenticationThe script will execute against your GCP environment and it will use your personal account to build out these resources. To setup the default account the script will use, run the following command to select the appropriate account:gcloud auth loginDeployment StepsTo deploy the demo execute the following commands.git clone https://github.com/GoogleCloudPlatform/gke-stateful-applications-democd gke-stateful-applications-demo./create.sh -c my-cluster-1Replace the text 'my-cluster-1' the name of the cluster that you would like tocreate.The create script will output the following message when complete.NAME MACHINE_TYPE DISK_SIZE_GB NODE_VERSIONnodepool-cassdemo-2 n1-standard-1 100 1.10.2-gke.3The script will:create a new Kubernetes Engine cluster in your default ZONE, VPC and network.install multiple Kubernetes manifests that can be found in themanifests directory.Because we are creating a cluster of 6 nodes, the cluster may go into a'RECONCILING' status as the control plane's instance size may be increased.Use the following command to view the current status of the cluster.gcloud container clusters listAn example of the output while the cluster is reconciling.NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUSmy-cluster-1 us-central1-a 1.10.2-gke.3 35.184.70.165 n1-standard-4 1.10.2-gke.3 6 RECONCILINGThe status will change to RUNNING once the masters have been updated.ValidationThe following script will validate that the demo is deployed correctly:./validate.sh -c my-cluster-1Replace the text 'my-cluster-1' the name of the cluster that you would like tovalidate.The validation script uses kubectl rollout status to test if the rollout iscomplete. If the cluster is in 'RECONCILING' state this script will fail aswell.If the script fails it will output:Validation Failed: Statefulset has not been deployedIf the script passes if will output:Validation Passed: the Statefulset has been deployedUsing CassandaraThese commands exercise the Cassandra cluster.Launch a Cassandra containerThese kubectl commands launch a K8s deployment, wait for the deployment,and exec into the shell.kubectl run cass-dev --image gcr.io/pso-examples/cassandra:3.11.3-cqlsh-v22 --command -- /bin/bash -c "tail -f /dev/null"kubectl rollout status deployment cass-devkubectl exec $(kubectl get po --no-headers | grep cass-dev | awk '{print $1}') -it -- /bin/bashThis will launch a bash prompt for the next steps.Connect to the ring with cqlshcqlsh command/usr/local/apache-cassandra/bin/cqlsh cassandra-0.cassandra.default.svc.cluster.localYou will now be using the interactive cqlsh shell.The output of the command:Connected to K8Demo at cassandra-0.cassandra.default.svc.cluster.local:9042.[cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]Use HELP for help.cqlsh>Create a keyspaceCREATE KEYSPACE greek_monsters WITH REPLICATION = { 'class' : 'SimpleStrategy' , 'replication_factor' : 3 };The user prompt is shown if the command is successful.Describe the keyspaceDESC greek_monsters;Example output:CREATE KEYSPACE greek_monsters WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;cqlsh>Use the new keyspaceUSE greek_monsters;The prompt will now include the keyspace that is selected.cqlsh:greek_monsters>Create a tableCREATE TABLE monsters (pet_id timeuuid, name text, description text, PRIMARY KEY (pet_id));The user prompt is shown if the command is successful.See the newly created tableDESCRIBE TABLES;This command will output:monsterscqlsh:greek_monsters>Insert data into the tableINSERT INTO monsters (pet_id,name,description) VALUES (now(), 'Cerberus (Hellhound)','The three-headed giant hound, that guarded the gates of the Underworld.');INSERT INTO monsters (pet_id,name,description) VALUES (now(), 'Orthrus','A two-headed dog, brother of Cerberus, slain by Heracles.');INSERT INTO monsters (pet_id,name,description) VALUES (now(), 'Argos','Odysseus faithful dog, known for his speed, strength and his superior tracking skills.');INSERT INTO monsters (pet_id,name,description) VALUES (now(), 'Golden Dog','A dog which guarded the infant god Zeus.');INSERT INTO monsters (pet_id,name,description) VALUES (now(), 'Guard Dog of Hephaestus Temple','The temple of Hephaestus at Mount Etna was guarded by a pack of sacred dogs.');INSERT INTO monsters (pet_id,name,description) VALUES (now(), 'Laelaps',' female dog destined always to catch its prey.');INSERT INTO monsters (pet_id,name,description) VALUES (now(), 'Maera','The hound of Erigone, daughter of Icarius of Athens.');The user prompt is shown if the command is successful.Select the data from the tableSELECT * from monsters ;Example output:pet_id | description | name--------------------------------------+----------------------------------------------------------------------------------------+--------------------------------ca3d9f20-6a89-11e8-a0dd-114bb9e30b07 | Odysseus faithful dog, known for his speed, strength and his superior tracking skills. | Argosca3c3f90-6a89-11e8-a0dd-114bb9e30b07 | A two-headed dog, brother of Cerberus, slain by Heracles. | Orthrusca42f650-6a89-11e8-a0dd-114bb9e30b07 | The hound of Erigone, daughter of Icarius of Athens. | Maeraca3f4cd0-6a89-11e8-a0dd-114bb9e30b07 | A dog which guarded the infant god Zeus. | Golden Dogca41bdd0-6a89-11e8-a0dd-114bb9e30b07 | female dog destined always to catch its prey. | Laelapsca40ac60-6a89-11e8-a0dd-114bb9e30b07 | The temple of Hephaestus at Mount Etna was guarded by a pack of sacred dogs. | Guard Dog of Hephaestus Templeca3a6ad0-6a89-11e8-a0dd-114bb9e30b07 | The three-headed giant hound, that guarded the gates of the Underworld. | Cerberus (Hellhound)ExitExit the pod with the following commands:exitexitThis will return you to your command line.Delete the deploymentExecute the following command to remove the deployment.kubectl delete deployment cass-devThe following message is displayed:deployment "cass-dev" deletedTear DownThe following script will tear down the Cassandra cluster and remove the Kubernetes Enginecluster../delete.sh -c my-cluster-1This output will change depending on the cluster name. In this example the name"my-cluster-1" was used.Replace the text 'my-cluster-1' the name of the cluster that you would like tovalidate. The last lines of the output will be:persistentvolumeclaim "cassandra-data-cassandra-0" deletedpersistentvolumeclaim "cassandra-data-cassandra-1" deletedpersistentvolumeclaim "cassandra-data-cassandra-2" deletedDeleting clusterThe tear down script removes all of the components in the manifests directory,and it also destroys the cluster. This script also waits 60 seconds beforeit removes the PVC storage components.TroubleshootingSince we are creating two nodepools in this demo, the cluster may upgradethe control plane and it may go to a "RECONCILING" state. Give the clustersome time and it will scale. The validate script will fail during this time.Run "gcloud container clusters list" command to check the cluster status.ClusterIf you get errors about quotas, please increase your quota in the project.See here for more details.A great diagnostic command to run is simply kubectl get pods. It will showthe running pods.Initially, the cluster may show as "RUNNING" but then go into a "RECONCILING" stateThe symptom will be timeouts when running kubectl get podsUse the "gcloud container clusters list" command to check the latest state, and wait until it changes back to "RUNNING"Relevant MaterialsThousand Instances of CassandraStateful SetsPersistent VolumesPod Disruption BudgetsPod SchedulingTaints and TolerationsHeadless ServicesGoogle Cloud QuotasSignup for Google CloudGoogle Cloud ShellThis is not an officially supported Google product