Cassandra Architecture FTW

Cassandra Architecture FTW SlideShare Explore You Successfully reported this slideshow.Cassandra Architecture FTWUpcoming SlideShareLoading in …5× 1 Comment 1 Like Statistics Notes slachiewicz No DownloadsNo notes for slide 1. CassandraArchitectureFTW! 2. CassandraArchitectureFTW! 3. Cassandra won because it solved aproblem that hadn’t yet been solved 4. What’s wrong with RDBMS?• Pros• Relational data modeling is wellunderstood• SQL is easy to use andubiquitous• ACID transactions - data integrity• Cons• Scaling is hard• Sharding and replication haveside-effects (performance,reliability, cost)• Denormalize to get performancegainsFor fun: Relational Scaling and the Temple of Gloom 5. The NoSQL Revolution (~2009)• Key-value – Dynamo, Riak, Voldemort, Redis,Memcached• Column-oriented – BigTable, Hbase• Document – Mongo DB, Document DB, CouchDB• Graph – Neo4J, DSE Graph• Multi-model – DataStax Enterprise, CosmosDB• Relational counter-revolution – AWS Aurora, GoogleSpanner 6. Apache Cassandra Architecture Qualities• Distributed,decentralized• Elastic scalability• Commodity hardware?• High performance• High availability / faulttolerant• Tuneable consistency Apache Cassandra ® Apache Software Foundation 7. Two lies and a truth• Cassandra is column-oriented• Cassandra is a schemaless database• Cassandra is eventually consistentThey’re all lies, in a way 8. Problems Cassandra is Especially Good At• Large scale storage• >10s of TB• Lots of writes• Time-series data, IoT• Statistics and analytics• For example, as a Spark datasource• Geographic distribution• Multiple data centersPersonalizationCustomer360RecommendationFraudDetectionInventoryManagementIdentityManagementSecuritySupplyChain 9. A system must be designed for distributionfrom the beginning in order to scale the mosteffectively© DataStax, All Rights Reserved.10 10. Cluster Topology• Organization– Nodes– Racks– Data Centers• Goals– Distribute copies (replicas) forhigh availability– Route queries to nearby nodesfor high performance• Approaches– Gossip– Snitches 11. Clusters and Rings• Organization– Tokens– Token Ring• Goals– Distribute data evenly acrossnodes• Approaches– Partitioners– Virtual nodes 12. Replication and Consistency• Organization– Client (with driver)– Coordinator Node– Replica nodes• Goals– Consistent data even whennodes are down/unresponsive• Approaches– Replication Factor / Strategy– Consistency Level 13. Cassandra works because it leveragesproven distributed system design patterns 14. Wait, who is this guy? 15. Proven Distributed System Techniques• Gossip• Failure detection (Phi)• Partitioning• Leaderless replication• Hinted Handoff• Bloom filters• Consensus algorithms(Paxos)• Log Structured Merge StorageEngine– Memtables– SSTables– Commit log– Compaction– Tombstones• Thrift API• SEDA 16. Log Structured Merge Storage Engine 17. Cassandra’s data model is the key enablerof its high performance and scalability 18. Terminology - Cassandra’s Data Model 19. Terminology – Partition and Clustering Keys 20. Partition and Clustering Key ExampleLabeling:K – partition keyC – clustering keyPRIMARY KEY ((customer_id), contact_time) 21. Cassandra survived because it didn’tover-extend itself 22. Tuneable Consistency and Consistency Levels• ONE, TWO, THREE• Useful for speed• ANY (Write only, use with care)• ALL• Number of nodes to respond =RF• Overly restrictive?• QUORUM• (RF / 2) + 1• Frequently very useful• LOCAL_ONE,LOCAL_QUORUM– Similar to above, but nodes mustbe in local data center• EACH_QUORUM– Quorum of nodes must respondin each data center 23. Strong Consistency vs. Eventual Consistency• Eventual consistency• i.e. W  ONE, R  QUORUM• Use cases• Write heavy• Data not read immediately• Strong Consistency• i.e. W QUORUM, R QUORUM• Use cases• Read after write• Data loading with validation• Strong Consistency Formula• R + W > RF = strong consistency• R: read replica count required byconsistency level• W: the write replica countrequired by consistency level• RF: replication factor• Example: W QUORUM, R QUORUM, RF = 3• 2 + 2 > 3• Implication: all client reads willsee the most recent write 24. Inventory and Tuneable Consistency© DataStax, All Rights Reserved.28Approach Example ScopeQUORUM consistencyfor reads and writesEnsuring latest inventory countsare always readData TierLightweight Transaction Updating inventory counts Data TierLogged BatchWriting to multiple denormalizedtablesData TierRetrying failed callsData synchronization,reservation processingService /Application TierCompensatingprocessesVerifying reservation processing SystemCustomer serviceremediationRebooking, back order,substitutionSystemEventualconsistencyStrongconsistency 25. CoreapplicationdataMicroservices and Polyglot Persistence© DataStax, All Rights Reserved.29Service AServiceBTabular Key-valueService CRelationalDocument GraphServiceDServiceEReferencedataContentCustomerrelationshipdataLegacy, lowvolume data 26. Cassandra will remain viable for many yearsbecause of its extensibility and pluggability 27. SnitchPartitionerCompactionReplicationSecondaryIndexStorageEngine?Auth/Auth 28. Cassandra’s deployment flexibility can’t bematched by cloud vendor database offerings 29. Consistency Level and DeploymentActive-Active, Multi-Region– EACH_QUORUM may limit availability if connection to one region/data center is down– LOCAL_QUORUM, relying on Cassandra to complete writes to remote data centers– QUORUM is a reasonable middle-ground approach 30. Consistency Level and DeploymentSeparate data center for analytics– Writes in ”online DC” – LOCAL_QUORUM• Writes to analytics DC in background• Analytic DC availability/performance decoupled from online DC• EACH_QUORUM, QUORUM overkill, unless “real-time” analytics required– Reads in “analytics DC” – LOCAL_QUORUM• Or even LOCAL_ONE 31. Cassandra is most powerful when combinedwith complementary technologies to form adata platform 32. Spark + Cassandra• Access Cassandra fromSpark via DataStax connectorDataStax is a registered trademarkof DataStax, Inc. and itssubsidiaries in the United Statesand/or other countries.36• Co-locate Spark andCassandraSparkSQLSparkStreamingMLibGraphXSpark RSpark Core EngineDataStax Spark-Cassandra ConnectorCassandra 33. DSE AnalyticsDataStax is a registered trademarkof DataStax, Inc. and itssubsidiaries in the United Statesand/or other countries.ApplicationReal Time OperationsCassandraAnalyticsAnalyticsQueriesYour AnalyticsReal Time ReplicationSingle DSE CusterStreaming, ad-hoc, and batch• High-performance• Workload management• SQL reportingCompared to self-managedSpark cluster:• No ETL• True HA without Zookeeper 34. DSE Core - Certified Apache Cassandra• The best distribution of Apache Cassandra™• Production certified Cassandra• Performance improvements• Advanced Security• Multi-tenancy through row-level access control• Advanced Replication• Great for retail and IoT use casesDataStax is a registered trademark of DataStax, Inc. and its subsidiaries in the United Statesand/or other countries. 35. https://github.com/killrvideoKillrVideo – a reference application 36. https://academy.datastax.comDataStax Academy – a place to learn 37. 41Contactjeff.carpenter@datastax.com@jscarpjeffreyscarpentermedium.com/@jscarp Recommended Patterns for Persistence and Streaming in Microservice ArchitecturesJeffrey Carpenter Creating a Python Microservice Tier in Four Sprints with Cassandra, Kafka, an...Jeffrey Carpenter Deploying Cassandra Multi-cloudJeffrey Carpenter Getting the Most Out of CassandraJeffrey Carpenter Data Model Meets WorldJeffrey Carpenter Can My Inventory Survive Eventual Consistency?Jeffrey Carpenter Data Modeling for Microservices with Cassandra and SparkJeffrey Carpenter About Blog Terms Privacy Copyright × Public clipboards featuring this slideNo public clipboards found for this slideSelect another clipboard ×Looks like you’ve clipped this slide to already.Create a clipboardYou just clipped your first slide! Clipping is a handy way to collect important slides you want to go back to later. Now customize the name of a clipboard to store your clips. Description Visibility Others can see my Clipboard