Bulk Loading Data into Cassandra

Bulk Loading Data into Cassandra SlideShare Explore You Successfully reported this slideshow.Bulk Loading Data into CassandraUpcoming SlideShareLoading in …5× 3 Comments 26 Likes Statistics Notes Padmabushan Reddy , Engineer at Qolsys 승완 조 , FusionData의 Intern at 홍익대학교 Jie Yao , 高级架构师 at 唯品会 at 唯品会 Steve Min , Data Engineer at Netmarble Games at Netmarble Games Steve Min , Data Engineer at Netmarble Games at Netmarble Games Show More No DownloadsNo notes for slide 1. Planet Cassandra 2014Bulk-Loading Data into CassandraPatricia Gorla@patriciagorlaCassandra Consultantwww.thelastpickle.com 2. About Us•Work with clients to deliver and improveApache Cassandra services•Apache Cassandra committer, DatastaxMVP, Hector maintainer, Apache Usergridcommitter•Based in New Zealand & USA 3. Why is bulk loading useful?•Performance tests 4. Why is bulk loading useful?•Performance tests•Migrating historical data 5. Why is bulk loading useful?•Performance tests•Migrating historical data•Changing topologies 6. !•How Data is Stored•Case Studies - Generating Dummy Data - Backfilling Historical Data - Changing Topologies•Conclusion 7. Cassandra Write Pathwrite[0] 8. Cassandra Write Path•write[0]Writes written to both the commit log andmemtable.commitlogmemtable 9. Cassandra Write Path••write[0]Writes written to both the commit log andmemtable.Memtable is sorted.commitlogmemtable 10. Cassandra Write Path•write[0]Memtable flushed out to sstables.commitlogmemtablesstable[0]sstable[2]sstable[1] 11. Cassandra Write Path•write[0]Compaction helps keep the read latencylow.commitlogmemtablesstable[0]sstable[2]sstable[1]sstable[n] 12. Sorted String Tablesmykeyspace-mycf-jb-1-CompressionInfo.dbmykeyspace-mycf-jb-1-Data.dbmykeyspace-mycf-jb-1-Filter.dbmykeyspace-mycf-jb-1-Index.dbmykeyspace-mycf-jb-1-Statistics.dbmykeyspace-mycf-jb-1-Summary.dbmykeyspace-mycf-jb-1-TOC.txt 13. Sorted String Tablesmykeyspace-mycf-jb-1-CompressionInfo.dbmykeyspace-mycf-jb-1-Data.dbmykeyspace-mycf-jb-1-Filter.dbmykeyspace-mycf-jb-1-Index.dbmykeyspace-mycf-jb-1-Statistics.dbmykeyspace-mycf-jb-1-Summary.dbmykeyspace-mycf-jb-1-TOC.txtContains all data needed to regenerate components 14. Sorted String Tablesmykeyspace-mycf-jb-1-CompressionInfo.dbmykeyspace-mycf-jb-1-Data.dbmykeyspace-mycf-jb-1-Filter.dbmykeyspace-mycf-jb-1-Index.dbmykeyspace-mycf-jb-1-Statistics.dbmykeyspace-mycf-jb-1-Summary.dbmykeyspace-mycf-jb-1-TOC.txtIndex of row keys 15. Sorted String Tablesmykeyspace-mycf-jb-1-CompressionInfo.dbmykeyspace-mycf-jb-1-Data.dbmykeyspace-mycf-jb-1-Filter.dbmykeyspace-mycf-jb-1-Index.dbmykeyspace-mycf-jb-1-Statistics.dbmykeyspace-mycf-jb-1-Summary.dbmykeyspace-mycf-jb-1-TOC.txtIndex summary from Index.db file 16. Sorted String Tablesmykeyspace-mycf-jb-1-CompressionInfo.dbmykeyspace-mycf-jb-1-Data.dbmykeyspace-mycf-jb-1-Filter.dbmykeyspace-mycf-jb-1-Index.dbmykeyspace-mycf-jb-1-Statistics.dbmykeyspace-mycf-jb-1-Summary.dbmykeyspace-mycf-jb-1-TOC.txtBloom filter over sstable 17. Sorted String Tablesmykeyspace-mycf-jb-1-CompressionInfo.dbmykeyspace-mycf-jb-1-Data.dbmykeyspace-mycf-jb-1-Filter.dbmykeyspace-mycf-jb-1-Index.dbmykeyspace-mycf-jb-1-Statistics.dbmykeyspace-mycf-jb-1-Summary.dbmykeyspace-mycf-jb-1-TOC.txtTable of contents of all components 18. !•How Data is Stored•Case Studies - Generating Dummy Data - Backfilling Historical Data - Changing Topologies•Conclusion 19. create keyspace testwith placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'and strategy_options = {replication_factor:1};!create column family testwith comparator = 'AsciiType'and default_validation_class = 'AsciiType'and key_validation_class = 'AsciiType';Set up keyspace and column family 20. AbstractSSTableSimpleWriter writer = new SSTableSimpleUnsortedWriter(directory,partitioner,keyspace,columnFamily,AsciiType.instance,null, // subcomparator for super columnssize_per_sstable_mb);SStableGen.java 21. AbstractSSTableSimpleWriter writer = new SSTableSimpleUnsortedWriter(directory,partitioner,keyspace,columnFamily,AsciiType.instance,null, // subcomparator for super columnssize_per_sstable_mb);SStableGen.java 22. AbstractSSTableSimpleWriter writer = new SSTableSimpleUnsortedWriter(directory,partitioner,keyspace,columnFamily,AsciiType.instance,null, // subcomparator for super columnssize_per_sstable_mb);SStableGen.java 23. ByteBuffer randomBytes = ByteBufferUtil.bytes(randomAscii(1024));KeyGenerator keyGen = new KeyGenerator();long dataSize = 0;writer = new SSTableSimpleUnsortedWriter(…);while (dataSize < max_data_bytes) {writer.newRow(key);for (int j=0; j<num_cols; j++) {ByteBuffer colName = ByteBufferUtil.bytes("col_" + j);ByteBuffer colValue = ByteBuffer.wrap(new byte[20]);randomBytes.get(colValue.array());colValue.position(0);writer.addColumn(colName, colValue, timestamp);if (randomBytes.remaining() < colValue.limit()) {randomBytes.position(0);}else {randomBytes.position(randomBytes.position() + colValue.limit());}}}} 24. patricia@dev:~/../data$total 64-rw-r--r-- 1 patricia-rw-r--r-- 1 patricia-rw-r--r-- 1 patricia-rw-r--r-- 1 patricia-rw-r--r-- 1 patricia-rw-r--r-- 1 patricia-rw-r--r-- 1 patricials -lh mykeyspace/mycfstaffstaffstaffstaffstaffstaffstaff43B79K16B36B4.3K80B79BFebFebFebFebFebFebFeb222222215:3115:3115:3115:3115:3115:3115:31mykeyspace-mycf-jb-1-CompressionInfo.dbmykeyspace-mycf-jb-1-Data.dbmykeyspace-mycf-jb-1-Filter.dbmykeyspace-mycf-jb-1-Index.dbmykeyspace-mycf-jb-1-Statistics.dbmykeyspace-mycf-jb-1-Summary.dbmykeyspace-mycf-jb-1-TOC.txtExamining sstable output 25. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/…/cassandra-2.0.4$ bin/sstableloader mykeyspace/mycf -d localhostStreaming relevant part of mykeyspace/mycf/mykeyspace-mycf-ic-1-Data.db to [/127.0.0.1]progress: [/127.0.0.1 1/1 (100)] [total: 100 - 0MB/s (avg: 0MB/s)] 26. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/…/cassandra-2.0.4$ bin/sstableloader mykeyspace/mycf -d localhostStreaming relevant part of mykeyspace/mycf/mykeyspace-mycf-ic-1-Data.db to [/127.0.0.1]progress: [/127.0.0.1 1/1 (100)] [total: 100 - 0MB/s (avg: 0MB/s)] 27. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/…/cassandra-2.0.4$ bin/sstableloader mykeyspace/mycf -d localhostStreaming relevant part of mykeyspace/mycf/mykeyspace-mycf-ic-1-Data.db to [/127.0.0.1]progress: [/127.0.0.1 1/1 (100)] [total: 100 - 0MB/s (avg: 0MB/s)] 28. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/…/cassandra-2.0.4$ bin/sstableloader mykeyspace/mycf -d localhostStreaming relevant part of mykeyspace/mycf/mykeyspace-mycf-ic-1-Data.db to [/127.0.0.1]progress: [/127.0.0.1 1/1 (100)] [total: 100 - 0MB/s (avg: 0MB/s)] 29. $ bin/sstableloader Keyspace1/ColFam1•Run command on separate server 30. $ bin/sstableloader Keyspace1/ColFam1•Run command on separate server•Throttle command 31. $ bin/sstableloader Keyspace1/ColFam1•Run command on separate server•Throttle command•Parallelise processes 32. !•How Data is Stored•Case Studies - Generating Dummy Data - Backfilling Historical Data - Changing Topologies•Conclusion 33. // list of orders by usercustomerOrders = new SSTableSimpleUnsortedWriter(…);// orders by order idorders = new SSTableSimpleUnsortedWriter(…);!// assume orders are in date orderfor (Order order : oldOrders) {customerOrders.newRow(ByteBufferUtil.bytes(order.customerId));customerOrders.addColumn(ByteBufferUtil.bytes(order.orderId), ByBufferUtil.EMPTY_BYTE_BUFFER,timestamp);!orders.newRow(ByteBufferUtil.bytes(order.userId));orders.addColumn(ByteBufferUtil.bytes(“customer_id), ByteBufferUtil.bytes(order.customerId),timestamp);orders.addColumn(ByteBufferUtil.bytes(“date), ByteBufferUtil.bytes(order.date), timestamp);orders.addColumn(ByteBufferUtil.bytes(“total), ByteBufferUtil.bytes(order.total), timestamp);}!customerOrders.close()orders.close() 34. // list of orders by usercustomerOrders = new SSTableSimpleUnsortedWriter(…);// orders by order idorders = new SSTableSimpleUnsortedWriter(…);!// assume orders are in date orderfor (Order order : oldOrders) {customerOrders.newRow(ByteBufferUtil.bytes(order.customerId));customerOrders.addColumn(ByteBufferUtil.bytes(order.orderId), ByBufferUtil.EMPTY_BYTE_BUFFER,timestamp);!orders.newRow(ByteBufferUtil.bytes(order.userId));orders.addColumn(ByteBufferUtil.bytes(“customer_id), ByteBufferUtil.bytes(order.customerId),timestamp);orders.addColumn(ByteBufferUtil.bytes(“date), ByteBufferUtil.bytes(order.date), timestamp);orders.addColumn(ByteBufferUtil.bytes(“total), ByteBufferUtil.bytes(order.total), timestamp);}!customerOrders.close()orders.close() 35. // list of orders by usercustomerOrders = new SSTableSimpleUnsortedWriter(…);// orders by order idorders = new SSTableSimpleUnsortedWriter(…);!// assume orders are in date orderfor (Order order : oldOrders) {customerOrders.newRow(ByteBufferUtil.bytes(order.customerId));customerOrders.addColumn(ByteBufferUtil.bytes(order.orderId), ByBufferUtil.EMPTY_BYTE_BUFFER,timestamp);!orders.newRow(ByteBufferUtil.bytes(order.userId));orders.addColumn(ByteBufferUtil.bytes(“customer_id), ByteBufferUtil.bytes(order.customerId),timestamp);orders.addColumn(ByteBufferUtil.bytes(“date), ByteBufferUtil.bytes(order.date), timestamp);orders.addColumn(ByteBufferUtil.bytes(“total), ByteBufferUtil.bytes(order.total), timestamp);}!customerOrders.close()orders.close() 36. !•How Data is Stored•Case Studies - Generating Dummy Data - Backfilling Historical Data - Changing Topologies•Conclusion 37. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/…/cassandra-2.0.4$ bin/sstableloader mykeyspace/mycf -d cass1,cass2,cass3!Streaming relevant part of mykeyspace/mycf/mykeyspace-mycf-ic-1-Data.db to [/cass1,cass2,cass3,cass4,cass5,cass6]!progress: [/cas1 3/3 (100)] [/cas2 0/4 (0)] [/cas3 0/0 (0)] [/cas4 0/0 (0)] [/cas5 0/0(0)] [/cas6 1/2 (50)] [total: 50 - 0MB/s (avg: 5MB/s)] 38. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/…/cassandra-2.0.4$ bin/sstableloader mykeyspace/mycf -d cass1,cass2,cass3!Streaming relevant part of mykeyspace/mycf/mykeyspace-mycf-ic-1-Data.db to [/cass1,cass2,cass3,cass4,cass5,cass6]!progress: [/cas1 3/3 (100)] [/cas2 0/4 (0)] [/cas3 0/0 (0)] [/cas4 0/0 (0)] [/cas5 0/0(0)] [/cas6 1/2 (50)] [total: 50 - 0MB/s (avg: 5MB/s)] 39. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/…/cassandra-2.0.4$ bin/sstableloader mykeyspace/mycf -d cass1,cass2,cass3!Streaming relevant part of mykeyspace/mycf/mykeyspace-mycf-ic-1-Data.db to [/cass1,cass2,cass3,cass4,cass5,cass6]!progress: [/cas1 3/3 (100)] [/cas2 0/4 (0)] [/cas3 0/0 (0)] [/cas4 0/0 (0)] [/cas5 0/0(0)] [/cas6 1/2 (50)] [total: 50 - 0MB/s (avg: 5MB/s)] 40. $ bin/sstableloader Keyspace1/ColFam1patricia@dev:~/.../cassandra-2.0.4$ bin/nodetool compactionstatspending tasks: 30Active compaction remaining time :n/a 41. !•How Data is Stored•Case Studies - Generating Dummy Data - Backfilling Historical Data - Changing Topologies•Conclusion 42. cqlsh> CREATE KEYSPACE "test"WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1 };!cqlsh> CREATE COLUMNFAMILY "test" (id text PRIMARY KEY ) ;CQL: Keep schema consistent 43. CQL3 Considerations•Uses CompositeType comparator 44. Planet Cassandra 2014Q&APatricia Gorla@patriciagorlaCassandra Consultantwww.thelastpickle.com Recommended Test Prep: GREOnline Course - LinkedIn Learning Learning the Basics of BrandingOnline Course - LinkedIn Learning Learning How to Increase Learner EngagementOnline Course - LinkedIn Learning Bulk Loading into CassandraBrian Hess Migration Best Practices: From RDBMS to Cassandra without a HitchDataStax Academy Webinar: DataStax Training - Everything you need to become a Cassandra RockstarDataStax Cassandra Community Webinar | Make Life Easier - An Introduction to Cassandra...DataStax Cassandra Virtual Node talkPatrick McFadin Cassandra Summit 2014: Apache Cassandra Best Practices at EbayDataStax Academy Webinar: Eventual Consistency != Hopeful ConsistencyDataStax About Blog Terms Privacy Copyright LinkedIn Corporation © 2020 × Public clipboards featuring this slideNo public clipboards found for this slideSelect another clipboard ×Looks like you’ve clipped this slide to already.Create a clipboardYou just clipped your first slide! Clipping is a handy way to collect important slides you want to go back to later. Now customize the name of a clipboard to store your clips. Description Visibility Others can see my Clipboard