A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) …

Successfully reported this slideshow.

A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassandra Summit 2016
Introduction to
Cassandra.yaml
& friends
Hi, I’m Edward Capriolo.
@edwardcapriolo
https://www.linkedin.com/in/edwardcapriolo
http://www.slideshare.net/edwardcaprio...
We help people deliver and improve Apache
Cassandra based solutions.
With staff in 5 countries and over 50 years
combined ...
This talk is the ʻgatewayʼ talk…
Many ʻpicklersʼ (TLP staff) are covering
some points I will quickly cover over in
depth i...
1. Key configuration settings
2. Configuration outside of the yaml
3. Multi-system configuration settings
4. Advanced setting...
1. $ wget <apache-cassandra*.tar.gz>
2. $ tar -xf <apache-cassandra*.tar.gz>
3. $ apache-cassandra*/bin/cassandra
Basic se...
Well almost…
We have to do a bit of configuration.
cqlsh> CREATE KEYSPACE test WITH replication = 

{'class': ‘SimpleStrategy', 'replication_factor' : 1};

cqlsh> USE test;
...
Single Data Center
Multiple Data Center
data_file_directories:
- /var/lib/cassandra/data
1. User data is stored in all listed directories
2. Do: fast seekʼing sto...
commitlog_directory:
- /var/lib/cassandra/commitlog
2. Donʼt: Assume these are log4j type logs
3. Do: use a dedicated disk...
Ok we now where
(most of) the data goes…
How do clients connect?
Default port binding
1. Cassandra does not bind to 0.0.0.0
2. 127.0.0.1 not web scale
3. 7000 is the “Storage Port” inter ...
start_native_transport: true (default)
native_transport_port: 9042 (default)
listen_address: localhost
1. Change listen_ad...
Outside the yaml file…
cassandra-env.sh (& friends)
1. JVM and startup params defined outside the YAML
2. Newer version of c* use jvm.options
#MAX_HEAP_SIZE="1G"
#HEAP_NEWSIZE="100M"
1. max(min(1/2 ram, 1024MB), 

min(1/4 ram, 8GB))
2. Do: set lower when experimen...
JMX
1. bin/nodetool uses JMX to administer Cassandra
2. All management tools require password if set
Check out Nate’s talk...
Multi-node configurations
# phi_convict_threshold: 8
1. Threshold for failure detector
3. Do: Raise for flaky WAN networks 10 - 12
Phi convict thresh...
# endpoint_snitch: SimpleSnitch
2. Do: use SimpleSnitch for single switch/LAN
3. Consider: Multi DC to start
Defining netwo...
3. Rack has impact on replication copies
4. Donʼt: Change rack unless you understand the impact
Gossiping Property File Sn...
internode_compression: all | dc | none
inter_dc_tcp_nodelay: false
Internode communications
1. WAN can benefit from reduced...
broadcast_address: 1.2.3.4
listen_on_broadcast_address: false
broadcast_rpc_address: 1.2.3.4
Broadcast address
1. Gossip a...
Broadcast address
Advanced settings
Write path
http://www.toadworld.com/platforms/nosql/w/wiki/11621.an-
introduction-to-apache-cassandra
#memtable_flush_writers: 1
1. Default One per data directory
Memtables
# memtable_cleanup_threshold
defaults to 1 /
(memta...
#If omitted, both set to 1/4 the heap
#memtable_heap_space_in_mb: 2048

#memtable_offheap_space_in_mb: 2048
1. Depending o...
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
Trickle fsync
1. Optimization to periodically f-sync large files
2...
Include image of compaction here
Compaction
https://www.instaclustr.com/blog/2016/01/27/apache-
cassandra-compaction/
concurrent_compactors: 1
compaction_throughput_mb_per_sec: 16
1. Control resources used by compaction
Compaction
2. Compac...
disk_failure_policy: stop
commit_failure_policy: stop
Disk Failure settings
1. stop_paranoid: shut down gossip and client
...
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_thread...
#disk_optimization_strategy: ssd
1. Tip for those with rotational disks
Disk optimization strategy
Exotic settings
auto_bootstrap : true(hidden variable)
Auto bootstrap
1. “Bootstrapping” here means: Should the node
joining attempt to ac...
incremental_backups: false

snapshot_before_compaction: false

auto_snapshot: true
Backup*Ish options
1. Enable with exter...
read_request_timeout_in_ms: 5000

write_request_timeout_in_ms: 2000

request_timeout_in_ms: 10000
Per operation default ti...
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32
Commit Log sync
1. Alternati...
Thanks!@edwardcapriolo
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassandra Summit 2016

Upcoming SlideShare

Loading in …5

×

  1. 1. Introduction to Cassandra.yaml & friends
  2. 2. Hi, I’m Edward Capriolo. @edwardcapriolo https://www.linkedin.com/in/edwardcapriolo http://www.slideshare.net/edwardcapriolo Consultant The Last Pickle Cassandra user since (v 0.6.5) White Plains, NY USA
  3. 3. We help people deliver and improve Apache Cassandra based solutions. With staff in 5 countries and over 50 years combined experience.
  4. 4. This talk is the ʻgatewayʼ talk… Many ʻpicklersʼ (TLP staff) are covering some points I will quickly cover over in depth in other talks.
  5. 5. 1. Key configuration settings 2. Configuration outside of the yaml 3. Multi-system configuration settings 4. Advanced settings Section Overview 5. Exotic settings
  6. 6. 1. $ wget <apache-cassandra*.tar.gz> 2. $ tar -xf <apache-cassandra*.tar.gz> 3. $ apache-cassandra*/bin/cassandra Basic setup Result: Web scale distributed storage Drop Mic.
  7. 7. Well almost… We have to do a bit of configuration.
  8. 8. cqlsh> CREATE KEYSPACE test WITH replication = 
 {'class': ‘SimpleStrategy', 'replication_factor' : 1};
 cqlsh> USE test;
 cqlsh:test> CREATE COLUMNFAMILY trip (src varchar, ... dest varchar, PRIMARY KEY (src,dest)); Before we dive into config cqlsh:test> SELECT * FROM trip; src | dest -----+------ ny | ca cqlsh:test> INSERT INTO trip (src, dest) VALUES ('ny', 'ca'); cqlsh:test> INSERT INTO trip (src, dest) VALUES ('fl', 'ca'); cqlsh:test> SELECT * FROM trip; src | dest -----+------ fl | ca ny | ca
  9. 9. Single Data Center
  10. 10. Multiple Data Center
  11. 11. data_file_directories: - /var/lib/cassandra/data 1. User data is stored in all listed directories 2. Do: fast seekʼing storage (SSD) Where does the data go? 3. Do: ample free space (30% overhead) 4. Donʼt: Store on a SAN
  12. 12. commitlog_directory: - /var/lib/cassandra/commitlog 2. Donʼt: Assume these are log4j type logs 3. Do: use a dedicated disk if possible Commit log storage 4. Do: provide at least 10GB (write velocity) 1. Stores unflushed mutations (write/deletes)
  13. 13. Ok we now where (most of) the data goes… How do clients connect?
  14. 14. Default port binding 1. Cassandra does not bind to 0.0.0.0 2. 127.0.0.1 not web scale 3. 7000 is the “Storage Port” inter node traffic 4. 9042 is the “Native Port” client traffic
  15. 15. start_native_transport: true (default) native_transport_port: 9042 (default) listen_address: localhost 1. Change listen_address to a client-reachable address Native transport 2. Do: consider transport security 3. Do: consider network routing performance 4. Donʼt: put nodes on a public network. EVAR
  16. 16. Outside the yaml file…
  17. 17. cassandra-env.sh (& friends) 1. JVM and startup params defined outside the YAML 2. Newer version of c* use jvm.options
  18. 18. #MAX_HEAP_SIZE="1G" #HEAP_NEWSIZE="100M" 1. max(min(1/2 ram, 1024MB), 
 min(1/4 ram, 8GB)) 2. Do: set lower when experimenting with workstation Memory usage 3. Do: leave ample free memory for disk cache
  19. 19. JMX 1. bin/nodetool uses JMX to administer Cassandra 2. All management tools require password if set Check out Nate’s talk on Securing Cassandra to learn more
  20. 20. Multi-node configurations
  21. 21. # phi_convict_threshold: 8 1. Threshold for failure detector 3. Do: Raise for flaky WAN networks 10 - 12 Phi convict threshold 2. False positives make nodes appear down to peers
  22. 22. # endpoint_snitch: SimpleSnitch 2. Do: use SimpleSnitch for single switch/LAN 3. Consider: Multi DC to start Defining network topology 1. Snitch with config data determines topology
  23. 23. 3. Rack has impact on replication copies 4. Donʼt: Change rack unless you understand the impact Gossiping Property File Snitch 2. DC may not be physical but is a replication unit conf/cassandra-rackdc.properties 
 dc=dc1 rack=rack1 1. Information is propagated around the cluster
  24. 24. internode_compression: all | dc | none inter_dc_tcp_nodelay: false Internode communications 1. WAN can benefit from reduced size server_encryption_options: internode_encryption: none internode_authenticator: o.a.c.auth.AllowAllInternodeAuthenticator 2. Settings which server nodes use to communicate
  25. 25. broadcast_address: 1.2.3.4 listen_on_broadcast_address: false broadcast_rpc_address: 1.2.3.4 Broadcast address 1. Gossip a specific address (not bind address) 2. Useful in NAT and cloud environments
  26. 26. Broadcast address
  27. 27. Advanced settings
  28. 28. Write path http://www.toadworld.com/platforms/nosql/w/wiki/11621.an- introduction-to-apache-cassandra
  29. 29. #memtable_flush_writers: 1 1. Default One per data directory Memtables # memtable_cleanup_threshold defaults to 1 / (memtable_flush_writers + #memtable_cleanup_threshold: 0.11 2. 1 / (1 + 1) = .5
  30. 30. #If omitted, both set to 1/4 the heap #memtable_heap_space_in_mb: 2048
 #memtable_offheap_space_in_mb: 2048 1. Depending on the next setting dictates how much of each memory type is used .5 of what you ask? #heap_buffers: on heap nio buffers
 #offheap_buffers: off heap nio buffers
 #offheap_objects: off heap objects
 #memtable_allocation_type: heap_buffers 2. Based on column value buffers vs objects may be better
  31. 31. trickle_fsync: false trickle_fsync_interval_in_kb: 10240 Trickle fsync 1. Optimization to periodically f-sync large files 2. Designed to prevent latency spikes in read path
  32. 32. Include image of compaction here Compaction https://www.instaclustr.com/blog/2016/01/27/apache- cassandra-compaction/
  33. 33. concurrent_compactors: 1 compaction_throughput_mb_per_sec: 16 1. Control resources used by compaction Compaction 2. Compaction throughput can be changed at runtime 3. Generally concurrent_compactors < 8 and > 1
  34. 34. disk_failure_policy: stop commit_failure_policy: stop Disk Failure settings 1. stop_paranoid: shut down gossip and client transports even for single-sstable errors, kill the JVM for errors during startup 2. die: shut down gossip and Thrift and kill the JVM, so the node can be replaced
  35. 35. hinted_handoff_enabled: true max_hint_window_in_ms: 10800000 hinted_handoff_throttle_in_kb: 1024 max_hints_delivery_threads: 2 hints_directory: /var/lib/cassandra/hints hints_flush_period_in_ms: 10000 max_hints_file_size_in_mb: 128
 hints_compression: LZ4Compressor Hints 1. Hints recently redesigned, again again 2. Donʼt: tune high and overwhelming recovering node 3. Donʼt: tune low and have out of sync data
  36. 36. #disk_optimization_strategy: ssd 1. Tip for those with rotational disks Disk optimization strategy
  37. 37. Exotic settings
  38. 38. auto_bootstrap : true(hidden variable) Auto bootstrap 1. “Bootstrapping” here means: Should the node joining attempt to acquire data from other nodes or startup empty 2. Can be used when bringing on new datacenter 3. Can be used when streaming/ join issues
  39. 39. incremental_backups: false
 snapshot_before_compaction: false
 auto_snapshot: true Backup*Ish options 1. Enable with external backup like tools 2. Creates hard link files operator must clean up 3. Enabling and not cleaning will cause disk fill up 4. Truncate/drop makes snapshot
  40. 40. read_request_timeout_in_ms: 5000
 write_request_timeout_in_ms: 2000
 request_timeout_in_ms: 10000 Per operation default timeouts 1. Each operation type has different timeout 2. Applied on the coordinator not the client 3. Previously was only global rpc_timeout
  41. 41. commitlog_sync: periodic commitlog_sync_period_in_ms: 10000
 commitlog_segment_size_in_mb: 32 Commit Log sync 1. Alternative batch mode blocks ack to clients 2. Commit logs persist until Memtableʼs flush
  42. 42. Thanks!@edwardcapriolo