Illustration Image

Cassandra.Link

The best knowledge base on Apache Cassandra®

Helping platform leaders, architects, engineers, and operators build scalable real time data platforms.

11/19/2020

Reading time:4 min

Apache Cassandra Lunch #26: Cassandra Troubleshooting with Logs - Business Platform Team

by John Doe

In case you missed it, this post is a recap of Cassandra Lunch #26, discussing common Cassandra log warnings and errors. We discussed the various resources that a Cassandra cluster needs, and how we can find problems with those resources via the logs generated by the cluster. The live recording of Cassandra Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at noon EST. Register here now!Cassandra System ResourcesDiskSince Cassandra is a database, it needs to have enough storage to function. Part of how Cassandra maintains speed and availability in a distributed system is to store multiple SSTables and taking up extra space with replicas and denormalized tables. One way to ensure continued operation as data is generated is to split the storage of particular data onto several drives. One disk can be made to hold the basic Cassandra data, like SSTables containing table data. The second disk could hold any data needed to reconstruct a node if it goes down. The commit logs, hints, and any saved caches go onto disk two. Disk three contains the log files so that even if other disks fill up we can continue to get log messages. MemoryA lot of Cassandra processing takes place in memory, like MemTables before that data gets written to disk, or various caches to increase read speed. Cassandra is a Java application so the amount of memory allocated is determined by configurable settings. The heap should be 20% to 50% of all memory and must be no more than 32GB. Some memory is allocated for off-heap cache and file system cache. The amount of memory used by the cluster can be manipulated via garbage collection configuration.CPUThe Cassandra cluster operations that might take up enough CPU power to cause warnings or errors are a large number of writes, which then trigger compaction. A large number of reads can cause the same problem. The repair process can also take up a lot of computing power.NetworkNetwork issues can also cause timeouts which can cause issues when trying to read or write data to a node.LoggingLog LevelsThe log levels appear in Cassandra logs to denote certain messages more or less important than other messages. INFO is a low level of alert that denotes normal messages with no chance of affecting service for the cluster. WARN denotes messages that may lead to errors or service interruptions. They do not represent imminent service interruption but can generally be found in the lead up to such incidents. The last level is ERROR which denotes significant errors that may affect the operation of the Cassandra cluster.Search TermsWhen searching through messages using something like the ELK stack, discussed here, you can look at the frequency of WARN/ERROR messages or the total number of log messages to get a general idea of cluster health. Specific messages to look for include hinted handoff messages. Hinted handoff happens when a node is down or too busy to receive messages from other nodes. These can be triggered if there is too much traffic involving a particular node, or if the cluster is too small to handle the load we are receiving. Hints are dropped after three hours of being unable to be written to a node, so hints dropped messages show that a node has been down or otherwise unable to communicate for three hours. Dropped mutations or task messages happen when a node is unable to complete a task which can hint at bigger problems. Repair errors can happen with particular threads. To find them, search AntiEntropyStage, ReadRepair, or RepairTask. By searching compacting large partition, we can find wide partitions that are causing problems. Tombstones can also cause problems. The message is triggered based on the tombstone_warn_threshold configuration settings. Messages mentioning java show problems with the Java application that is Cassandra. Things like out of memory errors will show up here. Disk space problems can be found by searching unable to write or corrupted. Timeouts can be found based on the connection or by failing reads and writes.Log AggregationA discussion of log aggregation methods can be found here.ReferencesChanging heap size parameters | DataStax Distribution of Apache CassandraDocumentation | HardwareCassandra AWS System Memory GuidelinesCassandra Logs — Apache Cassandra Documentation v4.0Documentation | Reading Logs Cassandra.LinkCassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email! Posted in Modern Business | Comments Off on Apache Cassandra Lunch #26: Cassandra Troubleshooting with Logs

Illustration Image

In case you missed it, this post is a recap of Cassandra Lunch #26, discussing common Cassandra log warnings and errors. We discussed the various resources that a Cassandra cluster needs, and how we can find problems with those resources via the logs generated by the cluster. The live recording of Cassandra Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at noon EST. Register here now!

Cassandra System Resources

Disk

Since Cassandra is a database, it needs to have enough storage to function. Part of how Cassandra maintains speed and availability in a distributed system is to store multiple SSTables and taking up extra space with replicas and denormalized tables. One way to ensure continued operation as data is generated is to split the storage of particular data onto several drives. One disk can be made to hold the basic Cassandra data, like SSTables containing table data. The second disk could hold any data needed to reconstruct a node if it goes down. The commit logs, hints, and any saved caches go onto disk two. Disk three contains the log files so that even if other disks fill up we can continue to get log messages. 

Memory

A lot of Cassandra processing takes place in memory, like MemTables before that data gets written to disk, or various caches to increase read speed. Cassandra is a Java application so the amount of memory allocated is determined by configurable settings. The heap should be 20% to 50% of all memory and must be no more than 32GB. Some memory is allocated for off-heap cache and file system cache. The amount of memory used by the cluster can be manipulated via garbage collection configuration.

CPU

The Cassandra cluster operations that might take up enough CPU power to cause warnings or errors are a large number of writes, which then trigger compaction. A large number of reads can cause the same problem. The repair process can also take up a lot of computing power.

Network

Network issues can also cause timeouts which can cause issues when trying to read or write data to a node.

Logging

Log Levels

The log levels appear in Cassandra logs to denote certain messages more or less important than other messages. INFO is a low level of alert that denotes normal messages with no chance of affecting service for the cluster. WARN denotes messages that may lead to errors or service interruptions. They do not represent imminent service interruption but can generally be found in the lead up to such incidents. The last level is ERROR which denotes significant errors that may affect the operation of the Cassandra cluster.

Search Terms

When searching through messages using something like the ELK stack, discussed here, you can look at the frequency of WARN/ERROR messages or the total number of log messages to get a general idea of cluster health. 

Specific messages to look for include hinted handoff messages. Hinted handoff happens when a node is down or too busy to receive messages from other nodes. These can be triggered if there is too much traffic involving a particular node, or if the cluster is too small to handle the load we are receiving. Hints are dropped after three hours of being unable to be written to a node, so hints dropped messages show that a node has been down or otherwise unable to communicate for three hours. Dropped mutations or task messages happen when a node is unable to complete a task which can hint at bigger problems.

Repair errors can happen with particular threads. To find them, search AntiEntropyStage, ReadRepair, or RepairTask. By searching compacting large partition, we can find wide partitions that are causing problems. Tombstones can also cause problems. The message is triggered based on the tombstone_warn_threshold configuration settings. Messages mentioning java show problems with the Java application that is Cassandra. Things like out of memory errors will show up here. Disk space problems can be found by searching unable to write or corrupted. Timeouts can be found based on the connection or by failing reads and writes.

Log Aggregation

A discussion of log aggregation methods can be found here.

References

Changing heap size parameters | DataStax Distribution of Apache Cassandra

Documentation | Hardware

Cassandra AWS System Memory Guidelines

Cassandra Logs — Apache Cassandra Documentation v4.0

Documentation | Reading Logs

Cassandra.Link

Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.

We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!

Related Articles

cluster
troubleshooting
datastax

GitHub - arodrime/Montecristo: Datastax Cluster Health Check Tooling

arodrime

4/3/2024

Checkout Planet Cassandra

Claim Your Free Planet Cassandra Contributor T-shirt!

Make your contribution and score a FREE Planet Cassandra Contributor T-Shirt! 
We value our incredible Cassandra community, and we want to express our gratitude by sending an exclusive Planet Cassandra Contributor T-Shirt you can wear with pride.

Join Our Newsletter!

Sign up below to receive email updates and see what's going on with our company

Explore Related Topics

AllKafkaSparkScyllaSStableKubernetesApiGithubGraphQl

Explore Further

troubleshooting