Welcome back, everyone. I have been working on Cassandra for quite some time now but never actually got to explore its working in depth.
We know that its decentralized nature, as well as its ability to handle such a large volume of writes, makes it really commendable. But how does it manage to be efficient? How is it able to achieve what it is so popular for? These are some questions that motivated me to dig into its working a bit more.
So, this blog is dedicated to one of the unsung heroes that make it all possible: The Commit Log. We will try to list down some of the answers to the possible questions people have in mind related to the Cassandra Commit log.
As Cassandra is well known for its fast writes, we will try to keep our focus on how the Commit Log helps us improve the speed of writing data into the Cassandra.
If you are new to Cassandra or want to explore a bit more about it before going into a bit detail, I would recommend you some of my previous blogs:
Before going through the internals, let’s have a quick recap about how Cassandra actually works.
BASIC FLOW OF DATA INTO CASSANDRA TABLES
If we go through the above diagram, we will notice that the very first step of writing the data to Cassandra involves adding the data to the commit log, only after which it gets stored in Memtables and SS Tables.
Now the main questions we have in our mind are that what a commit log actually is and how does it affect Cassandra’s performance.
Definition: Commitlogs are an append only log of all mutations local to a Cassandra node. Any data written to Cassandra will first be written to a commit log before being written to a memtable.
To clarify this definition a bit, You can think of the commit log as an optimization, but Cassandra would be unusably slow without it.
When MemTables get written to disk we call them SSTables. SSTables are immutable, meaning once Cassandra writes them to disk it does not update them. So when a column changes Cassandra needs to write a new SSTable to disk. If Cassandra was writing these SSTables to disk on every update it would be completely IO bound and very slow.
So Cassandra uses a few tricks to get better performance. Instead of writing SSTables to disk on every column update, it keeps the updates in memory and flushes those changes to disk periodically to keep the IO to a reasonable level. But this leads to the obvious problem that if the machine goes down or Cassandra crashes you would lose data on that node. To avoid losing data, in addition to keeping recent changes in memory, Cassandra writes the changes to its CommitLog.
Now let’s try to answer some of the questions related to Cassandra Commit log which we found a bit interesting.
What is the Purpose of Commit Log??
In the above statement, we noted that to avoid losing data, in addition to keeping recent changes in memory, Cassandra writes the changes to its CommitLog. What this means is that in a case where a node crashes or gets rebooted, Cassandra should be able to recreate the memtable. This is important, since the memtable only gets flushed to disk when it’s ‘full’ – meaning the configured memtable size is exceeded – or the flush is performed by nodetool or opscenter. So the data in memtable is not persisted directly.
Why writing to the CommitLog is better than just writing the SSTables?
- The CommitLog is optimized for writing. Unlike SSTables which store rows in sorted order, the CommitLog stores updates in the order which they were processed by Cassandra.
- The CommitLog also stores changes for all the column families in a single file so the disk doesn’t need to do a bunch of seeks when it is receiving updates for multiple column families at the same time.
- Basically writing the CommitLog to the disk is better because it has to write less data than writing SSTables does and it writes all that data to a single place on the disk.
Cassandra also keeps track of what data has been flushed to SSTables and is able to truncate the Commit log once all data older than a certain point has been written.
What to do to prevent Cassandra commit logs filling up disk space?
In some scenarios, there are cases that the commit logs fill up all the disk space on the allotted mount and one might face some issues related to refusal of connection.
Some of the possible solutions to that are:
- Try lowering the commitlog_total_space_in_mb setting in your cassandra.yaml(The default is 8192MB for 64-bit systems)
- In addition to the above solution, a proper solution to ensure it won’t happen again could be to have the monitoring of the disk setup so that you are alerted when it’s getting full and have time to act/increase the disk size.
Apart from the Commit log, the MemTable and the SS Tables also play an important role in making Cassandra an effective database. We will try to cover them in the upcoming blogs.
For now, in this blog, we tried to cover about the Commit log in Cassandra and also tried to answer some of the questions that one might get while dealing with Cassandra commit log.
To know more about the Commit log, you can go through the documentation here.
Hope this helps. Stay tuned for more. 🙂
