{"componentChunkName":"component---src-templates-article-single-page-js","path":"/post/cassandra-compaction-throughput-performance-explained","result":{"pageContext":{"obj_id":"bac4a4fc-4db8-5d99-be5e-ba971e77a129","node":{"content":"

This is the second post in my series on improving node density and lowering costs with Apache Cassandra. In the previous post, I examined how streaming performance impacts node density and operational costs. In this post, I’ll focus on compaction throughput, and a recent optimization in Cassandra 5.0.4 that significantly improves it, CASSANDRA-15452.

This post assumes some familiarity with Apache Cassandra storage engine fundamentals. The documentation has a nice section covering the storage engine if you’d like to brush up before reading this post.

The Compaction Bottleneck

Compaction in Cassandra is the process of merging multiple SSTables and writing out new ones, discarding tombstones, resolving overwrites, and generally organizing data for efficient reads. It’s an I/O intensive background operation that directly competes with foreground operations for system resources. In a later post I’ll look at how compaction strategies impact node density, but for now, I’ll just focus on throughput.

Why Compaction Throughput Matters for Node Density

As we continue to increase the amount of data we store per node, compaction performance becomes increasingly important. It affects:

Simply put: as your data volume and write throughput increase, compaction throughput must as well. If it doesn’t, you’ll hit a performance wall that effectively caps your maximum practical node density.

Despite the significant improvements to compaction throughput over the years, there are some circumstances where compaction performance is inadequate. Let’s take a look at the reason why, then dive into what can be done about it.

When doing any performance evaluation, it’s important to understand how to measure where your time is spent. A lot of folks make incorrect assumptions, and then waste a lot of time trying to optimize something that doesn’t matter. I’ve written several posts about how useful profiling with the async-profiler can be from an application perspective. For looking at the OS and hardware, the eBPF based toolkit bcc-tools can help you identify process bottlenecks. I’ve used these tools extensively over the years, and in this post I’ll show how they’ve helped identify two major performance bottlenecks in compaction. My easy-cass-lab software includes all these tools, as well as integration with AxonOps for Cassandra dashboards and operational tooling.

Being 10x Smarter With Our Disk Access

When investigating compaction behavior, I discovered an major inefficiency in how Cassandra was accessing disk. The problem was especially severe in cloud environments with disaggregated storage like AWS EBS, where IOPS (Input/Output Operations Per Second) are both limited and expensive when used improperly.

When Cassandra would read in data during compaction, it would read individual compressed chunks off disk, one small read at a time. Using bcc-tools, we can monitor every filesystem operation. Here I’m using xfsslower to record every read operation on the filesystem (original headers back in for clarity):

$ sudo /usr/share/bcc/tools/xfsslower 0 -p 26988 | awk '$4 == \"R\" { print $0 }'\nTracing XFS operations\nTIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME\n22:27:38 CompactionExec 26988  R 4096    0           0.01 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988  R 4096    4           0.00 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988  R 2062    8           0.00 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988  R 14907   0           0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14924   14          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14896   29          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14844   43          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14923   58          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14931   72          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14905   87          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14891   101         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14919   116         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14965   130         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14918   145         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14930   160         0.01 nb-7-big-Data.db\n

The above is showing we’re reading about 14KB at a time. That’s the size of the compressed page. This pattern is terrible for performance on cloud storage systems like EBS, where:

  1. Each read operation, no matter how small, counts against your provisioned IOPS
    2. \n
  2. Small reads waste IOPS quota while delivering minimal data
    3. \n
  3. You pay for IOPS allocation whether you use it efficiently or not
    4. \n

Looking at a wall clock performance profile, we can see compaction is spending a LOT of time waiting on disk, in the really wide column with the pread call at the top:

\"wall-clock-profile-compaction.png\"

Readahead is a disk optimization strategy where the operating system reads a larger block of data than was requested into memory. The objective is reduce latency and improve performance for sequential read operations. Unfortunately, when you don’t need the data it’s reading, it can be the source of major performance problems. In my experience, read ahead is one of the worst culprits in the world of Cassandra performance. It’s especially terrible for lightweight transactions and counters, where we perform a read before write.

My advice to Cassandra operators is to reduce readahead to 4KB to avoid unnecessary read amplification on the read path.

Readahead does have one place, however, where it can benefit performance. You may have already guessed that it’s compaction. Let’s take a step back and look at how the size of our reads impacts our throughput in a simple benchmark. Larger reads, initiated either from read ahead or the user, should deliver improved throughput, especially when we’re dealing with a quota on our IOPS (EBS), our drives have higher latency (SAN), or both.

Benchmarking

I ran benchmark tests with sequential fio workloads using different request sizes on a 3K IOPS GP3 EBS volume. Here’s the configuration used:

[global]\nrw=read\ndirectory=data\ndirect=1\ntime_based=1\nfile_service_type=normal\nstonewall\nsize=100M\nnumjobs=12\ngroup_reporting\n[bs4]\nstonewall\nruntime=60s\nblocksize=4k\n[bs8]\nstonewall\nruntime=60s\nblocksize=8k\n[bs16]\nstonewall\nruntime=60s\nblocksize=16k\n[bs32]\nstonewall\nruntime=60s\nblocksize=32k\n[bs64]\nstonewall\nruntime=60s\nblocksize=64k\n[bs128]\nstonewall\nblocksize=128k\nruntime=60s\n[bs256]\nstonewall\nruntime=60s\nblocksize=256k\n

When reviewing the results, the benefits of using larger request sizes were evident:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Request SizeIOPSThroughput
4K304911.9 MB/s
8K301223 MB/s
16K301347 MB/s
32K301394 MB/s
64K1938121 MB/s
128K957120 MB/s
256K478120 MB/s

The data shows that using 256KB reads instead of 16KB reads would deliver almost 3x the throughput while using only 1/6th of the provisioned IOPS. That’s a massive efficiency improvement. Rather than chewing through all our IOPS to deliver a paltry 47MB/s of throughput, we’re only using about 500 for 120MB/s. That means if we can see these gains in the database, we’ll be able to compact faster, put more data on each node, and lower our total cost.

The Solution: Internally Buffering Sequential Reads

In CASSANDRA-15452, I worked with my fellow Cassandra committer Jordan West to implement a solution: an efficient, internal read-ahead buffer for bulk reading operations. Here’s how it works:

  1. Instead of reading tiny chunks, we use a 256KB off-heap buffer
    2. \n
  2. Each read operation pulls in a full 256KB of data at once
    3. \n
  3. Compressed chunks are extracted from this buffer as needed
    4. \n
  4. The buffer is refilled only when necessary
    5. \n

This approach maximizes IOPS efficiency by using larger reads during compaction (as well as repair and range reads) that deliver more data per operation. For cloud environments, it’s a game-changer that directly aligns with storage provider recommendations. AWS EBS, for instance, considers any I/O operation up to 256KB as a single operation, so by using the largest possible size we should get optimal performance.

Real-World Impact: A Major Improvement in Compaction Throughput

When Jordan and I tested the implementation using easy-cass-lab on EBS, the results were nothing short of spectacular. The 10.0.2.171 node is running our patched version, the other two nodes are running an unpatched release. The graphs clearly show a 2-3x improvement to throughput and a 3x reduction in IOPS.

\"15452-bytes-read.png\"

\"Compaction

You can see the results in the flamegraph as well. The calls to pread take up significantly less time.

\"wall-clock-profile-compaction-after-15452.png\"

We can use xfsslower from bcc-tools again to watch the filesystem access:

$ sudo /usr/share/bcc/tools/xfsslower 0 -p $(cassandra-pid) | awk '$4 == \"R\" { print $0 }'\nTIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME\n14:40:29 CompactionExec 1782   R 262144  256         0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  512         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  768         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1024        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1280        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1536        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 241123  1792        0.07 nb-4-big-Data.db\n

This is a lot better, now we’re fetching 256KB at a time using way fewer requests.

The EBS test configuration used a GP3 volume with 3K IOPS and 256MB throughput. With the existing code, compaction was bottlenecked by IOPS, peaking at exactly 3K IOPS but achieving only about 51MB/s throughput. With our optimization, the same operation used only ~500 IOPS to achieve around 106MB/s—a more than 2x improvement in throughput with 1/3IOPS.

In our most aggressive testing, we actually hit the EBS throughput limit rather than the IOPS limit. That’s a significant transformation in Cassandra’s resource utilization profile.

The patch also has the benefit of applying to anti-compaction, repair, and range reads. We can see a significant reduction in range reads, aka table scans:

\"15452-range-reads.png\"

If you’re running Spark jobs using the Cassandra connector, you should see an improvement in performance, and your repair times should decrease.

What’s next? Can we do more?

Yes, absolutely! There’s several more improvements to IO that will help improve things. I’ll cover them here very quickly, and if there’s interest I’ll write about them in detail in a future post.

Avoid Reading the Statistics

When compacting, we read data out of the Statistics.db file before reading the data itself. This is completely unnecessary, as it’s stats about the data we’re about to read. Skipping this can reduce IO even further. Looking at a compaction’s IO activity, I see about 30% of the filesystem access is reading from Statistics.db:

14:40:29 CompactionExec 1782   R 4096    0           0.00 nb-3-big-Statistics.db\n14:40:29 CompactionExec 1782   R 701     4           0.00 nb-3-big-Statistics.db\n14:40:29 CompactionExec 1782   R 4096    0           0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782   R 4096    4           0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782   R 1962    8           0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782   R 262144  0           0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 2115    0           0.01 nb-3-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  256         0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  512         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  768         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1024        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1280        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1536        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 241123  1792        0.07 nb-4-big-Data.db\n

This has already been fixed in trunk by Branimir Lambov in CASSANDRA-20092 and is being backported to 5.0 by Jordan.

Direct I/O for Compaction

Let’s talk more about page cache. Since we go through the Linux page cache when doing reads, we want to make sure it’s working optimally. Page cache lets us avoid going to disk! Unfortunately we also use it when reading for compaction. This is a problem because we’re pulling data into the page cache that we plan on deleting. To make room for the new data, other data will be evicted. If we compact 10GB of data, we’re pushing out a lot of valuable data from the page cache, meaning it needs to be fetched back into memory later on. Using Direct I/O we can bypass the page cache entirely, which will prevent data from being evicted. This can be a huge help in latency sensitive systems or systems where IOPS are limited like EBS.

I’ve filed CASSANDRA-19987 to look at this.

Non Blocking Compression

Next, compression. When we’re writing to disk, we fill a buffer, sized by the chunk_length_in_kb table setting, compress, and write to disk. The compression here is a blocking call, which means we can spend a lot of time waiting on compression to finish, when we could be reading and merging the next chunk in parallel This can show up as a performance bottleneck, so I’ve filed CASSANDRA-20085 to look into it.

Better Memory Management

When a system is not bottlenecked on disk I/O, such as when using NVMe, the main issue we’ll run into is our heap allocation rate. I’ll go into details in a future post, but for now, it’s enough to know that the more memory we allocate, the worse our performance. Being smart about memory allocations can make a big difference in overall time spent, as allocations aren’t free. It also reduces both the frequency and duration of Garbage Collection. Big wins all around.

I recently profiled an instance where the row size was about 2KB (not out of the ordinary) and found that a single call was accounting for roughly 50% of memory allocated. Fixing this one thing has the potential to deliver a massive performance improvement, especially in workloads where we have either lots of fields, or large fields like serialized blobs.

Reaching again for async-profiler, this time we run it with -e alloc to track allocations and --reverse to reverse the stacks. I do this because the same underlying call comes from the read path and compaction, and I want to see the time in aggregate.

\"allocation-profile-compaction.png\"

Addressing this single allocation won’t just deliver faster compaction, but will reduce pressure on the heap, which in turn reduces GC overhead. As part of this series I’ll also be covering GC, as a lot’s changed since I wrote about it last.

I’ve filed CASSANDRA-20428 and there’s already a fair bit of discussion about different approaches to solving the problem.

Conclusion

Maximizing compaction throughput is critical for achieving higher node density with Apache Cassandra. The improvements in CASSANDRA-15452 have removed one of the primary bottlenecks that previously limited practical node size in a lot of clusters.

By upgrading to Cassandra 5.0.4 (or later) you can:

  1. Dramatically improve compaction throughput
    2. \n
  2. Reduce IOPS consumption significantly
    3. \n
  3. Improve overall system stability during write-heavy workloads
    4. \n
  4. Increase the maximum practical data density per node
    5. \n
  5. Significantly reduce your cloud storage costs
    6. \n

This improvement, combined with the streaming optimizations discussed in the previous post, creates a multiplier effect on your ability to increase node density. Each optimization removes a bottleneck, allowing you to push your hardware further and achieve more with less.

In my next post, I’ll be discussing how and why compaction strategies affect node density. Picking the right strategy can have a significant impact on your cluster’s performance and cost efficiency. Make sure you sign up for my mailing list if you’re interested in getting notified when it’s released!

If you found this post helpful, please consider sharing to your network. I'm also available to help you be successful with your distributed systems! Pleasereach outif you're interested in working with me, and I'll be happy to schedule a free one-hour consultation.","id":"bac4a4fc-4db8-5d99-be5e-ba971e77a129","title":"Cassandra Compaction Throughput Performance Explained","origin_url":"https://rustyrazorblade.com/post/2025/04-compaction-throughput/","url":"https://rustyrazorblade.com/post/2025/04-compaction-throughput/","wallabag_created_at":"2025-04-24T12:03:02+00:00","published_at":"2025-04-16T00:00:00+00:00","published_by":"['']","reading_time":14,"domain_name":"rustyrazorblade.com","preview_picture":"https://rustyrazorblade.com/images/2025/wall-clock-profile-compaction.png","tags":["cassandra","performance"],"description":"This is the second post in my series on improving node density and lowering costs with Apache Cassandra. In the previous post, I examined how streaming performance impacts node density and operational..."},"relatedArticles":[{"content":"
\"image\"

Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system used by the largest tech companies in the world. There are numerous reasons behind its popularity, including no single point of failure, exceptional horizontal scaling with a data layout designed as a perfect fit for time-series data.

However, despite these perks, like any other system, Cassandra is prone to performance issues. This makes monitoring imperative. And it all starts with knowing what to measure. In this article, we will explain the key Cassandra performance metrics you should monitor to make sure everything is up and running at all times.

What Is Cassandra and How Does It Work?

Let’s keep it short – Apache Cassandra is a distributed NoSQL database designed to provide fault-tolerant and highly available architecture with performance in mind.

As a distributed system Cassandra is built out of nodes. A node is a single instance of Apache Cassandra that can operate on its own. Multiple nodes can form a cluster – a distributed system holding common data and responding to query requests. Cassandra works in a master-less architecture where each node communicates in a peer to peer fashion using a protocol known as Gossip. The gossip protocol is designed so that each node is informed about the state of all other nodes and a single node performs gossip communication with up to three other nodes every second.

The cluster can be divided into data centers and racks, just like the real-life data centers are divided. In Cassandra terminology, a data center is designed to hold multiple racks and a single rack holds a complete replica of the data.

\"image\"

Cassandra Cluster Logical Overview

When it comes to the data, Cassandra stores it in tables that are organized the same way as in any other database – in rows and columns. A single table is called a column family. The tables themselves are grouped into keyspaces, where a keyspace usually holds logically similar data – for example, from a business perspective. The keyspace is also used for data replication, and the replication itself is configured on a keyspace level.

Getting back to the tables. Each table defines a primary key that is built of the partition key and the clustering columns. Cassandra uses the partition key to index the data. All data that share a common partition key make a single data partition – a basic unit for data retrieval and storage. The clustering columns are optional.

Needless to say, Apache Cassandra is a complicated, distributed system and it’s not uncommon for users to encounter operation problems and difficulties. Everything breaks eventually, from the low-level bare metal components, up to the high-level software. It is not unusual for users to deal with network issues and CPU utilization problems, especially on very large clusters. Cassandra is written in Java and uses both off-heap and heap memory, which means that as the volume of data grows, you may hit issues with the garbage collector. Finally, because of the amount of data that you will process you may need to deal with the hard disk space and performance of your I/O subsystem. All of these can be avoided by keeping an eye for the relevant metrics with the help of a good Cassandra monitoring tool.

How Is Cassandra Performance Measured?

The most complex, distributed systems provide a set of metrics that you should take care of, monitor, and alert on to ensure that your system is healthy and working well. Apache Cassandra is no different. It provides a plethora of performance metrics which we can divide into three categories:

  • Dedicated Apache Cassandra metrics that describe how the system and its parts perform.
  • Java Virtual Machine metrics that tell you about the execution environment on which Apache Cassandra is running.
  • Operating system metrics describing the metrics related to the bare metal servers, virtual machines, or containers, depending on the environment that you are using.

Dedicated Cassandra Performance Metrics

When monitoring Apache Cassandra clusters, is the metrics that the distributed data store exposes via the JMX interface. There are many Cassandra performance metrics exposed in the JMX and having visibility into most of them is a good idea. You never know what can be useful when troubleshooting.

Nodes

One of the most important Cassandra metrics is the number of nodes that are currently available and connected to form a cluster. The ability to store the data and respond to queries is directly related to the availability of nodes.

Compaction Metrics

Compaction is the operation of merging multiple smaller instances of SSTable into one bigger SSTable that contains all the data from the smaller tables. Because of that, it can be very expensive and resource-consuming. Having visibility into compaction performance is critical for long-term observability – the Cassandra monitoring tool of your choice needs to provide the number of compactions and the number of compacted bytes.

During compaction, until the process ends, the total disk space used may be double that before the compaction. Because of that, you should consider leaving about 50% of space free to account for compactions and, of course, set up appropriate alerts to inform you when the amount of free disk space is close to a level where compaction could fail.

Read and Write Performance Metrics

The next set of metrics is dedicated to clients and the read and write side of the operations. You should measure the number of reads happening in a given period, the request latency, and the number of timeouts and failures. Your Cassandra monitoring tool should provide the top-level view and allow for slicing and dicing through the data showing you the aggregated view, per node view, per keyspace view, and per table view. The same goes for write operations.

You should see the number of write requests happening and write latency. Local writes and reads may also be important when troubleshooting.

Table Metrics

Table metrics are also essential. The ones you should pay close attention to are partition size, tombstone scans, and the number of SSTables per read.

Partition Size

Partition size is crucial for cluster performance. Cassandra uses it as a unit of data storage, replication, and retrieval, thus directly dictating the performance of your Cassandra tables. The ideal partition size varies but is usually below 100MB and not less than 10 – 20MB.

Tombstones

Cassandra produces tombstones when you delete the data. They are markers of the deleted data. Data in Cassandra is immutable by design, and because of that, it can only be physically removed from the SSTable during compactions. Because of that, you should keep an eye on how they affect your disk space.

SSTables Per Read

Similar to tombstones, the number of SSTables per read is related to the immutability of the data in Cassandra. A single table can be built of multiple SSTables, which are written sequentially. A single read operation can result in reading multiple SSTables to retrieve the relevant data. The more SSTables Cassandra needs to read to return the data, the more resources are required to complete the read operation. This is why you should minimize the number whenever possible.

Other Metrics

As we mentioned earlier, other Apache Cassandra performance metrics can be helpful and you should consider monitoring them.

Caches

There are two types of caches in Cassandra – the key cache and the row cache. Cassandra uses the key cache to store the location of row keys in memory so that the rows can be accessed without the need to hit the disk. The row cache stores the rows themselves in memory. By using the caches, Cassandra reduces the need to read the data from the disk and trades the memory usage for performance.

You need to monitor the key cache requests and row cache requests, which tell how many requests to a given cache type were made, and the key cache hit ratio and the row cache hit ratio, which show the percentage of results retrieved from the cache instead of the disk.

Threadpool

Cassandra is designed to handle the high load, withstand backpressure, and perform asynchronous tasks. Monitoring various thread pools is crucial for understanding Cassandra’s performance and bottlenecks. Each thread pool exposes the number of active, pending, and blocked tasks. Accumulated, pending, and blocked tasks usually tell about performance issues and the need for more processing power or different data and query architecture.

Bloom Filter

In the read path, Cassandra merges the data stored on a disk inside the SSTables with the data stored in memory. To minimize the amount of checking for data existence in the SSTables on the disk Cassandra uses a data structure called bloom filter.

The bloom filter is a probabilistic data structure that can tell Cassandra that the data is definitely not in a given file or that the data may be present in a given file. The key metrics to monitor here are the amount of space used by bloom filters, the number of false positives, and the ratio. You can reduce the number of false positives by assigning more memory to the bloom filters.

Java Virtual Machine Metrics

Apache Cassandra is a JVM-based application that comes with all the usual JVM pros and cons. From the developer’s perspective, memory management is easier and requires less hassle – you just use an object and forget about it, letting the JVM do the cleaning up. But that means that something has to clean up all the unused objects in memory. This is where the Java Garbage Collection comes in and the metrics that come with it.

A proper Cassandra monitoring tool should provide metrics that allow you to check and troubleshoot issues with the Java Virtual Machine, such as JVM memory utilization and garbage collection count and time. You can read more about them in our guide about JVM metrics.

Operating System Metrics

You can’t ignore Operating System metrics either. Information such as CPU utilization, memory usage, and disk usage is essential and can play a major role when it comes to Cassandra performance.

CPU Utilization

Your CPU is used for data processing and query handling. The more spare CPU cycles you have on a given node, the data and queries it can process. The user part of the CPU usage will show you your Cassandra process needs, while the wait can point to a bottleneck in I/O or network. As with every Java application, CPU cycles are also needed for garbage collection, so keep that in mind when planning.

Memory Usage

Memory usage is crucial for every JVM-based application. The newest version of Cassandra leverages both off-heap and heap memory. This means that you not only need to set the heap size of your Cassandra nodes correctly but also have enough off-heap memory for keeping your cluster performance at its best.

Disk Usage

Disk and I/O are crucial – Cassandra keeps its data on the disk, and each query may require a substantial number of I/O operations to return the results. You need to be sure that your hardware can handle your data retrieval needs. You also need to be sure that you have enough space to hold your data and handle the compaction process.

Monitor Cassandra Performance with Sematext

\"monitoring

Sematext Cloud and its Apache Cassandra monitoring integration provide all that you need to monitor your distributed database. Everything is within a single view available without distractions:

  • The overview report gives you a perfect start point for your metrics, painting a picture of the whole cluster.
  • A dedicated Cassandra report that provides an in-depth view of all relevant metrics related to the distributed database.
  • The OS report provides necessary operating system metrics such as CPU and memory utilization and visibility into your network traffic.
  • Finally, the JVM metrics give the full view of the Java Virtual Machine, such as metrics related to garbage collection and per-heap space memory utilization.

Using the dedicated split-view, you can correlate all the available metrics with other metrics, logs, and real user monitoring data, making Sematext a perfect visibility tool.

Sematext allows you to set up alerts on any metric or log the event and supports both threshold-based and anomaly-based alerts for full flexibility. You don’t have to watch your metrics over and over. Once you configure your alerts, you can sleep well, and Sematext will let you know if something is wrong.

If you want to see how Sematext stacks against similar solutions, read our article about the best Cassandra monitoring tools available today.

Get Started with Cassandra Monitoring

As the distributed database Apache Cassandra can quickly become an operational challenge without visibility into what is happening from a global perspective as well as on a node level. You need to have full visibility from top to bottom, but that is not enough. You need to be sure that your monitoring system can notify you when an issue happens and also predict issues before your customers notice them.

One of the tools that will give you all of that is Sematext’s Apache Cassandra monitoring integration. String monitoring your Cassandra cluster by creating the Sematext Cloud account and then the Cassandra monitoring App. Don’t forget to create a Logs App as well to ship you Cassandra logs for a full observability experience.

You might also like

Products

Services

About

Contact

\n© Sematext Group. All rights reserved\n
Terms Of Service · Privacy Policy

\"Software\n\"Software\n\"GetApp\n\"Crozdesk\n\"Crozdesk\n\"Crozdesk
","id":"070dd491-a966-5c75-86e3-0327d7804b22","title":"How Do You Monitor Cassandra Performance: Key Metrics to Measure","origin_url":"https://sematext.com/blog/cassandra-monitoring/","url":"https://sematext.com/blog/cassandra-monitoring/","wallabag_created_at":"2021-11-08T17:30:27+00:00","published_at":"2021-10-04T10:55:25+00:00","published_by":"['Rafal Kuć']","reading_time":11,"domain_name":"sematext.com","preview_picture":"https://sematext.com/wp-content/uploads/2021/10/critical-cassandra-metrics-to-monitor.jpg","tags":["monitoring","cassandra","performance"],"description":"Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system..."},{"content":"

This is our third post in our series on performance tuning with Apache Cassandra. In our first post, we discussed how we can use Flame Graphs to visually diagnose performance problems. In our second post, we discussed JVM tuning, and how the different JVM settings can have an affect on different workloads.

In this post, we’ll dig into a table level setting which is usually overlooked: compression. Compression options can be specified when creating or altering a table, and it defaults to enabled if not specified. The default is great when working with write heavy workloads, but can become a problem on read heavy and mixed workloads.

\n

Before we get into optimizations, let’s take a step back to understand the basics of compression in Cassandra. Once we’ve built a foundation of knowledge, we’ll see how to apply it to real world workloads.

\n

How it works

\n

When we create a table in Cassandra, we can specify a variety of table options in addition to our fields. In addition to options such as using TWCS for our compaction strategy, specifying gc grace seconds, and caching options, we can also tell Cassandra how we want it to compress our data. If the compression option is not specified, LZ4Compressor will be used, which is known for it’s excellent performance and compression rate. In addition to the algorithm, we can specify our chunk_length_in_kb, which is the size of the uncompressed buffer we write our data to as an intermediate step before writing to disk. Here’s an example of a table using LZ4Compressor with 64KB chunk length:

\n
create table sensor_data ( \n    id text primary key, \n    data text) \nWITH compression = {'sstable_compression': 'LZ4Compressor', \n                    'chunk_length_kb': 64};\n
\n

We can examine how well compression is working at the table level by checking tablestats:

\n
$ bin/nodetool tablestats tlp_stress\nKeyspace : tlp_stress\n\tRead Count: 89766\n\tRead Latency: 0.18743983245326737 ms\n\tWrite Count: 8880859\n\tWrite Latency: 0.009023213069816781 ms\n\tPending Flushes: 0\n\t\tTable: sensor_data\n\t\tSSTable count: 5\n\t\tOld SSTable count: 0\n\t\tSpace used (live): 864131294\n\t\tSpace used (total): 864131294\n\t\tOff heap memory used (total): 2472433\n\t\tSSTable Compression Ratio: 0.8964684393508305\n\t\tCompression metadata off heap memory used: 140544\n
\n

The SSTable Compression Ratio line above tells us how effective compression is. Compression ratio is calculated by the following:

\n
compressionRatio = (double) compressed/uncompressed;\n
\n

meaning the smaller the number, the better the compression. In the above example our compressed data is taking up almost 90% of the original data, which isn’t particularly great.

\n

How data is written

\n

I’ve found digging into the codebase, profiling and working with a debugger to be the most effective way to learn how software works.

\n

When data is written to / read from SSTables, we’re not dealing with convenient typed objects, we’re dealing with streams of bytes. Our compressed data is written in the CompressedSequentialWriter class, which extends BufferedDataOutputStreamPlus. This writer uses a temporary buffer. When the data is written out to disk the buffer is compressed and some meta data about it is recorded to a CompressionInfo file. If there is more data than available space in the buffer, the buffer is written to, flushed, and the buffer starts fresh to be written to again (and perhaps flushed again). You can see this in org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.java:

\n
@Override\npublic void write(byte[] b, int off, int len) throws IOException\n{\n    if (b == null)\n        throw new NullPointerException();\n    // avoid int overflow\n    if (off < 0 || off > b.length || len < 0\n        || len > b.length - off)\n        throw new IndexOutOfBoundsException();\n    if (len == 0)\n        return;\n    int copied = 0;\n    while (copied < len)\n    {\n        if (buffer.hasRemaining())\n        {\n            int toCopy = Math.min(len - copied, buffer.remaining());\n            buffer.put(b, off + copied, toCopy);\n            copied += toCopy;\n        }\n        else\n        {\n            doFlush(len - copied);\n        }\n    }\n}\n
\n

The size of this buffer is determined by chunk_length_in_kb.

\n

How data is read

\n

The read path in Cassandra is (more or less) the opposite of the write path. We pull chunks out of SSTables, decompress them, and return them to the client. The full path is a little more complex - there’s a a ChunkCache (managed by caffeine) that we go through, but that’s beyond the scope of this post.

\n

During the read path, the entire chunk must be read and decompressed. We’re not able to selectively read only the bytes we need. The impact of this is that if we are using 4K chunks, we can get away with only reading 4K off disk. If we use 256KB chunks, we have to read the entire 256K. This might be fine for a handful of requests but when trying to maximize throughput we need to consider what happens when we have requests in the thousands per second. If we have to read 256KB off disk for ten thousand requests a second, we’re going to need to read 2.5GB per second off disk, and that can be an issue no matter what hardware we are using.

\n

What about page cache?

\n

Linux will automatically leverage any RAM that’s not being used by applications to keep recently accessed filesystem blocks in memory. We can see how much page cache we’re using by using the free tool:

\n
$ free -mhw\n              total        used        free      shared     buffers       cache   available\nMem:            62G        823M         48G        1.7M        261M         13G         61G\nSwap:          8.0G          0B        8.0G\n
\n

Page cache can be a massive benefit if you have a working data set that fits in memory. With smaller data sets this is incredibly useful, but Cassandra was built to solve big data problems. Typically that means having a lot more data than available RAM. If our working data set on each node is 2 TB, and we only have 20-30 GB of free RAM, it’s very possible we’ll serve almost none of our requests out of cache. Yikes.

\n

Ultimately, we need to ensure we use a chunk length that allows us to minimize our I/O. Larger chunks can compress better, giving us a smaller disk footprint, but we end up needing more hardware, so the space savings becomes meaningless for certain workloads. There’s no perfect setting that we can apply to every workload. Frequently, the most reads you do, the smaller the chunk size. Even this doesn’t apply uniformly; larger requests will hit more chunks, and will benefit from a larger chunk size.

\n

The Benchmarks

\n

Alright - enough with the details! We’re going to run a simple benchmark to test how Cassandra performs with a mix of read and write requets with a simple key value data model. We’ll be doing this using our stress tool, tlp-stress (commit 40cb2d28fde). We will get into the details of this stress tool in a later post - for now all we need to cover is that it includes a key value workload out of the box we can leverage here.

\n

For this test I installed Apache Cassandra 3.11.3 on an AWS c5d.4xlarge instance running Ubuntu 16.04 following the instructions on cassandra.apache.org, and updated all the system packages using apt-get upgrade. I’m only using a single node here in order to isolate the compression settings and not introduce noise from the network overhead of running a full cluster.

\n

The ephemeral NVMe disk is using XFS and mounted it at /var/lib/cassandra. I set readahead using blockdev --setra 0 /dev/nvme1n1 so we can see the impact that compression has on our disk requests and not hide it with page cache.

\n

For each workload, I put the following command in a shell script, and ran tlp-stress from a separate c5d.4xlarge instance (passing the chunk size as the first parameter):

\n
$ bin/tlp-stress run KeyValue -i 10B -p 10M --populate -t 4 \\\n  --replication \"{'class':'SimpleStrategy', 'replication_factor':1}\" \\\n  --field.keyvalue.value='book(100,200)' -r .5  \\\n  --compression \"{'chunk_length_in_kb': '$1', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}\" \\\n  --host 172.31.42.30\n
\n

This runs a key value workload across 10 million partitions (-p 10M), pre-populating the data (--populate), with 50% reads (-r .5), picking 100-200 words from of one of the books included in the stress tool (--field.keyvalue.value='book(100,200)'). We can specify a compression strategy using --compression.

\n

For the test I’ve used slightly modified Cassandra configuration files to reduce the effect of GC pauses by increasing the total heap (12GB) as well as the new gen (6GB). I spend a small amount of time on this as optimizing it perfectly isn’t necessary. I also set compaction throughput to 160.

\n

For the test, I monitored the JVM’s allocate rate using the Swiss Java Knife (sjk-plus) and disk / network / cpu usage with dstat.

\n

Default 64KB Chunk Size

\n

The first test used the default of 64KB chunk length. I started the stress command and walked away to play with my dog for a bit. When I came back, I was through about 35 million requests:

\n

\"stress

\n

You can see in the above screenshot our 5 minute rate is about 22K writes / second and 22K reads/ second. Looking at the output of dstat at this time, we can see we’re doing between 500 and 600MB / second of reads / second:

\n

\"DStat

\n

Memory allocation fluctuated a bit, but it hovered around 1GB/s:

\n

\"sjk

\n

Not the most amazing results in the world. Of the disk reads, some of that throughput can be attributed to compaction, which we’ll always have to contend with in the real world. That’s capped at 160MB/s, leaving around 400MB/s to handle reads. That’s a lot considering we’re only sending 25MB across the network. That means we’re doing over 15x the disk I/O than our network I/O. We are very much disk bound in this workload.

\n

4KB Chunk Size

\n

Let’s see if the 4KB chunk size does any better. Before the test I shut down Cassandra, cleared the data directory, and started things back up. I ran the same stress test above using the above shell script, passing 4 as the chunk size. I once again played fetch with my dog for a bit and came back after around the same time as the previous test.

\n

Looking at the stress output, it’s immediately obvious there’s a significant improvement:

\n

\"stress\"

\n

In almost every single metric reported by the metric library the test with 4KB outperforms the 64KB test. Our throughput is better (62K ops / second vs 44K ops / second in the 1 minute rate), and our p99 for reads is better (13ms vs 24ms).

\n

If we’re doing less I/O on each request, how does that impact our total disk and network I/O?

\n

\"dstat

\n

As you can see above, there’s a massive improvement. Disk I/O is significantly reduced from making smaller (but more) requests to disk, and our network I/O is significantly higher from responding to more requests.

\n

\"sjk

\n

It was initially a small surprise to see an increased heap allocation rate (because we’re reading WAY less data into memory), but this is simply the result of doing a lot more requests. There are a lot of objects created in order to satisfy a request; far more than the number created to read the data off disk. More requests results in higher allocation. We’d want to ensure those objects don’t make it into the Old Gen as we go through JVM tuning.

\n

Off Heap Memory Usage

\n

The final thing to consider here is off heap memory usage. Along side each compressed SSTable is compression metadata. The compression files have names like na-9-big-CompressionInfo.db. The compression metadata is stored in memory, off the Cassandra heap. The size of the offheap usage is directly proportional to the amount of chunks used. More chunks = more space used. More chunks are used when a smaller chunk size is used, hence more offheap memory is used to store the metadata for each chunk. It’s important to understand this trade off. A table using 4KB chunks will use 16 times the memory as one using 64KB chunks.

\n

In the example I used above the memory usage can be seen as follows:

\n
Compression metadata off heap memory used: 140544 \n
\n

Changing Existing Tables

\n

Now that you can see how a smaller chunk size can benefit read heavy and mixed workloads, it’s time to try it out. If you have a table you’d like to change the compression setting on, you can do the following at the cqlsh shell:

\n
cqlsh:tlp_stress> alter table keyvalue with compression = {'sstable_compression': 'LZ4Compressor', 'chunk_length_kb': 4};\n
\n

New SSTables that are written after this change is applied will use this setting, but existing SSTables won’t be rewritten automatically. Because of this, you shouldn’t expect an immediate performance difference after applying this setting. If you want to rewrite every SSTable immediately, you’ll need to do the following:

\n
nodetool upgradesstables -a tlp_stress keyvalue\n
\n

Conclusion

\n

The above is a single test demonstrating how a tuning compression settings can affect Cassandra performance in a significant way. Using out of the box settings for compression on read heavy or mixed workloads will almost certainly put unnecessary strain on your disk while hurting your read performance. I highly recommend taking the time to understand your workload and analyze your system resources to understand where your bottleneck is, as there is no absolute correct setting to use for every workload.

\n

Keep in mind the tradeoff between memory and chunk size as well. When working with a memory constrained environment it may seem tempting to use 4KB chunks everywhere, but it’s important to understand that it’ll use more memory. In these cases, it’s a good idea to start with smaller tables that are read from the most.

","id":"b16bdb40-b8c3-54e5-8afa-60ee43153f3d","title":"Apache Cassandra Performance Tuning - Compression with Mixed Workloads","origin_url":"https://thelastpickle.com/blog/2018/08/08/compression_performance.html","url":"https://thelastpickle.com/blog/2018/08/08/compression_performance.html","wallabag_created_at":"2019-12-02T18:51:11+00:00","published_at":null,"published_by":"['']","reading_time":11,"domain_name":"thelastpickle.com","preview_picture":"https://thelastpickle.com/android-chrome-192x192.png","tags":["cassandra","troubleshooting and tuning","performance"],"description":"This is our third post in our series on performance tuning with Apache Cassandra. In our first post, we discussed how we can use Flame Graphs to visually diagnose performance problems. In our second..."},{"content":"

There are multiple dimensions where Cassandra performance can be tuned. Some of them are described below:
Write Operations:
Commit log and data dirs (sstables) should be on different disks. Commit log uses sequential write however, if SSTables share the same drive with commit log , I/O contention between commit log & SSTables may deteriorate commit log writes and SSTable reads.
 
Read Operations:
A good rule of thumb is 4 concurrent_reads per processor core. May increase the value for systems with fast I/O storage.
 
Cassandra Compaction Contention:
Reduce the frequency of memtable flush by increasing the memtable size or preventing too pre-mature flushing. Less frequent memtable flush results in fewer SSTables files and less compaction. Fewer compaction reduces SSTables I/O contention, and therefore improves read operations. Bigger memtables absorb more overwrites for updates to the same keys, and therefore accommodating more read/write operations between each flushes.
 
Memory Cache:
Do not increase Cassandra cache size unless there is enough physical memory (RAM). Avoid memory swapping at any cost.
 
Row Cache:
The row cache holds the entire content of a row in memory. It provides data caching instead of reading data from the disk. good if column’s data is small so the cache is big enough to hold most of the hotspot data. Bad if column’s data is too large so the cache is not big enough to hold most of the hotspot data. It’s bad for high write/read ratios. By default, it is off. If hit ratio is below 30%, row cache should be disabled.
 
Key Cache Tuning:
The key cache holds the location of data in memory for each column family. Its Effective if there are hot data spot & cannot use row cache effectively because of the large column size. By default, Cassandra caches 200000 keys per column family. Use absolute number for keys_cached instead of percentage.
 
JVM:
Minimum and Maximum Java Heap Size should be half of available physical memory. Size of young generation heap should be 1/4 of Java Heap. Do NOT increase the size without confirming there are enough available physical memory- Always reserves memory for OS File cache.
A detailed understanding of Apache Cassandra is available in this blog post for your perusal!

","id":"2f1edaee-2329-5d58-a6d0-6dce016c28f4","title":"Tuning Cassandra Performance","origin_url":"https://intellipaat.com/blog/tutorial/cassandra-tutorial/tuning-cassandra-performance/","url":"https://intellipaat.com/blog/tutorial/cassandra-tutorial/tuning-cassandra-performance/","wallabag_created_at":"2019-12-02T18:50:11+00:00","published_at":"2019-09-05T08:00:00+00:00","published_by":"['']","reading_time":1,"domain_name":"intellipaat.com","preview_picture":"https://intellipaat.com/blog/wp-content/uploads/2020/09/Certification-in-Bigdata-Analytics-IITG.jpg","tags":["cassandra","troubleshooting and tuning","performance"],"description":"There are multiple dimensions where Cassandra performance can be tuned. Some of them are described below:Write Operations:Commit log and data dirs (sstables) should be on different disks. Commit log u..."},{"content":"

In this topic, i will  cover the basics of general Apache Cassandra performance tuning: when to do performance tuning, how to avoid and identify problems, and methodologies to improve.

When do you need to tune performance ?

optimizing:

when things work but could be better. we want to get better performance.

Troubleshooting:

fixing a problem that impact performance could actually be broken, could just be slow in clusters, something broken can manifest as slow performance.

What are some examples of performance related complaints an admin might receive regarding Cassandra ?

Performance-related complaints:

Latency, Throughput and the U.S.E Method:

Bad methodology
how not to approach performance-related problems?

streetlight anti-method\n\trandom change anti-method\n\tblame someone else anti-method

In performance tuning, what are we trying to improve ?

latency – how long a cluster,node,server or I/O subsystem takes to respond to a request
throughput – how many transactions of a given size (or range) a cluster,node or I/O subsystem can complete in a given timeframe ?
how many operations by seconds our cluster node are processing ?
But we can’t forget COST !!

How are latency and throughput related ?

theoretically, they are independent of each other.
however, change in latency can have a proportional effect on throughput

What causes a change in latency and throughput ?

Understanding performance tuning:utilization,saturation,errors,availability

What is utilization ? saturation ? errors ? availability ?

Utilization – how heavily are the resources being stressed.
Saturation – the degree to which a resource has queued work it cannot service
Errors – recoverable failure or exception events in the course normal operation
Availability – whether a given resource can service work or not

GOAL:

What is the first step in achieving any performance 
tuning goal ?
setting a goal!
what are some examples of commonly heard cassandra performance tuning goals?
reads should be faster
writes to table x should be faster
the cluster should be able to complete x transactions per second

what should a clearly defined performance goal take into account ?

Writing SLA (service level agreement)
we need to know :
what is the type of operation or query ?
read or write workload
select, insert,update or delete
we need to understand latency: expressed as percentile
rank e.g 95th percentile read latency is 2 ms
throughput: operations per second
size: expressed in average bytes
We have to think about duration: expressed in minutes or hours
scope: keyspace, table, query
Example of SLA: “the cluster should be able to sustain
20000 2KB read operations per second from table X for
two hours with a 95th percentile read latency of 3 ms.”

After setting a goal, how can achievement of a goal verified?

timing tooks in your application
query tracing
jmeter test plan
customizable cassandra-stress

Time in computer performance tuning:

how long is a millisecond ?
why do we care about milliseconds ?

Common latency timings in cassandra:

  reads from main memory should take between 36 and 130 microseconds
\n  reads from an SSD should take between 100 microseconds and 12 milliseconds
\n  reads from a Serial Attached SCSI rotational drive should take between 8 milliseconds and 40 milliseconds
\n  reads from a SATA rotational drive take more than 15 milliseconds
\n

Example:
Workload characterization:
classroom use case and cassandra story middle sized financial firm uses cassandra to manage distributed data 42 million stock quotes driven by a particular set of queries

what queries drove this data model ?

retrieve information for a specific stock trade by trade ID find all information about stock trades for a specific stock ticker and range timestamps find all information about stock trades that occurred on a specific date over a short period of time

How do you characterize the workload ?

what is the load being placed on your cluster
calling application or API
remote IP address

Who is causing the load ?

code path or stack trace

Why is the load being called ?

What are the load characteristics
throughput
Direction(read/write)
include variance
keyspace and column family
How do you characterize your workload ?
how is the load changing over time and is there
a daily pattern ?
is your workload read heavy or write heavy?
how big is your data ?
how much data on each node (bytes on node=data density)?
does active data fit in buffer cache ?

Performance impact of Data Model: How does the data model affect performance ?

poorly shaped rows (too narrow or too wide (we have partition to large)
hotspots (particular areas with a lot of reads/writes)
poor primary or secondary indexes
too many tombstones (lot of delete)

So data model considerations:

understand how primary key affects performance
take a look at query patterns and adjust how tables are modeled
see how replication factor and/or consistency level impact performance
change in compaction strategy can have a positive (or negative) impact
parallelize reads/writes if necessary
look at how moving infrequently accessed data can improve performance
see how per column family cache is having an impact
what is the relationship between the data model and cassandra’s read path optimizations (key/row cache,bloom filters, index) ?
nesting data (allows for greater degree of flexibility in the column family structure.) (keep all data to the same partition to satisfy a given query) but it can be easy to find model to keep most active data sets in cache, frequently accessed data which are in cache can improve performance.

Methodologies:

active performance tuning: we focus in the particular problem and we verify if it is fixed – suspect there’s a problem, isolate problem using tools, determine if problem is in cassandra ,environment or both.
verify problems and test for reproducibility,fix problems using tuning strategies, test, test and test again
Passive performance tuning : regular system “sanity checks”: looking some giving threshold,something we adjust for growth regularly monitor key health areas in cassandra/environment.
identify and tune for future growth/scalability.
apply tuning strategies as needed.
we have to use the USE Method as tool for troubleshooting
this method gives us the methodology to look for on all components of the system not only one. it is the strategy defined by Brendan Gregg http://www.brendangregg.com/USEmethod/use-linux.html.

Also performs a health check of various system components to identity bottlenecks and errors
separated by components, type and metric to narrow scope and find location of problem.

what are two things performance tuning to improve ? latency and throuput.
what are two types of performance tuning methodologies ? active and passive

what tool can be used to get a performance baseline ? jmeter or cassandra-stress

Cassandra-stress:

Interpreting the output of cassandra-stress

Each line reports data for the interval between the last elapsed time and current elapsed time, which is set by the –progress-interval option (default 10 seconds).

[hduser@base ~]$ cassandra-stress write -node 192.168.56.71 
\nINFO  04:11:53 Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
\nINFO  04:11:56 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
\nINFO  04:11:56 New Cassandra host /192.168.56.71:9042 added
\nINFO  04:11:56 New Cassandra host /192.168.56.72:9042 added
\nINFO  04:11:56 New Cassandra host /192.168.56.73:9042 added
\nINFO  04:11:56 New Cassandra host /192.168.56.74:9042 added
\nConnected to cluster: Training_Cluster
\nDatatacenter: datacenter1; Host: /192.168.56.71; Rack: rack1
\nDatatacenter: datacenter1; Host: /192.168.56.72; Rack: rack1
\nDatatacenter: datacenter1; Host: /192.168.56.73; Rack: rack1
\nDatatacenter: datacenter1; Host: /192.168.56.74; Rack: rack1
\nCreated keyspaces. Sleeping 1s for propagation.
\nSleeping 2s...
\nWarming up WRITE with 50000 iterations...
\nFailed to connect over JMX; not collecting these stats
\nWARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level analysis only
\nRunning with 4 threadCount
\nRunning WRITE with 4 threads until stderr of mean < 0.02
\nFailed to connect over JMX; not collecting these stats
\ntype,      total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
\ntotal,           402,     403,     403,     403,    10,4,     5,9,    31,4,    86,2,   132,7,   132,7,    1,0,  0,00000,      0,      0,       0,       0,       0,       0
\ntotal,           979,     482,     482,     482,     8,1,     6,3,    17,5,    45,9,    88,3,    88,3,    2,2,  0,06272,      0,      0,       0,       0,       0,       0
\ntotal,          1520,     530,     530,     530,     7,5,     6,4,    14,8,    24,3,   103,9,   103,9,    3,2,  0,07029,      0,      0,       0,       0,       0,       0
\ntotal,          1844,     321,     321,     321,    11,9,     6,4,    33,0,   234,5,   248,5,   248,5,    4,2,  0,06134,      0,      0,       0,       0,       0,       0
\ntotal,          2229,     360,     360,     360,    11,3,     5,4,    43,6,   127,6,   145,4,   145,4,    5,3,  0,06577,      0,      0,       0,       0,       0,       0
\ntotal,          2457,     199,     199,     199,    20,2,     5,9,    82,2,   125,4,   203,6,   203,6,    6,4,  0,11009,      0,      0,       0,       0,       0,       0
\ntotal,          2904,     443,     443,     443,     8,9,     7,0,    23,0,    37,3,    56,1,    56,1,    7,4,  0,09396,      0,      0,       0,       0,       0,       0
\ntotal,          3246,     340,     340,     340,    11,7,     7,4,    41,6,    87,0,   101,0,   101,0,    8,5,  0,08625,      0,      0,       0,       0,       0,       0
\ntotal,          3484,     235,     235,     235,    16,6,     7,2,    76,1,   151,8,   152,1,   152,1,    9,5,  0,09208,      0,      0,       0,       0,       0,       0
\ntotal,          3679,     196,     196,     196,    19,5,     8,1,    86,0,   156,5,   174,8,   174,8,   10,5,  0,09960,      0,      0,       0,       0,       0,       0
\ntotal,          4083,     369,     369,     369,    11,1,     7,3,    38,3,    90,8,   114,9,   114,9,   11,6,  0,09041,      0,      0,       0,       0,       0,       0
\ntotal,          4411,     320,     320,     320,    11,4,     7,4,    39,7,    61,1,    93,6,    93,6,   12,6,  0,08422,      0,      0,       0,       0,       0,       0
\ntotal,          4683,     227,     227,     227,    17,1,     6,0,    90,1,   153,3,   199,8,   199,8,   13,8,  0,08478,      0,      0,       0,       0,       0,       0
\ntotal,          5131,     445,     445,     445,     8,9,     7,6,    19,3,    26,8,    50,8,    50,8,   14,8,  0,07997,      0,      0,       0,       0,       0,       0
\ntotal,          5661,     521,     521,     521,     7,5,     5,4,    17,5,    63,1,    89,2,    89,2,   15,8,  0,07788,      0,      0,       0,       0,       0,       0
\ntotal,          6179,     512,     512,     512,     7,7,     6,2,    16,7,    39,3,    59,1,    59,1,   16,8,  0,07578,      0,      0,       0,       0,       0,       0
\ntotal,          6427,     245,     245,     245,    15,9,     6,1,    56,3,    94,0,   180,6,   180,6,   17,8,  0,07567,      0,      0,       0,       0,       0,       0
\ntotal,          6831,     394,     394,     394,    10,2,     5,6,    39,0,    90,4,   111,7,   111,7,   18,9,  0,07129,      0,      0,       0,       0,       0,       0
\ntotal,          7071,     235,     235,     235,    16,9,     7,4,    58,3,   149,7,   235,3,   235,3,   19,9,  0,07150,      0,      0,       0,       0,       0,       0
\ntotal,          7532,     455,     455,     455,     8,7,     6,1,    17,2,    90,5,   142,2,   142,2,   20,9,  0,06840,      0,      0,       0,       0,       0,       0
\ntotal,          7890,     353,     353,     353,    10,9,     7,3,    35,4,    80,6,   149,5,   149,5,   21,9,  0,06532,      0,      0,       0,       0,       0,       0
\ntotal,          8172,     288,     288,     288,    13,6,     7,1,    45,6,    89,1,    89,7,    89,7,   22,9,  0,06374,      0,      0,       0,       0,       0,       0
\ntotal,          8355,     171,     171,     171,    22,7,     9,1,    82,5,   137,7,   151,5,   151,5,   24,0,  0,06656,      0,      0,       0,       0,       0,       0
\ntotal,          8614,     235,     235,     235,    16,9,     8,0,    56,4,    98,7,   139,5,   139,5,   25,1,  0,06622,      0,      0,       0,       0,       0,       0
\ntotal,          9027,     402,     402,     402,     9,6,     8,5,    20,7,    26,5,    30,8,    30,8,   26,1,  0,06346,      0,      0,       0,       0,       0,       0
\ntotal,          9496,     463,     463,     463,     8,5,     7,6,    17,4,    23,2,    30,1,    30,1,   27,1,  0,06139,      0,      0,       0,       0,       0,       0
\ntotal,          9912,     408,     408,     408,     9,6,     7,5,    23,4,    33,0,    38,0,    38,0,   28,1,  0,05903,      0,      0,       0,       0,       0,       0
\ntotal,         10275,     359,     359,     359,    11,0,     8,8,    26,4,    33,7,    44,7,    44,7,   29,1,  0,05693,      0,      0,       0,       0,       0,       0
\ntotal,         10528,     251,     251,     251,    15,6,     9,5,    45,4,   176,1,   295,9,   295,9,   30,1,  0,05602,      0,      0,       0,       0,       0,       0
\ntotal,         10711,     181,     181,     181,    21,1,     7,8,    53,8,   340,8,   396,8,   396,8,   31,1,  0,05597,      0,      0,       0,       0,       0,       0
\ntotal,         10947,     233,     233,     233,    17,3,     9,3,    55,5,    94,7,   123,7,   123,7,   32,1,  0,05584,      0,      0,       0,       0,       0,       0
\ntotal,         11130,     177,     177,     177,    21,0,    10,6,    86,7,   115,6,   151,9,   151,9,   33,2,  0,05706,     
\n...
\n
\nResults:
\nop rate                   : 388 [WRITE:388]
\npartition rate            : 388 [WRITE:388]
\nrow rate                  : 388 [WRITE:388]
\nlatency mean              : 10,2 [WRITE:10,2]
\nlatency median            : 7,1 [WRITE:7,1]
\nlatency 95th percentile   : 25,9 [WRITE:25,9]
\nlatency 99th percentile   : 71,2 [WRITE:71,2]
\nlatency 99.9th percentile : 150,3 [WRITE:150,3]
\nlatency max               : 396,8 [WRITE:396,8]
\nTotal partitions          : 63058 [WRITE:63058]
\nTotal errors              : 0 [WRITE:0]
\ntotal gc count            : 0
\ntotal gc mb               : 0
\ntotal gc time (s)         : 0
\navg gc time(ms)           : NaN
\nstdev gc time(ms)         : 0
\nTotal operation time      : 00:02:42
\nSleeping for 15s
\n
\n
\nData\t                            Description
\n--------------------------------------------------------------------------------------------------
\ntotal\t                           :Total number of operations since the start of the test.
\ninterval_op_rate\t           :Number of operations performed per second during the interval 
\n(default 10 seconds).
\ninterval_key_rate\t           :Number of keys/rows read or written per second during the interval 
\n(normally be the same as interval_op_rate unless doing range slices).
\nlatency\tAverage latency            : for each operation during that interval.
\n95th                               :95% of the time the latency was less than the number displayed in the column 
\n(Cassandra 1.2 or later).
\n99th                               :99% of the time the latency was less than the number displayed in the column 
\n(Cassandra 1.2 or later).
\nelapsed\tNumber of seconds          :elapsed since the beginning of the test.
\n

Cassandra tuning:

successfull performance tuning is all about understanding.
understanding what some of metrics means.
how the software which we are tuning is architecture ?
in the distributed system we have to know how the software of one node
is working together with software of others nodes ?
that is for the next section come from :
how different pieces of cassandra can be put together ?

we can be talk in about cassandra, data model.
how we can get metric and how they can be performing.

The next section, we talk about environment tuning: JVM and operating system
The next section will be focussing on disk tuning and compaction tuning

Examine cluster and node health and tuning:

Discuss table design and tuning: successfull performance tuning is all about understanding.
understanding what kind of some metrics means ? how the software which we are
tuning is architecture ? with the distributed system, we have to how the software in one node works together with the software with others nodes?
some metrics we will expose
we are talking about cassandra, data model then we will talk about the environment tuning as the JVM and the operating system, also we will discuss disk tuning and compaction tuning

let’s talk about cluster and node tuning:what activities are in a cassandra cluster
happen between nodes ?
what happening on the network ? when we dig into some performance problem, we will look into all nodes not only one:
answers are: coordinator,gossip,replication,repair,read repair,bootstrapping,
node removal, node decommissioning.
we have to dig not only one node but additional node

Deeling in one node, whay does a cassandra node do ? read performance, write,
monitor,participate in the clusters, maintain consistency.
internally cassandra has lot of things to do, there is a architecture that
how is it organize ? how does cassandra organize all of that work ?
the answer of this is SEDA (staged event driven architecture), so we have
several thread pool and the messaging service to know how the queue works.

What is a thread pool ?
example, we have thread pool:workers:6

                                                                                              worker thread
\n  task          queue:max pending tasks:7                worker thread
\n  task task  task task  task  task task task task =>     worker thread 
\n  task                                task               worker thread
\n  blocked tasks
\n

what are cassandra’s thread pools ?

read readstage:32

write (mutationstage,flushwriter,memtablepostflusher,countermutation,
migrationstage)
monitor (memorymeter:1,tracing)
participate in cluster (requestresponsestage:%CPUs,pendingrangecalculator:1,gossipstage:1)
maintain consistency(commitlogarchiver:1,miscstage:1 snapshoting/replicate data after node remove)
antientropystage:1 repair consistency – merkle tree build,internal responsestage:#CPUs,HintedHanoff:1,readrepairstage:#CPUs)
we will know , for example the single thread on monitor (memorymeter:1)
which one has the configurable number of threads ? its read (readstage:32*)

the utility like nodetool tpstats which can give us metrics inside into , how much work each these pools are doing ? how many appending, active blocks:
active – number of messages pulled off the queue,currently being processed by a thread.
pending – number of messages in queue waiting for a thread.
completed – number of messages completed.
blocked -when a pool reaches its max thread count it will begin queuing.
until the max size is reached. when this is reached it will block until there is room in the queue.
total blocked/all time blocked – total number of messages that have been blocked.

cassandra thread pools: multi-threaded stages:
readstage (affected by main memory,disk) -perform local reads
mutationstage (affected by CPU,main memory,disk) – perform local insert/update,
schema merge,commit log replay,hints in progress)
requestresponsestage (affected by network,other nodes) – when a response to a request is received this is the stage used to execute any callbacks that were created with the original request.
flushwriter (affected by CPU,disk) -sort and write memtables to disk
hintedhandoo (one thread per host being sent hints,affected by disk,network,others nodes) –
sends missing mutations to others nodes
memorymeter (several separate threads) – measure memory usage and live ratio of a memtable.
readrepairstage (affected by network,others nodes) -perform read repair
countermutation (formerly replicateOnWriteStage) -performs counter writes on non-coordinator nodes and replicates after a local write
internalresponsestage -responds to non-client initiated messages, including bootstrapping and schema checking.

single-threaded stages:
gossipstage (affected by network) – gossip communication
antientropystage(affected by network,other nodes) – build merkle tree and repair consistency
migrationstage(affected by network,other nodes) – make schema changes
miscstage (affected by disk,network,pthers nodes) – snapshotting,replicating data after node remove
memtablepostflusher (affected by disk) -operations after flushing the memtable.
Discard commit log files that have all data in them persisted to sstables.
flush non-column family backed secondary indexes
tracing -for query tracing
commitlogarchiver (formally commitlog_archiver) -back up or restore a commit log.

Messages types:

handled by the readstage thread pool
(read: read data from cache or disk,range_slice: read a range of data,paged_range:
read part of a range of data)
)

handled by the mutationstage thread pool: (mutation:write (insert or update) data,
counter_mutation:changes counter columns,read_repair:update out-of-sync data discovered during a read)

handled by readresponsestage:(request_response:respond to a coordinator,_trace:
used to trace a query (enable tracing or every(trace probability) queries)
binary:deprecated

Question:why are some messages types “droppable” ?
if the message set in one of the queue to long and the to long will be decide by the timeout by the cassandra.yaml, it will be dropped.
why cassandra will do that ? why cassandra does not do my work ?
if the node has a suffisant resource to do the job, it does not dropped it.
in practice, if i have a read to give to cassandra and the timeout is elapsed and it timeout. what does the coordinator in this situation do ? it depends, maybe it can be satisfy by the consistency level to another node, then the consistency level gives the query or the return the timeout then back to the client if it is the to high consistency level.
In the read, there is not the deal, usally we can reexecute the query.

however we have another mutation (insert/delete/update) they can be to the queue to long (2 seconds) then no thing is done,
so what happen ? the coordinator can be experienced the
timeout, may be it store the hint, may be it can be back to the driver to be retry, we
don’t know, but the important thing some recovery actions can be taking in this point.
still that node who has a write doesn’t perform, that thing like read repair or repair command can back result to the client
why are some messages types “droppable”
some messages can be dropped to prioritize activities in cassandra when high resource contention occurs.
the number of these messages dropped is in the nodetool tpstats output
if you see dropped messages, you should investigate.
Question:What is the state of your cluster ? we have nodetool to gather information on our cluster or OpsCenter to gather information on our cluster.
nodetool gives us informations what happen right now, it is cummulatitve
but we do not know what happen last second,it is good for a general assessment
and also for comparaison between nodes (nodetool -h node0 tpstats, nodetool -h node0 compactionstats)

nodetool -h node0 netstats (we have to look for repair statistics) read repair
is when we are looking for discrepancy and if we find discrepancy between nodes, we will fixed by increasing cch (blocking), another read repair is the background, if we run the query probably consistency level 1 and we return no response, if we get a discrepancy. then we will increase the cch (background).

too wide: We have partition too large
too narrow:partition is to small and our query takes row on all bunch of partition
hotspots: we have a customer who decides to design partition by country code then 90% of all rows are in US and the rest on all others countries
poor primary and secondary indexes:
too many tombstones: we have a workload doing a lot of deleting and then reading data around the same partition, those things can have a huge impact on read performance.

OpsCenter

Changing logging levels with nodetool setlogginglevel:
nodetool getlogginglevels: use to get the current runtime logging levels

root@ds220:~# nodetool getlogginglevels
\nLogger Name                                                                 Log Level
\nROOT                                                                        INFO
\nDroppedAuditEventLogger                                                     INFO
\nSLF4JAuditWriter                                                            INFO
\ncom.cryptsoft                                                               OFF
\ncom.thinkaurelius.thrift                                                    ERROR
\norg.apache.lucene.index                                                     INFO
\norg.apache.solr.core.CassandraSolrConfig                                    WARN
\norg.apache.solr.core.RequestHandlers                                        WARN
\norg.apache.solr.core.SolrCore                                               WARN
\norg.apache.solr.handler.component                                           WARN
\norg.apache.solr.search.SolrIndexSearcher                                    WARN
\norg.apache.solr.update                                                      WARN
\n

nodetool setlogginglevel: used to set logging level for a service can be used instead of modifying the logback.xml file possible levels:

ALL
\nTRACE
\nDEBUG
\nINFO
\nWARN
\nERROR
\nOFF
\n

we can increase the level logging for example for namespace only to DEBUG or even TRACE

Data Model Tuning:

One off the key component of cassandra is the data model.
for performance tuning, we have the workload which place on the database. it is in the given place on cassandra table, that where the data model come .then we have software tuning (OS,JVM, Cassandra) and Hardware.
in many cases, updating hardware is an option.

When we need to diagnose the data model, cassandra provides many tools to know how our table is performing, how the query is performing.
Here is one nodetool cfstats, additionaly nodetool cfhistograms

when we identify the query which we may suspect, we have CQL tracing but we also have nodetool settraceprobability
nodetool cfstats example:
\"cfstats1\"
This one below is the aggregate number of the keyspace not for this particular table:
\"cfstats2\"
Then below the aggregate number of the table stock only:
\"cfstats3\"
Here we have 13 sstable on the disk for this table.
How much data we are store on the disk with the metrics space used? in this case we have live bytes and the total bytes is the same, that means in this particular table, we don’t have delete, ever update only insert ,however, if I do some delete or update, we will see the difference between live and total space used, the total number will be bigger than the live.
Off heap memory used (1928102) shows us how many memory this table are using. We have also other couple of metrics as bloom filter space used, bytes: 7088, and Bloom filter off heap memory used, bytes: 6984, we also have Index summary off heap memory used: 390, and compression metadata off heap memory used, bytes : 1920728, and memTable data size, bytes (36389573)
Conclusion: If I look at these numbers:
1928102 + 7088 + 6984 + 1920728 +36389573=40252475 bytes (39 MB)

I can know how much memory of ram are consuming by this table.

SSTable compression ratio: 0.173624: if I enable the compression like here. We have 17 % of compression of this table.
Number of keys: 1664 this is counting just the partition, so if we have a simple table schema , only have a partition key no cluster column , this number (1664) will be a rough number of rows on a given server.
These all metrics can be different on every server, so we have to go on each server to look at.

Memtable cell count (126342), memTable data size, bytes (36389573) and memTable switch count (107). MemTable data size (36389573) , this is a data which is storing in memory currently and they will go to flush on the disk. For memTable switch count, every time I flush memTable data to disk, this metric will increase. Memtable cell count indicates how many cell we have in memory.if we divide that by column count looking in schema definition (example 5 columns), 126342/5 = 25268,4, we can know approximativement the number of rows because Cassandra stores cell else the rows.

Local read count, local read latency, local write count and local write latency show us metrics read, write for this particular table. Local read count and local write count give us the indication to know if we have read heavy or write heavy. For our example, we have write heavy (4352651 greater than 942517).

Pending tasks: 0 this is counting any tasks pending, we saw that earlier on the tpstats.

Bloom filter false positives, bloom filter false ratio, bloom filter space used, bytes, bloom filter off heap memory used,bytes: the main metric here is the false ratio (0.0000) that takes the number of bloom filter false positives (48566) divided by the number of local read count (942517) , that can give us the false ratio. What did we pay to get 48566 of bloom filter false positives to get 942517 local read count? We paid these memory ram bloom filter space used, bytes: 7088 plus bloom filter off heap memory used, bytes: 6984, total of these memories: 7088 + 6984 =14072 bytes approximatively: 14 KB of RAM to get false positives to avoid unnecessary disk seek. If it is unacceptable, we can tune it by paying more memory and to get lower false positives and consequently less fewer unnecessary less disk seek, so read performance will go UP.

Index summary off heap memory used, bytes: 390: Is a memory structure we use to help us to jump to correct place in the partition index in memory. In this case we paid 390 bytes of memory that is tunable also but in Cassandra 2.1 we cannot adjust it automatically.

Compression metadata off heap memory used, bytes: 1920728 bytes, it is the amount of ram consume for compression. We can look at also.

Compacted partition minimum bytes, compacted partition maximum bytes, compacted partition mean bytes: very useful metrics for getting indication for how we have setup our partition. What I mean by that ? if we look at compacted partition minimum bytes (35426 bytes = 35 KB) and compacted partition mean bytes (122344216 bytes =116.68 MB) and we look at the maximum bytes (557074610 bytes =531,28 MB), we have the idea of the distribution of the size of the partition. If we look at the number of means bytes (122344216 bytes =116.68 MB), there is a huge discrepancy between the min and the max, probably we have the good indication of the hotspot of the data , the partition is abnormally much larger than the other partitions.

Average live cells per slice, average tombstones per slice: these metrics are only useful on production.
Average live cells per slice (last five minutes): 2.0: it has been reading on average 2.0 and 0.0 for tombstones, that good. If it is for example 2.0 for tombstones, that means for every 2.0 last five minutes for read, probably it pick at 2.0 tombstones that gives 50% over read performance to tombstones. Unfortunately, this number of tombstones cannot be the same. That the good indication that we have exception large delete of data.
For all these metrics we have some tunable.

Notes:
The bloom_filter_fp_chance and read_repair_chance control two different things. Usually you would leave them set to their default values, which should work well for most typical use cases.
bloom_filter_fp_chance: controls the precision of the bloom filter data for SSTables stored on disk. The bloom filter is kept in memory and when you do a read, Cassandra will check the bloom filters to see which SSTables might have data for the key you are reading. A bloom filter will often give false positives and when you actually read the SSTable, it turns out that the key does not exist in the SSTable and reading it waste the time. The better the precision used for the bloom filter, the fewer false positives it will give (but the more memory it will need).
From the documentation:
0 Enables the unmodified, effectively the largest possible, Bloom filter
1.0 Disables the Bloom Filter
The recommended setting is 0.1. A higher value yields diminishing returns.
So a higher number gives a higher chance of a false positive (fp) when reading the bloom filter.
read_repair_chance: controls the probability that a read of a key will be checked against the other replicas for that key. This is useful if your system has frequent downtime of the nodes resulting in data getting out of sync. If you do a lot of reads, then the read repair will slowly bring the data back into sync as you do reads without having to run a full repair on the nodes. Higher settings will cause more background read repairs and consume more resources, but would sync the data more quickly as you do reads.
See documentation on this link: http://docs.datastax.com/en/cql/3.1/cql/cql_reference/tabProp.html
You can read also this document:
http://www.datastax.com/dev/blog/common-mistakes-and-misconceptions
Here is the description of that table:
\"desc-table1\"
Below we have tunable parameters for this table:
\"desc-table2\"
These parameters influence directly the metrics we saw earlier.
Example:
bloom_filter_fp_chance=0.010000, that impact how many false positives we got and how much memory were are going to give up to low or high.
caching=’KEYS_ONLY’: In this case it is key only, we just go to cache only key but we have the ability to cache rows cache , so we will increase the memory utilization of this table, that also have the big impact.
dclocal_read_repair_chance=0.100000 and read_repair_chance=0.00000: these parameters influence the no blocking shows in tpstats.
gc_grace_seconds=864000: it impacts the tombstones, how long we can hold on the tombstones but it impacts the last line we saw earlier (average tombstones per slice (last five minutes), so it is tunable.
index_interval=128: in Cassandra 2.1, a max and min interval, those will impact index summary off memory used, bytes saw earlier. That is tunable.
Populate_io_cache_on_flush=’false’: in Cassandra 2.0 it allows us to populate if we want to flush to disk and say.
MemTable_flush_period_in_ms=0 if we want to flush on the disk by scheduling, most people do that.
Compression= {‘sstable_compression’:’LZ4compressor’}: that can be impact the compression ratio we saw early. If we want, we can tune off the compression. Why do we change that? May be we have the high compression ratio probably trading off CPU cycle, we can change it.
Speculative_retry=’99.0PERCENTILE’: this is not impacting in the tpstats output. When we are performing a read, we have additional replicat could be used to feel the request, Cassandra will wait for this long 99.0 % , if it is the quorum read each replicat will go to ask others replicat to get the data , Cassandra will wait this long 99.0 PERCENTILE in terme in milliseconds.

NODETOOL CFHISTOGRAMS:

Let‘s go to node0 cfhistograms: in Cassandra 2.0, it is pretty long, it gives us very fine bucket. Also we have a new format for percentile sstables writes latency ,read latency , partition size , cell count
\"cfhistogram1\"
\"cfhistogram2\"
How do we read us?
sstables per read:
\"cfhistogram3\"
31261 read will be satisfy by only one sstable but we have 240663 read operations will be satisfy by adding another sstable, that two disk seeks so the performance is there, then when we go down here, we have 397454 read operations by adding another sstables, we have three sstables, that three disk seeks. When we go, the slow the performance get, we have the performance problem,we need to get inside to know how it is performing.
Generally if we have this problem, it usually the function of compaction that getting behind. Compaction is the low priority task, it contains disk io, we don’t do that when we have a huge activity on the cluster.
Consequence: we have to read on the sstables to satisfy more read.
We see here on the five sstables we have 272481 read operations to satisfy read that is the problem.
Next is the write latency:
\"cfhistogram4\"
\"cfhistogram5\"
We saw in cfstats, local read latency: 0.000 ms and local write latency: 0.000 ms.
In write latency, us is the microseconds, here we see 2415947 writes operations will be completed in 50 us (microseconds) , this is the buck, my write operations completed in 50 us, so we have a long tail , then when we come down on write latency, the max was 126934 us:1
For the read latency, 50 us (microseconds) of read operations has 12976, this is a buck for read operations .this is proof that with Cassandra write latency is much much lower than read latency.
\"cfhistogram6\"
Another trend is the notion of two bumps:
Here is the first bump:
\"cfhistogram7\"
Second bump:
\"cfhistogram8\"
This is an indication of read from ram for the first bump and read from disk for the second bump.
Partition size:
So we have fine great output for partition size, these data generally is representative of cfstats by the metrics follow (compacted partition minimum bytes, compacted partition maximum bytes and compacted partition mean bytes).
We can see here we have one partition with 42Kb and at the end my largest has 25109160 bytes (24MB). 0, 1, 2 mean we create the bucket.
\"cfhistogram9\"

Cell count per partition:
Sometimes people have large volume of data in each cell this definitively influence number of partition size but we have a large data and the small number of cells (0, 1, 2) . Combining information below with which we saw on the partition size, we can see how the data is lay out on the term of data model. That is looking at the table schema

\"cfhistogram10\"
So this line means 3 of my partition have 4866323 cells

\"cfhistogram11\"

Quiz:
Which of the following information does CQL display for a query trace? Elapsed time
Which of the following components would you not find statistics for in a nodetool cfstats output? row cache
nodetool cfhistograms shows per-keyspace statistics for read and write latency? false only for the table.

Environment Tuning:

Some we have to tune others components. For example:
The most common bottlenecks in Cassandra? Performance checkpoint, things that slowing down the performance of the system.
The most common bottlenecks are:
Inadequate hardware.
Poorly configured JVM parameters: Cassandra runs on top on this virtual machine.
High CPU utilization: Particularly for write workload which we can see the Cassandra CPU bond, CPU pick..
Insufficient or incorrect memory cache tuning: This referring for read. This point is very outside of Cassandra.
I like to drive this point.
Question: What is the best way to cope with inadequate node hardware in a Cassandra cluster?
Lot of people ask, we have a machine then we want the performance then what can we do?
Upgrade the hardware. This is the bad new.
If we don’t get adequate equipment for the job, it is not a good idea to do the job.
We could say to add another node if the existing machine not performing.

What are some of the technologies upon which a Cassandra node depends?
Java, JVM, JNA, JMX and a bunch of others stuff that starts with “j”

JVM:

Cassandra is the java program, we will know how the jvm works, so we can tune it.
When we can for tuning, JVM has a lot options.
If we run this command:

java -XX:+PrintFlagsFinal

We will see all of the options.
JVM and Garbage Collection (GC): what is Garbage collection? Java gives the developer to allocate and deallocate memory. Java can complain all of the resources on the machine, lot of ram, CPU, IO.
JVM generational Heap:
When we start the java process, jvm will allocate a big chunk of ram and that is the chunk will be managed can be half. Inside that chunk this is a heap, it divides in several pieces, we have the new gen, old gen and perm gen. after java 8, perm gen change. What is does? It stores all of classe definitions.
Important thing that Cassandra boot load all code in most part of perm gen which could not change in size.

\"jvm1\"
With Cassandra, we can do anything in perm gen.
But in New gen, we have Eden and survival spaces. So, when we create a new objects any kind of objects, that object will be created in Eden. Let say we created it inside the function then it will be created in the Eden and as the space of eden fills Up with objects allocations that can the garbage collection kik send , that the first garbage collector.
This piece (new gen) is called parallel new.
Important thing is if Eden fills up, we need to look for garbage, we will find and drop all objects which will not references and deallocate them. However, there will be others objects which are always references, they can be move to survival area in New gen.
And after the survival, this process could be promoted to Old gen.
There are lot of options to do how many time the objects can be on Eden or to be move to survival and also to Old gen. Lot of options.
This are (old gen), can be fill up with the options.
Take a look at this.
I have in my test cluster, 4 nodes working in different virtual machine.

nodetool status
\n[hduser@base cassandra]$ nodetool status
\nDatacenter: datacenter1
\n=======================
\nStatus=Up/Down
\n|/ State=Normal/Leaving/Joining/Moving
\n--  Address        Load       Tokens       Owns    Host ID                               Rack
\nUN  192.168.56.72  40.21 MB   256          ?       5ddb3532-70de-47b3-a9ca-9a8c9a70b186  rack1
\nUN  192.168.56.73  50.88 MB   256          ?       ea5286bb-5b69-4ccc-b22c-474981a1f789  rack1
\nUN  192.168.56.74  48.63 MB   256          ?       158812a5-8adb-4bfb-9a56-3ec235e76547  rack1
\nUN  192.168.56.71  48.52 MB   256          ?       a42d792b-1620-4f41-8662-8e44c73c38d4  rack1
\n

Now we can do the command:

cassandra-stress write -node 192.168.56.71

Result:

INFO 23:56:42 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter 
\nname with DCAwareRoundRobinPolicy constructor) INFO 23:56:42 New Cassandra host /192.168.56.71:9042 added INFO 23:56:42 New Cassandra 
\nhost /192.168.56.72:9042 added INFO 23:56:42 New Cassandra host /192.168.56.73:9042 added INFO 23:56:42 New Cassandra host /192.168.56.74:9042 
\nadded Connected to cluster: Training_Cluster Datatacenter: datacenter1; Host: /192.168.56.71; Rack: rack1 Datatacenter: datacenter1; 
\nHost: /192.168.56.72; Rack: rack1 Datatacenter: datacenter1; Host: /192.168.56.73; Rack: rack1 Datatacenter: datacenter1; Host: /192.168.56.74; 
\nRack: rack1 Created keyspaces. Sleeping 1s for propagation. Sleeping 2s... Warming up WRITE with 50000 iterations... Failed to connect over JMX; 
\nnot collecting these stats WARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level 
\nanalysis only Running with 4 threadCount Running WRITE with 4 threads until stderr of mean < 0.02 Failed to connect over JMX; not collecting 
\nthese stats type, 
\ntotal ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb 
\ntotal, 2086, 2086, 2086, 2086, 1,9, 1,5, 4,2, 7,0, 46,4, 58,0, 1,0, 0,00000, 0, 0, 0, 0, 0, 0 
\ntotal, 4122, 2029, 2029, 2029, 2,0, 1,6, 4,8, 8,0, 14,0, 15,3, 2,0, 0,02617, 0, 0, 0, 0, 0, 0 
\ntotal, 6171, 2029, 2029, 2029, 1,9, 1,5, 5,1, 7,6, 12,0, 13,6, 3,0, 0,02038, 0, 0, 0, 0, 0, 0 
\ntotal, 8466, 2288, 2288, 2288, 1,7, 1,4, 4,2, 6,1, 11,9, 14,4, 4,0, 0,02715, 0, 0, 0, 0, 0, 0

We can run the program called jvisualvm
If we have the JDK installed this java visual vm
\"java1\"

We can see the available plugins on the Tools windows and activate some plugins like VisualVM-Glassfish, visual GC:
\"java2\"
We can see the Eden space:
\"java3\"
\"java4\"
In the long time, Cassandra recommended the CMS collector. In java 7, G1 exist and in java 8 it is very good.so depending of the version of Cassandra which you are running, it will be CMS or G 1.
CMS and G1 have both old generation and the permanent generation.
\"java5\"
The difference is in G1 we have several contigus chunk of memory like this:
\"java6\"
\"java7\"
G1 is very very well with very large heap. We generally recommend 8 GB heap for Cassandra.
When the old gen fill up we can have the capacity and we do a merging garbage collection, this pause time could last for second. That bring the pausis. What is the notion of pausis ?when we are doing the garbage collection in CMS or G1, some part of have to do the stop the work pause, that where they stop our program running at all and in eden and survivor space to find any unnecessary objects so we can clean them up.
How long this pause is last? is the function of couple of different things
How many objects are still in live?
The number of CPU available to the jvm is also the big determine. How long the garbage collection pause there
Additionally CMS offers one of the other thing called the heap fragmentation. Any way CMS defragment those is to do that the full stop the wall pause by the serial collector which is another garbage collection but it is single threaded. That the extreme long pause come from.
For G1 the only option we have is the target pause time and the minimum is 12 hundred milliseconds.
We have a couple of tools available:

1. java visual vm

2. OpsCenter
\"java8\"

3. Jconsole and jvisualvm

\"java9\"
4. And the last is jstat

jstat -gccause 1607 5000 (1607 is the process id for Cassandra)

Notes:

The most significant impact for Java virtual machine on Cassandra performance is Garbage collection.
The G1 collector is the preferred choice for garbage collection over CMS.
Metaspace does NOT exist in the new generation part of the JVM heap memory.

JVM Tools and Tuning Strategies:

If we have the server with 126 GB of RAM, it starts Cassandra by allocated 8 GB heap, what are it doing of the rest of the memory?
Page cache.
What is page caching? This is useful for Cassandra improving read performance. it can cache data that people are accessing frequently and get it up quicker than in the disk.

[hduser@base ~]$ free -m
\n             total       used       free     shared    buffers     cached
\nMem:          5935       4271       1664         13        104       1189
\n-/+ buffers/cache:       2977       2957 
\nSwap:         2047          0       2047 
\n

Here we have 5935 MB of ram but 4271 MB are used, so only 1189 MB are cached.

How does Cassandra utilize page caching?

It also make a write efficient but it cannot improve a lot write.
How do you triage root cause for out of memory (OOM) errors? If we don’t have enough memory.
It can be java errors not only Cassandra.
All of the challenge can be to buffering data in case of writing. It tries to write data in disk

Quiz:

What is the benefit of using the page cache? Reads are more efficient, Writes are more efficient, and Repairs are more efficient
Memory used by the page cache will not be available to other programs until explicity freed. ? true
Which of the following is most likely to cause an out-of-memory error? Client-side joins
CPU
CPU intensive:
Writes (INSERT, UPDATE, DELETE), encryption, compression, Garbage collection is CPU-intensive.
The more CPU which you can give to garbage is the faster garbage collection will run.
If you want to activate the compression or the encryption, you have to monitor the CPU utilization with the tools like dstat or opscenter.
What to do ?
Add nodes
Use nodes that have more and faster CPUs
However, if you have saturation of the CPU, we have a couple of options:
1. Turn off encryption, turn off compression
2. Add nodes
3. Alternatively upgrade theses nodes with more CPU
Quiz:
Which of the following operations would significantly benefit from faster CPUs? Writes and Garbage collection
OpsCenter is a tool that can be used to monitor CPU usage ? true
What would be the best course of action to resolve issues with CPU saturation? Add more nodes.

Disk Tuning:

In this section we are going to talk about disk tuning and compaction.
Question: How do disk considerations affect performance?
When we operate in the database like Cassandra where we have active dataset that have a large available RAM, disk. So, we have to take these things into account.
SSD: spinning, or rotation, disks must move a read/write mechanical head to the portion of the disk that is being written to or read.
Cassandra is architecture around rotation drive with sequential write and sequential read. However, SSD is very faster. If you have a latency application, SSD is crutial.

Some of the tuning in the Cassandra.yaml file that affect disk are:
Configuring disks in the Cassandra.yaml file:
1. Disk_failure_policy: what should occur if a data disk fails? Not for performance.
• By default, we do stop which Cassandra can detect some kinds of corruption, shut down gossip and thrift.
• Best-effort -stop using failed disk and respond using remaining sstables on others disks -obsolete data can be returned if the consistency level is one
• Ignore -ignore fatal errors and let requests fail
2. Commit_failure_policy: – What should occur if a commit log disk fails?
• Stop -same as above
• Stop_commit – shutdown commit log, let writes collect but continue to serve reads
• Ignore -ignore fatal errors and let batches fail
3. Concurrent_reads -typically set to 16 * number of drives: how many threads we can allocate for reads pool
4. Trickle_fsync -good to enable on SSDs but very bad for rotation drive. In SSD, it will do big flash

Tools to diagnose relative disks issue:

Using Linux sysstat tools to discover disk statistics:
System activity reporter (sar) -can get information about system buffer activity, system calls, block device, overall paging, semaphore and memory allocation, and CPU utilization.
Flags identify the item to check:

sar  -d for disk 
\nsar  -r for memory 
\nsar  -S for swap space used 
\nsar  -b for overall I/O activities 
\ndstat  -a Leatherman tool for Linux -versatile replacement 
\nfor sysstat (vmstat, iostat, netstat, nfsstat, and ifstat) 
\nwith colors
\nFlags identify the item to check:                      
\ndstat      -d             for disk 
\ndstat      -m             for memory, etc
\n

Can also string flags to get multiple stats:


\ndstat    -dmnrs
\n

Step to install dstat on RHEL/CentOS 5.x/6.x and fedora 16/17/18/19/20:
Installing RPMForge Repository in RHEL/CentOS
For RHEL/CentOS 6.x 64 Bit

sudo rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
\nsudo yum repolist
\nsudo yum install dstat
\n

If we want to use linux script to monitor our statistics, we can use cron jobs like:

sudo dstat  -drsmn --output /var/log/dstat.txt 5 3 >/dev/null
\n
#!/bin/bash
\ndstat -lrvn 10  --output /tmp/dstat.csv -CDN 30 360 mutt -a /tmp/dstat.csv  -s \"dstat report\"  me@me.com >/dev/null

Output can be displayed on webpage for monitoring
Output could be piped into graphical programs, like Gnumeric, Gnuplot, and Excel for visual displays.

Another tool is: nodetool cfhistograms to discover disk issues:

For SSD disk io disk performing, this tool can tell us something. The problem could be the JVM, the size of the RAM, there are a lot of thing it could be but disk is one of the things it could be.
With cfhistograms, there are two groups of bumps. It could be 1 of three things, usually we see two bumps.
One relative to read coming from RAM and big one coming from disk. Sometimes, it could be anything else like a lot of compaction. Compaction can cause disk contention and this contention can cause the read disk goes UP. because the compaction uses the disk by reading.
So, how do we deep that? How to fix it?
We have the utility Throttle down compaction by reducing the compaction_throughput_mb_per_sec.
Using nodetool proxyhistograms to discover disk issues: will show the full request latency recorded by the coordinator.

Using CQL TRACING :
To distinguish between slow disk response and slow query: slow disk response will be evident in how long it takes to access each drive.
If your queries need to look for SSTables on too many partitions to complete, you will see this in the trace
These issues will have different patterns.
Here we can see lot off informations like looking on the tombstoned, etc. We can know if our latency on the query is coming from the disk.and it shows us the source on which machine is experiencing those longer latencies.
Where is tracing information stored?
events table gives us lot of relative details for this particular query.
What role does disk readahead play in performance? We read a head a couple of blocks and that tunable, how many blocks to read a head? The problem is in Cassandra we don’t know exactly how much data we want to read a head
We recommend people to use readahead value of 8 for SSDs
Command to do that is:

blockdev -setra 8  
\n

QUIZ:

nodetool cfhistograms shows the full request latency recorded by the coordinator.false
Which of the following statements is NOT true about the readahead setting?
Which of the following tools can be used to monitor disk statistics? iostat, dstat, sysstat, sar.

Disk Tuning:

Compaction

How does compaction impact performance?

Compaction is the most io intensive operation in Cassandra cluster. So, some of the choice we make around is our compaction strategies and how we throllet, some things happen on the disk, how the big impact on the disk utilization in our cluster.
ction Strategy DTCS  for the time series data model.
Adjust compaction is to look at the impact of min/max SSTable thresholds.
Understand the connecUnderstand compaction strategies: there are 3 currently (SizeTiered Compaction Strategy STCS, DateTiered Compaction Strategy DTCS, Leveled Compaction Strategy LTCS)
For write and intensive workload, SizeTiered Compaction Strategy STCS can be the best.
Leveled Compaction Strategy is generaly recommended for read and workload only if we are using SSDs.
DateTiered Compation between number of SSTables and compaction as it affects performance
See what options are available for compaction to improve performance
How do tombstones affect compaction?
Compaction evicts tombstones and removes deleted data while consolidating multiple SSTables into one. More tombstones means more time spent during compaction of SSTables.
Once a column is marked with a tombstone, it will continue to exist until compaction permanently deletes the cell. Note that if a node is down longer than gc_grace_seconds and brought back online, it can result in replication of deleted data -zombie!
To prevent issues, repair must be done on every node on a regular basis.

Best practices: nodes should be repaired every 7 days when gc_grace_seconds is 10 days (the default setting)

How data modelling affects tombstones?

If a data model requires a lot of of deletes within a partition of data then a lot of tombstones are created. Tombstones identify stale data awaiting deletion – which data will have to be read until it is removed by compaction.
More effective data modelling will alleviate this issue. Ensure that your data model is more likely to delete whole partitions, rather than columns from a partition.
The data model has a significant impact on performance. Careful data modelling will avoid the pitfalls of rampant tombstones that affect read performance.
Tombstones are normal writes but will not otherwise affect write performance.
If you know you do a lot of delete you discover to do a long delete which will affect read performance we probably need to do something to fix it, example changing a data model or the way our workload use this data model.

Using nodetool compactionstats to investigate issues:

This tool can be used to discover compaction statistics while compaction occurs. Reports how much still needs compacting and the total amount of data getting compacted.
But by using CQL tracing to investigate issues, we can see how many nodes and partitions are accessed.
The number of tombstones will be shown.
The read access time can be observed as decreasing after a compaction is complete. It can also be seen to take longer while a compaction is in progress.
Why is a durable queue an anti-pattern that can cause compaction issues?
A lot people for another reason like to use Cassandra for durable queue because of this problem of tombstones, this is a use anti pattern. Generally what happen, only reading and delete on the same place then read performance can grow up. If you try to corrige a queue use a queue, use something like KAFKA that the perfect durable queue. Cassandra is not a tool to use as a queue.

How do disk choices affect compactions issues?

Compaction is the most disk IO with intensive operation of Cassandra performance.so having a good disks has a good positive affect on it. Conversely, have a slow disks can have a very detrimental effect on it. When you have compaction going very slow, you can increase SSTable for read then read performance can suffer.
Use nodetool cfhistograms to look at the read performance.
QUIZ:
Which of the following compaction strategies should be used for read-heavy workloads, assuming certain hardware conditions are met? Leveled Compaction
What tool, command, or setting can be used to investigate issues with tombstones? CQL tracing
Compaction can potentially utilize not only a significant amount of disk I/O, but also disk space as well.true

Disk Tuning: Easy Wins and Conclusion:

To end this paper, we need to revisit:
• Performance tuning methodology
• Outline easy performance tuning wins
• Outline Cassandra or environment anti-patters

How does this all fit together?

1. We need to understand performance and Cassandra at the high level. That is general performance tuning techniques and some of the terminology as well as how Cassandra itself works.
2. Collect performance data on the following things to know where to look for data and what that data means in term of tuning or isolation problem:
• workload and data model
• cluster and nodes
• operating system and hardware
• disk and compaction strategies.
3. Parsing the information gathered and begin formulating a plan:
• Based on metrics collected, where are the bottlenecks?
• What tools are available to fix issues that come up?
4. Apply solutions to any/all areas required and test solutions:
• Using tools and knowledge gained, apply solutions, test solutions applied and start cycle again as needed.

Question: What was that performance tuning methodology again?

We have:
Active performance tuning – suspect there’s a problem?
• Determine if problem is in Cassandra, environment or both.
• Isolate problem using tool provided.
• Verify problems and test for reproductibility.
• Fix problems using tuning strategies provided.
• Test, test, and test again.
• Verify that your “fixes” did not introduce additional problems.
Passive performance tuning – regular system “sanity checks”
• Regularly monitor key health areas in Cassandra / environment using tools provided.
• Identify and tune for future growth/scalability.
• Apply tuning strategies as needed.
• Periodically apply the USE Method for system health check.

Easy Cassandra performance tuning wins:

• Increase flushwriters, if blocked: Flushing memTable to sstables, if we look in nodetool tpstats tool ,we will see flushwriters regularly getting blocked and we only have one flushwriters which is common on system which has only one disk, we can increase it to 2 that can resolve the problem.
• Decrease concurrent compactors: we see lot of people to set their concurrent compactors to high, we recommend at 2 watch for CPU saturation. And if saturated, we can drop it to 1 which will make compaction single thread that is by default.
• Increase concurrent reads and writes appropriately: Write is very affecting by CPU then read is very affecting by disk (the kind of disks, the number of disks available) so adjust concurrent reads and writes appropriately.
• Nudge Cassandra to leverage OS cache to read based workloads: nudge means more RAM so the more we can read in RAM, better will be the performance.
• In cloud environment, sometimes we need to increase phi_convict_threshold for cloud deployments or those with bad network connectivity.
• Increase compaction_throughput if disk I/O is available and compactions are falling behind: take the default for 15 MB/s, if we have lot of disk IO available, just increase this, then the compaction will complete quickly.
• Increase streaming_throughput to increase the pace of streaming:the default is 200MB/s when we bring the node online and the last repair but if we want to bring the node online faster, this is the parameter which we can increase.
• In terme of tuning data model: if we have a time series data modelling, we can learn a lot of by reading the link: http://planetcassandra.org/blog/getting-started-with-time-series-data-modeling/
• Avoid creating more than 500 tables in Cassandra: if it is empty, these tables take at least 1MB of space on heap.
• Keep wide rows under 100 MB or 100000 columns: we remember the in memory compaction by default has 15 MB, that the reason, it is a bad idea to have wide rows below 100 MB.
• Leverage wide rows instead of collections for high granularity items: sometimes, people have partition that contains lot of list, for that, it is recommended to use clustering column.
• Avoid data modelling hotspots by choosing a partition key that ensures read/write workload is spread across cluster: try to find the right partition key, not to large partition key not to small.
• Avoid tombstone build up by leveraging append only techniques.
You can read also these documents about tombstones:
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
http://www.jsravn.com/2015/05/13/cassandra-tombstones-collections.html
You can view this video about tombstones:
https://www.youtube.com/watch?v=olTsTxpBFqc&feature=youtu.be&t=270
• Use DESC sort to minimize impact of tombstones, if I do a descending sort by trying to get recent data.
You can read also this document:
http://www.sestevez.com/range-tombstones/
• Use inverted indexes to help where data duplication or nesting is not appropriate.
• Use DataStax drivers to ensure coordinator workload is spread evenly across cluster.
• Use the token Aware load balancing policy: allows you to go directly to the data of the data by avoiding to go to the coordinator.
• Use Prepared Statements (where appropriate), if you do a query a lot of time, you will see performance gain if you use that.
• Put OpsCenter’s database on a dedicated cluster.
• Size the cluster for peak anticipated workload: for example for load balancing or bench marking.
• Use a 10G network between nodes to avoid network bottlenecks.
• On JVM hand memery size, the number one is RAM and the CPU if we want Garbage Collection to run faster, so ensure there is adequate RAM to keep active data in memory.
• Understand how heap allocation affects performance.
• Look at how the key cache affects performance in memory.
• Understand bloom filters and their impact in memory: we can tune it on the table by the false positive bloom filter to eliminate unnecessary disk seek.
• Disable swap: swap can cause problem and it will be very very difficult to reproduce, we have to disable it by the command:

sudo swapoff –a

• Remove all swap files on /etc/fstab by:

sed -i 's/^\\(.*swap\\)/#\\1/'  /etc/fstab

• Look at the impact of memtables on performance: by default, memtables take records by heap then the more memtables we have, the more flushing them on the disk, then the more disk IO, so understanding, how many we have, how many they are flushing

What are some Cassandra/environment anti-patterns?

1. Network attached storage. Bottlenecks include: when you put Cassandra on the SAN, it like you are putting it on the top on storage system also Cassandra has sequential pattern of reads and writes. Advice is don’t use SAN plus it will be cheaper.
Use Cassandra on the SAN will increase network latency to all of operations.
• Router latency
• Network Interface Card (NIC)
• NIC in the NAS device
2. Shared network file systems.
3. Excessive heap space size: that can cause the JVM pause time very high because running it memory takes time.
• Can impair the JVM’s ability to perform fluid garbage collection.
4. Load balancers: don’t put load balancers between applications in Cassandra because Cassandra has the load balancer built into the drivers.
5. Queues and queue-like datasets: don’t use Cassandra like a queue.
• Deletes do not remove rows/columns immediately.
• Can cause overhead with RAM/disk because of tombstones.
• Can affect read performance if data not modelled well.

","id":"09f92fe1-9ee7-57a9-90d5-775990c3f999","title":"Cassandra Operations and Performance Tuning","origin_url":"https://dmngaya.wordpress.com/2015/10/10/cassandra-performance-and-tuning/","url":"https://dmngaya.wordpress.com/2015/10/10/cassandra-performance-and-tuning/","wallabag_created_at":"2019-12-02T18:42:19+00:00","published_at":"2015-10-10T07:44:25+00:00","published_by":"['']","reading_time":53,"domain_name":"dmngaya.wordpress.com","preview_picture":"https://i0.wp.com/dmngaya.wordpress.com/wp-content/uploads/2016/07/wallpaper-nature-hd-2.jpg?fit=1200%2C675&ssl=1","tags":["cassandra","troubleshooting and tuning","performance"],"description":"In this topic, i will  cover the basics of general Apache Cassandra performance tuning: when to do performance tuning, how to avoid and identify problems, and methodologies to improve.When do you need..."},{"content":"

You can enable row caching. It can avoid significant READ load on your disks. First, in cassandra.yaml, define the amount of heap space you want to dedicate to row caching. Then you can activate it per table to set how much data to fit per partition key in the row cache.

Of course, row cache is better suited with read intensive, size limited tables. Eg: on James, this perfectly fits our denormalized metadata tables.

War story: You need to reboot the node when enabling row-cache though row_cache_size_in_mb cassandra.yaml configuration file. nodetool will not be enough.

Once this (annoying) configuration parameter is enabled, you can use CQL per table to enable row cache:

Avoid reads on SSTables, and thus on disks can not be always avoided. In this case, we want to limit the amount of I/Os to the maximum.

Our problem is that writes to given key might be spread across several SSTables. For instance, this is the case for tables with many updates, deletes or with Clustering keys.

We can hopefully trade compaction time and I/Os against read efficiency. The idea is to switch compaction algorithm from Size Tiered Compaction Strategy to Levelled Compaction Strategy. Levelled compaction strategy will by its structure limit the number of SSTables a given key can belong to. Of course, read intensive tables, with updates, deletes or partition keys will benefit a lot from this change. Avoid it on immutable, not clustered tables, as you will not get collocation benefits, but will pay the costs of more expensive compactions.

Date Tiered Compaction Strategy stores data written within a certain period of time in the same SSTable. It’s very useful for time series of tables implying the use of TTL, where entries expire.

Updating the compaction strategy of your tables can be done without downtime, at the cost of running compactions. Warning: this might consume IO and memories, and thus decrease performances when the compaction is running.

You need to modify the CQL table declaration to change the compaction strategy:

use apache_james ;
ALTER TABLE modseq
WITH compaction = { ‘class’ : ‘LeveledCompactionStrategy’ };

For the changes to take effect, you need to compact the SSTables. To force this, you need to use nodetool:

nodetool compact keyspace table

Be careful not to ommit keyspace or table, if you do not want to trigger a global compaction…

For the following compaction on large tables, you can use:

nodetool compactionstats

The rule of thumb for compaction time estimate is, with our hardware (16GB, HDD), approximatively one hour per GB stored on the table.

Bloom filters

If you have a high false positive rate, then you might consider increasing the memory dedicated to bloom filters. Again, this parameter can be set per table. I will not detail it here as it was not a problem for us.

Compression

Compression leads to less data being read and less data being written. If I/O bound, this is a nice trade-off. It turns out default behaviour is to compress SSTables by chunks using LZ4. Of course, this can be tuned on the table level.

Commitlog

An obvious tip is to store your commitlog on a different disk than the SSTables. Thus I/Os are shared across disk.

As a conclusion, Cassandra offers a data model that can be optimised in details. It requires knowing well the way Cassandra works. It also demands knowing well your data model. But significant improvments can be gained.

By applying these configuration changes on the table level, we achieved a x3 reduction in read latencies on some of our read intensive tables. We have a 62% row cache hit rate. And our Cassandra seems now to handle read load better. This tuning session has been both instructive, and promising. We even now have identified new room for improvements, for example on blob storage. Note that with a bad schema, good performances can not be achieved.

You don’t want your RAM to end up like this!

Also please note, that as Cassandra is a JVM application, single node performance is also impacted by your Garbage Collection settings. We decided not to cover this aspect of Cassandra configuration, as we did not have enough free memory to switch to the G1 garbage collector, and we would have ended describing minor settings. You can read this blog post, wich covers the topic pretty well.

","id":"9a05fe53-d7a3-52f3-8936-ba3a25f01f25","title":"Tuning Cassandra performances","origin_url":"https://medium.com/linagora-engineering/tunning-cassandra-performances-7d8fa31627e3","url":"https://medium.com/linagora-engineering/tunning-cassandra-performances-7d8fa31627e3","wallabag_created_at":"2019-12-02T14:56:48+00:00","published_at":"2017-10-09T01:41:36+00:00","published_by":"['']","reading_time":3,"domain_name":"medium.com","preview_picture":"https://miro.medium.com/v2/resize:fit:1200/1*IHRDtx285TH0clzHAkQIzQ.jpeg","tags":["cassandra","troubleshooting and tuning","performance"],"description":"You can enable row caching. It can avoid significant READ load on your disks. First, in cassandra.yaml, define the amount of heap space you want to dedicate to row caching. Then you can activate it pe..."},{"content":"
$ bin/tlp-stress\nUsage: tlp-stress [options] [command] [command options]\n  Options:\n    --help, -h\n      Shows this help.\n      Default: false\n  Commands:\n    run      Run a tlp-stress profile\n      Usage: run [options]\n        Options:\n          --cl\n            Consistency level for reads/writes (Defaults to LOCAL_ONE).\n            Default: LOCAL_ONE\n            Possible Values: [ANY, ONE, TWO, THREE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM, SERIAL, LOCAL_SERIAL, LOCAL_ONE]\n          --compaction\n            Compaction option to use.  Double quotes will auto convert to\n            single for convenience.  A shorthand is also available: stcs, lcs,\n            twcs.  See the full documentation for all possibilities.\n            Default: <empty string>\n          --compression\n            Compression options\n            Default: <empty string>\n          --concurrency, -c\n            Concurrent queries allowed.  Increase for larger clusters.\n            Default: 100\n          --coordinatoronly, --co\n            Coordinator only made.  This will cause tlp-stress to round robin\n            between nodes without tokens.  Requires using -Djoin_ring=false in\n            cassandra-env.sh.  When using this option you must only provide a\n            coordinator to --host.\n            Default: false\n          --cql\n            Additional CQL to run after the schema is created.  Use for DDL\n            modifications such as creating indexes.\n            Default: []\n          --csv\n            Write metrics to this file in CSV format.\n            Default: <empty string>\n          --dc\n            The data center to which requests should be sent\n            Default: <empty string>\n          --drop\n            Drop the keyspace before starting.\n            Default: false\n          --duration, -d\n            Duration of the stress test.  Expressed in format 1d 3h 15m\n            Default: 0\n          --field.\n            Override a field's data generator.  Example usage:\n            --field.tablename.fieldname='book(100,200)'\n            Syntax: --field.key=value\n            Default: {}\n          -h, --help\n            Show this help\n          --host\n            Default: 127.0.0.1\n          --id\n            Identifier for this run, will be used in partition keys.  Make\n            unique for when starting concurrent runners.\n            Default: 001\n          --iterations, -i, -n\n            Number of operations to run.\n            Default: 0\n          --keycache\n            Key cache setting\n            Default: ALL\n          --keyspace\n            Keyspace to use\n            Default: tlp_stress\n          --paging\n            Override the driver's default page size.\n          --partitiongenerator, --pg\n            Method of generating partition keys.  Supports random, normal\n            (gaussian), and sequence.\n            Default: random\n          --partitions, -p\n            Max value of integer component of first partition key.\n            Default: 1000000\n          --password, -P\n            Default: cassandra\n          --populate\n            Pre-population the DB with N rows before starting load test.\n            Default: 0\n          --port\n            Override the cql port. Defaults to 9042.\n            Default: 9042\n          --prometheusport\n            Override the default prometheus port.\n            Default: 9500\n          --rate\n            Rate limiter, accepts human numbers. 0 = disabled\n            Default: 0\n          --readrate, --reads, -r\n            Read Rate, 0-1.  Workloads may have their own defaults.  Default\n            is dependent on workload.\n          --replication\n            Replication options\n            Default: {'class': 'SimpleStrategy', 'replication_factor':3 }\n          --rowcache\n            Row cache setting\n            Default: NONE\n          --ssl\n            Enable SSL\n            Default: false\n          --threads, -t\n            Threads to run\n            Default: 1\n          --ttl\n            Table level TTL, 0 to disable.\n            Default: 0\n          --username, -U\n            Default: cassandra\n          --workload., -w.\n            Override workload specific parameters.\n            Syntax: --workload.key=value\n            Default: {}\n    info      Get details of a specific workload.\n      Usage: info\n    list      List all workloads.\n      Usage: list\n    fields      null\n      Usage: fields
","id":"7c01b35f-b29c-5158-9c81-3d8c5d4df5f4","title":"tlp-stress","origin_url":"http://thelastpickle.com/tlp-stress/","url":"http://thelastpickle.com/tlp-stress/","wallabag_created_at":"2019-08-16T16:08:58+00:00","published_at":null,"published_by":null,"reading_time":2,"domain_name":"thelastpickle.com","preview_picture":null,"tags":["stress","performance","cassandra","cassandra.stress"],"description":"$ bin/tlp-stress\nUsage: tlp-stress [options] [command] [command options]\n Options:\n --help, -h\n Shows this help.\n Default: false\n Commands:\n run Run a tlp-stress profile\n U..."},{"content":"","id":"a12aecc1-b8d3-50a9-8725-d487a9d21f96","title":"10 Easy Ways to Tune Your Cassandra Cluster with John Haddad | DataStax Accelerate 2019","origin_url":"http://www.youtube.com/oembed?format=xml&url=https://www.youtube.com/watch?v=swL7bCnolkU","url":"http://www.youtube.com/oembed?format=xml&url=https://www.youtube.com/watch?v=swL7bCnolkU","wallabag_created_at":"2019-08-16T15:56:52+00:00","published_at":null,"published_by":null,"reading_time":null,"domain_name":"www.youtube.com","preview_picture":"https://i.ytimg.com/vi/swL7bCnolkU/maxresdefault.jpg","tags":["cassandra","performance","video"],"description":"[embedded content]"},{"content":"

This post is part 2 of a 3-part series about monitoring Apache Cassandra. Part 1 is about the key performance metrics available from Cassandra, and Part 3 details how to monitor Cassandra with Datadog.

If you’ve already read our guide to key Cassandra metrics, you’ve seen that Cassandra provides a vast array of metrics on performance and resource utilization, which are available in a number of different ways. This post covers several different options for collecting Cassandra metrics, depending on your needs.

Like Solr, Tomcat, and other Java applications, Cassandra exposes metrics on availability and performance via JMX (Java Management Extensions). Since version 1.1, Cassandra’s metrics have been based on Coda Hale’s popular Metrics library, for which there are numerous integrations with graphing and monitoring tools. There are at least three ways to view and monitor Cassandra metrics, from lightweight but limited utilities to full-featured, hosted services:

Nodetool is a command-line utility for managing and monitoring a Cassandra cluster. It can be used to manually trigger compactions, to flush data in memory to disk, or to set parameters such as cache size and compaction thresholds. It also has several commands that return simple node and cluster metrics that can provide a quick snapshot of your cluster’s health. Nodetool ships with Cassandra and appears in Cassandra’s bin directory.

Running bin/nodetool status from the directory where you installed Cassandra outputs an overview of the cluster, including the current load on each node and whether the individual nodes are up or down:

$ bin/nodetool status\nDatacenter: datacenter1\n=======================\nStatus=Up/Down\n|/ State=Normal/Leaving/Joining/Moving\n--  Address    Load       Owns    Host ID   Token                  Rack\nUN  127.0.0.1  14.76 MB   66.7%   9e524995  -9223372036854775808   rack1\nUN  127.0.0.1  14.03 MB   66.7%   12e12ead  -3074457345618258603   rack1\nUN  127.0.0.1  13.92 MB   66.7%   44387d08   3074457345618258602   rack1\n

nodetool info outputs slightly more detailed statistics for an individual node in the cluster, including uptime, load, key cache hit rate, and a total count of all exceptions. You can specify which node you’d like to inspect by using the --host argument with an IP address or hostname:

$ bin/nodetool --host 127.0.0.1 info \nID                     : 9aa4fe41-c9a8-43bb-990a-4a6192b3b46d\nGossip active          : true\nThrift active          : false\nNative Transport active: true\nLoad                   : 14.76 MB\nGeneration No          : 1449113333\nUptime (seconds)       : 527\nHeap Memory (MB)       : 158.50 / 495.00\nOff Heap Memory (MB)   : 0.07\nData Center            : datacenter1\nRack                   : rack1\nExceptions             : 0\nKey Cache              : entries 26, size 2.08 KB, capacity 24 MB, 87 hits, 122 requests, 0.713 recent hit rate, 14400 save period in seconds\nRow Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds\nCounter Cache          : entries 0, size 0 bytes, capacity 12 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds\nToken                  : -9223372036854775808\n

nodetool cfstats provides statistics on each keyspace and column family (akin to databases and database tables, respectively), including read latency, write latency, and total disk space used. By default nodetool prints statistics on all keyspaces and column families, but you can limit the query to a single keyspace by appending the name of the keyspace to the command:

$ bin/nodetool cfstats demo\nKeyspace: demo\n    Read Count: 4\n    Read Latency: 1.386 ms.\n    Write Count: 4\n    Write Latency: 0.71675 ms.\n    Pending Flushes: 0\n        Table: users\n        SSTable count: 3\n        Space used (live), bytes: 16178\n        Space used (total), bytes: 16261\n        ...\n        Local read count: 4\n        Local read latency: 1.153 ms\n        Local write count: 4\n        Local write latency: 0.224 ms\n        Pending flushes: 0\n        ...\n

nodetool compactionstats shows the compactions in progess as well as a count of pending compaction tasks.

$ bin/nodetool compactionstats\npending tasks: 5\n          compaction type        keyspace           table       completed           total      unit  progress\n               Compaction       Keyspace1       Standard1       282310680       302170540     bytes    93.43%\n               Compaction       Keyspace1       Standard1        58457931       307520780     bytes    19.01%\nActive compaction remaining time :   0h00m16s\n

nodetool gcstats returns statistics on garbage collections, including total number of collections and elapsed time (both the total and the max elapsed time). The counters are reset each time the command is issued, so the statistics correspond only to the interval between gcstats commands.

$ bin/nodetool gcstats\nInterval (ms)  Max GC Elapsed (ms)  Total GC Elapsed (ms)  Stdev GC Elapsed (ms)  GC Reclaimed (MB)  Collections  Direct Memory Bytes\n     73540574                   64                    595                      7         3467143560           83             67661338\n

nodetool tpstats provides usage statistics on Cassandra’s thread pool, including pending tasks as well as current and historical blocked tasks.

$ bin/nodetool tpstats\nPool Name                    Active   Pending      Completed   Blocked  All time blocked\nReadStage                         0         0          11801         0                 0\nMutationStage                     0         0         125405         0                 0\nCounterMutationStage              0         0              0         0                 0\nGossipStage                       0         0              0         0                 0\nRequestResponseStage              0         0              0         0                 0\nAntiEntropyStage                  0         0              0         0                 0\nMigrationStage                    0         0             10         0                 0\nMiscStage                         0         0              0         0                 0\nInternalResponseStage             0         0              0         0                 0\nReadRepairStage                   0         0              0         0                 0\n

Collecting metrics with JConsole

JConsole is a simple Java GUI that ships with the Java Development Kit (JDK). It provides an interface for exploring the full range of metrics Cassandra provides via JMX. If the JDK was installed to a directory in your system path, you can start JConsole simply by running:

jconsole\n

Otherwise it can be found in your_JDK_install_dir/bin

To pull up metrics in JConsole, you can select the relevant local process or monitor a remote process using the node’s IP address (Cassandra uses port 7199 for JMX by default):

\"cassandra

The MBeans tab brings up all the JMX paths available:

\"cassandra

Out of the box, org.apache.cassandra.metrics (based on the Metrics library) provides almost all of the metrics that you need to monitor a Cassandra cluster. (See the first footnote on the table below for exceptions.) Prior to Cassandra 2.2, many identical or similar metrics were also available via alternate JMX paths (org.apache.cassandra.db, org.apache.cassandra.internal, etc.), which, while still usable in some versions, reflect an older structure that has been deprecated. Below are modern JMX paths, which mirror the JConsole interface’s folder structure, for the key metrics described in this article:

Throughput (writes|reads)org.apache.cassandra.metrics:
type=ClientRequest,scope=(Write|Read),name=Latency
Attribute: OneMinuteRate
Latency (writes|reads)*org.apache.cassandra.metrics:
type=ClientRequest,scope=(Write|Read),name=TotalLatency
Attribute: Count

org.apache.cassandra.metrics:
type=ClientRequest,scope=(Write|Read),name=Latency
Attribute: Count

Key cache hit rate*org.apache.cassandra.metrics:
type=Cache,scope=KeyCache,name=Hits
Attribute: Count

org.apache.cassandra.metrics:
type=Cache,scope=KeyCache,name=Requests
Attribute: Count

Loadorg.apache.cassandra.metrics:
type=Storage,name=Load
Attribute: Count
Total disk space usedorg.apache.cassandra.metrics:
type=ColumnFamily,keyspace=(KeyspaceName),scope=(ColumnFamilyName),name=TotalDiskSpaceUsed
Attribute: Count
Completed compaction tasksorg.apache.cassandra.metrics:
type=Compaction,name=CompletedTasks
Attribute: Value
Pending compaction tasksorg.apache.cassandra.metrics:
type=Compaction,name=PendingTasks
Attribute: Value
ParNew garbage collections (count|time)java.lang:
type=GarbageCollector,name=ParNew
Attribute: (CollectionCount|CollectionTime)
CMS garbage collections (count|time)java.lang:
type=GarbageCollector,name=ConcurrentMarkSweep
Attribute: (CollectionCount|CollectionTime)
Exceptionsorg.apache.cassandra.metrics:
type=Storage,name=Exceptions
Attribute: Count
Timeout exceptions (writes|reads)org.apache.cassandra.metrics:
type=ClientRequest,scope=(Write|Read),name=Timeouts
Attribute: Count
Unavailable exceptions (writes|reads)org.apache.cassandra.metrics:
type=ClientRequest,scope=(Write|Read),name=Unavailables
Attribute: Count
Pending tasks (per stage)**org.apache.cassandra.metrics:
type=ThreadPools,path=request,scope=(CounterMutationStage|MutationStage|ReadRepairStage|ReadStage|RequestResponseStage), name=PendingTasks
Attribute: Value
Currently blocked tasks**org.apache.cassandra.metrics:
type=ThreadPools,path=request,scope=(CounterMutationStage|MutationStage|ReadRepairStage|ReadStage|RequestResponseStage), name=CurrentlyBlockedTasks
Attribute name: Count

* The metrics needed to monitor recent latency and key cache hit rate are available in JConsole, but must be calculated from two separate metrics. For read latency, to give an example, the relevant metrics are ReadTotalLatency (cumulative read latency total, in microseconds) and the “Count” attribute of ReadLatency (the number of read events). For two readings at times 0 and 1, the recent read latency would be calculated from the deltas of those two metrics:

(ReadTotalLatency1−ReadTotalLatency0)/(ReadLatency1−ReadLatency0)\n

** There are five different request stages in Cassandra, plus roughly a dozen internal stages, each with its own thread pool metrics.

Collecting metrics via JMX/Metrics integrations

Nodetool and JConsole are both lightweight and can provide metrics snapshots very quickly, but neither are well suited to the kinds of big-picture questions that arise in a production environment: What are the long-term trends for my metrics? Are there any large-scale patterns I should be aware of? Do changes in performance metrics tend to correlate with actions or events elsewhere in my environment?

To answer these kinds of questions, you need a more sophisticated monitoring system. The good news is, virtually every major monitoring service and tool supports Cassandra monitoring, whether via JMX plugins; via pluggable Metrics reporter libraries; or via connectors that write JMX metrics out to StatsD, Graphite, or other systems.

The configuration steps depend greatly on the particular monitoring tools you choose, but both JMX and Metrics expose Cassandra metrics using the taxonomy outlined in the table of JMX paths above.

Conclusion

In this post we have covered a few of the ways to access Cassandra metrics using simple, lightweight tools. For production-ready monitoring, you will likely want a more powerful monitoring system that ingests Cassandra metrics as well as key metrics from all the other technologies in your stack.

At Datadog, we have developed a Cassandra integration so that you can start collecting, graphing, and alerting on metrics from your cluster with a minimum of overhead. For more details, check out our guide to monitoring Cassandra metrics with Datadog, or get started right away with a free trial.

Source Markdown for this post is available on GitHub. Questions, corrections, additions, etc.? Please let us know.

","id":"758bc758-7efe-5f1c-9dc6-a58eea511aab","title":"How to collect Cassandra metrics","origin_url":"https://www.datadoghq.com/blog/how-to-collect-cassandra-metrics/","url":"https://www.datadoghq.com/blog/how-to-collect-cassandra-metrics/","wallabag_created_at":"2017-01-10T15:31:58+00:00","published_at":"2015-12-03T00:00:00+00:00","published_by":"['']","reading_time":8,"domain_name":"www.datadoghq.com","preview_picture":"https://web-assets.dd-static.net/42588/1776356135-how-to-collect-cassandra-metrics-cassandra-blog-2.png","tags":["cassandra","metrics","performance"],"description":"This post is part 2 of a 3-part series about monitoring Apache Cassandra. Part 1 is about the key performance metrics available from Cassandra, and Part 3 details how to monitor Cassandra with Datadog..."},{"content":"

Amy's Cassandra 2.1 Tuning Guide (2015)

I really appreciate all the folks who have told me that this guide helped them\nin some way. I'm happy to hear that.

I've pushed this small update to change my name from Albert to Amy and haven't\nchanged anything else at this point. I'm also leaving the URL the same because\nsome folks have it bookmarked and I don't mind my old name being around so long\nas folks call me Amy from now on :)

This guide is not intended to be complete and focuses on techniques I've used\nto track down performance issues on production clusters.

This version of the guide has not had a lot of peer review, so there may be some\nmistakes as well as things I'm just outright wrong about.

If you find any errors, please (really!) submit an issue at\nhttps://github.com/tobert/tobert.github.io.

Observation is a critical skill to develop as a systems administrator. A wide\nvariety of tools are available for observing systems in different ways. Many of\nthem use the same few system metrics in different ways to provide you with a\nview into your system. Understanding low-level metrics (e.g. /proc/vmstat)\nallows you to better reason about higher-level displays such as OpsCenter\ngraphs.

It's important to remember that most metrics we consume are some kind of\naggregate; computers today are so fast that precise bookkeeping is too expensive\nto do all of the time. The critical implication of this is that we have to read\nbetween the lines; averages lie and the larger the sample is, the larger the\nlie.

Monitoring systems are tempting choices for gathering performance metrics, but\nthey usually end up having to trade off resolution for economy of storage and\nrarely have a resolution higher than 1 sample/minute. Low-resolution metrics are\ncertainly useful for capacity planning, useless for performance tuning. One\nexception is collectd + related tooling configured at a 10s resolution by\ndefault. This is much better than average but not good enough for all\nsituations, but still check it out.

dstat

dstat is by far my favorite tool for observing performance metrics on a Linux\nmachine. It provides the features of many tools all in one with high-resolution\n(1s). My go-to command to start it is:

dstat -lrvn 10

\"image

That runs dstat with the load average (-l), disk IOPS (-r), vmstats (-v), and\nnetwork throughput (-n). The 10 tells dstat to keep updating the current line\nevery second and roll to a new line every 10 seconds. It can do fancier like\nreport per-disk metrics and per-network interface, not to mention all of the\nvarious plugins. Dig in. This the most useful stats tool on the Linux command\nline. If you ask me for help, be prepared to get screenshots of a few minutes'\nactivity like the one above.

Reading the Matrix: I leave dstat running inside GNU Screen (or tmux if you\nprefer) pretty much all the time I'' connected to a cluster. Even when there\naren't problems. Maybe especially then. While running benchmarks or production\nload, I'll flip through my screens (ctrl-a n) and glance at the dstat output.\nOnce it has been running for a little while, the whole terminal should be full.\nWhat I'm looking for is vertical consistency (or lack thereof) and outliers. On\na fully warmed-up system, memory should be around 95% in-use, with most of it in\nthe cache column. CPUs should be in use with no more than 1-2% of iowait and\n2-15% system time. The network throughput should mirror whatever the application\nis doing, so if it's cassandra-stress, it should be steady. If it's a Hadoop\njob writing in big batches, I'll expect big spikes. Think through the workload\nand learn what to expect and learn what is normal.

reading vmstats

I am a huge fan of dstat. Back in the Bad Old Days, I had to switch between 3-4\ntools to get an idea of how a system was doing. With dstat I can see all the\nkey metrics in one view and that is important. With a snapshot every 10\nseconds, a full-screen terminal can show me the last few minutes of data,\nmaking it easy to scan the columns visually for patterns and anomalies.\nThis is plain old pattern recognition most of the time; even without knowing\nwhat the stats mean, you should be able to correlate changes in system stats\nwith changes in the client workload. For practice, fire up dstat on an idle\nCassandra node, then fire up a simple cassandra-stress load. The change in\nmetrics over the first couple minutes of load are instructive. How long does\nit take to level out? Does it level out or are metrics swinging wildly?\nOnce the patterns are identified, then it's time to understand whats happening\nunder the hood. Going from left to right of my usual dstat -vn 10:

run/blocked/new tasks

These show how many processes or threads were running/blocked/created\nduring the sample window. run=~n_cores is ideal for ROI/efficiency,\nbut makes a lot of admins nervous. run=2*n_cores isn't necessarily bad.\nrun>cores is healthy with some head room. Any blocked processes is\nconsidered bad and you should immediately look at the iowait %.\n1-2% iowait isn't necessarily a problem, but it usually points at\nstorage as a bottleneck.

memory

Memory is perhaps the easiest. Used will usually be your heap +\noffheap + ~500MB. If it's significantly higher, find out where that memory\nwent! Buffers should be a couple hundred MB, rarely more than a gigabyte.\nCache should account for almost all the remaining memory if data > RAM.\nFree memory should be in the 200-300MB range unless the working data size is\nsmaller than installed memory.

swap

dstat shows the swap columns along with memory. These should always be zeroes.\nAny swap activity whatsoever is a source of hiccups and must be eliminated\nbefore anything else.

disk throughput

This tells you how many bytes are going in and out of the storage every second.\nIt is fairly accurate and tends to be more useful than IOPS. There isn't a\nstandard range here. Try saturation testing the cluster with large objects to\nfind out what the high end is, then try to size the cluster/tuning to keep it\n25% or so below peak load, or more if there is a tight SLA. You will often see\nactivity on the disks after a load test has completed and that's most likely\ncompaction.

interrupts (int) & context switches (ctx)

An interrupt occurs when a device needs the CPU to do something, such as pick\nup data from the network or disks. Context switches occur when the kernel has\nto switch out a task on the CPU. Most interrupts are tied to a ctx, which is\nwhy ctx is almost always > interrupts. On a healthy system, ctx should be\n20-40% higher than interrupts. If it's a lot higher, take a look at system\ncall counts with strace. futex is almost always the top call and indicates\na high amount of lock contention.

user/system/idle/wait/hiq/siq CPU time

User time is CPU time used to run userland processes, i.e. JVM threads.\nSystem time is time spent in kernel code. Idle time is exactly as it sounds.\nhiq/siq are for time spent processing hardware and soft interrupts. These\nare usually zeroes but you'll occasionally see siq time under heavy load.\nA few % is fine. Any more indicates a serious problem with the kernel\nand/or hardware.

network send/recv

Shows the amount of data flowing over the network.\nI can saturate 10gig links with Cassandra 2.1 and large objects, so\nthis is more important to look at than it has been in the past. This shows\nhow much data is flowing over CQL and storage protocols.\nThe differential between network / disk io / CPU usage gives a good picture\nof how efficiently a system is running. I do this in my head….

sjk-plus

https://github.com/aragozin/jvm-tools\nis the newest addition to the stable and has one tool in particular that is\nuseful for DSE: ttop a.k.a. \"thread top\", which is exactly what it sounds like.\nWhile top and htop are useful for looking at processes, they cannot tell which JVM\nthread is which and that's where ttop comes in. It's a single jar so it's easy to\npush to a machine and does not require any GUI so it works fine over ssh. So far,\nI've found the following command to be the most useful:

\"image

Which threads trend towards the top is workload-dependent. Most threads in DSE are\nnamed except for all the stuff in the shared pool, which is a mess of things\nunfortunately. On the upside, in my observations of clusters under load they are\nnot a common source of problems.

The real killer feature is the heap allocation rates, which are directly correlatable\nto GC work. Try setting -o ALLOC instead of CPU to see which threads are putting\npressure on GC.

htop

Most Linux users are familiar with the 'top' command but it's fairly limited in\nwhat it can display and doesn't look as nice. htop has a nice default display\nthat shows per-core load and all the threads. It can also be configured by hitting\nF2, which is occasionally handy when you want to sort by specific fields or\ndisplay something outside the defaults, e.g. soft page faults.

In the following screenshot, system time is red, nice time is blue, and user\ntime is green. This may not be consistent across terminals or themes. To see\nthe legend, hit the 'h' key.

@phact recommends enabling \"[x] Detailed CPU time\"\nunder Setup (F2) to make things like iowait and steal time visible.\nWithout the option, they appear to be unused capacity, which is misleading.

\"image

powertop and i7z

Powertop is not often useful, but is worth checking at least once if you're\nseeing odd fluctuations in performance on bare metal. What you're looking for is\nthe second tab, so start it up with 'sudo powertop' then hit tab once to get to\nthe \"Idle stats\" tab. This will show you the \"C-states\" of the processors and\nhow much time is being spent in them. In short, the higher the C-state number,\nthe higher the latency cost to come out of it. C1E-HSW is cheaper than C3-HSW\nand so on. A lot of articles about latency tuning recommend disabling C-states\naltogether, but I don't think this is responsible or necessary. You probably\nwant to disable the deeper C-states (>3) for Cassandra nodes that will be\nbusy. The power management code in the kernel should handle the rest.

The next tab is \"Frequency stats\". This will show you if frequency scaling is\nenabled on the CPU. In general, frequency scaling should never be enabled on\nCassandra servers. Even if it \"works\" (and it does), it makes the TSC clock\n(the fastest one) unstable, causing more clock drift, which is bad for\nCassandra systems. Modern Xeon and AMD processors have specializations for\nmaking the TSC more stable, so it's worth measuring before you give up on it.

The next couple tabs aren't very useful for Cassandra work, but the last\none is in a non-intuitive way. You want all of the items in the \"Tunables\"\ntab to say \"Bad!\" Power management and high performance are almost always\nat odds. You can safely ignore devices that aren't in use by Cassandra,\ne.g. USB controllers and sound devices.

\"image

i7z is an alternative to powertop that was brought to my attention but\nI have not tried it. https://code.google.com/p/i7z/

/proc

Most of the stats displayed by the tools already discussed come from text files\nin /proc. dstat reads /proc/loadavg for the load average, /proc/stat for the\nVM stats, /proc/diskstats for disk IO, and so on. A few files are really handy\nfor quickly checking things that aren't exported by the usual tools.

For example, hugepages. In my experience, transparent hugepages aren't\n(usually) a problem for Cassandra like they are for many other databases.\nThat said, you need to be able to know if your JVM is using them or not,\nso the easy thing to do is:

\"image

If the AnonHugePages is slightly larger than your heap, you're all set with THP.\nIf you thought you disabled it, restart Cassandra again to get back to 4K pages.

Pro tip: you can make a top-like view of /proc files using the watch(1) command, e.g.

watch grep Dirty /proc/meminfo\n

/proc/interrupts is useful for figuring out which CPUs are handling IO. It is often\ntoo large to display on a screen, so a little awk or scripting may be in order to\nget it down to size. For quick checks, simply cat'ing the file will do.

dmesg

dumps the kernel's error buffer to stdout. Sometimes you find things really quickly,\nsometimes not. I almost always look at it on both healthy and problematic systems to\nsee if anything is going on. For example, if an application has been OOM killed,\nthere will be a detailed report in the dmesg log. The error buffer is a\nstatically-sized ring, so sometimes when things are really hairy the important\ninformation will get scrolled off, forcing you to grep around in /var/log.

strace

a.k.a. the life saver. Strace has helped me discover obscure failures more than\nany other tool in my toolbox. The simplest usage of strace involves printing\nout every system call made by a process. Idle Cassandra systems make a huge\nnumber of system calls, so filtering with the -e flag is highly recommended.\nIt can fork a process for tracing by prefixing a command (e.g. strace ls) or\nyou can attach to a running process with -p PID. The -f flag is required to\nget useful dumps from Cassandra - it makes strace attach to all tasks (threads)\nin a given process.

In order to trace a particular subsystem, e.g. networking, use the -e flag.\nThere are some preset groups of syscalls like \"network\", “file”, and o\nhers. Check out the man page.

\"image

One of the more useful ways to use strace with Cassandra is to see how often\nthe futex() syscall is being used. This is interesting because the syscall\nis only called on contended locks. For that, use\nstrace -f -c -e futex -p $PID. Let it run for a few seconds then hit Ctrl-C.

\"image

smartctl

Most hard drives made in the last decade support a protocol called\nSMART. SATA and SAS SSDs support\nit as well and it's the first place I look when I suspect a problem with\nrive. It isn't always installed by default. The package is usually called\n\"smartmontools\".

If the system has anything but a simple JBOD controller (e.g. SATA AHCI,\nLSI SAS x008), you may need to specify the device type so smartctl can\nquery the HBA properly. The man page has instructions for doing so.

The most important section for troubleshooting is the attributes, specifically\nthe RAW_VALUE column. The other columns often have unreliable values,\nespecially the TYPE column, which always looks scarier than things really are.

Here is a screenshot from \"smartctl -A /dev/sdc\" on a Samsung 840 Pro SSD.\nThis is where you look to find out if flash cells are dying or you suspect\nother kinds of errors.

\"image

The full smartctl -a output for an HDD and SSD are available as a gist:\nhttps://gist.github.com/tobert/c3f8ca20ea3da623d143

iperf

is a network load testing tool that is quick & easy to use. I use it to\nfind the real capability of a network interface, which is often surprisingly\nlower than advertised. On bare metal, you should expect to be able to get\n> 900mbit/s (remember to divide by 8 for bytes!) on 1gig links. For\n10gig links, you may need to use the --parallel option.

These screenshots show how simple it is to use iperf to find out how much\nbandwidth I can get between my Linux workstation and my Mac over wifi.

\"image

\"image

There are a number of switches available, but most of the time you don't need them.\nThe most important two to add are --time and --num to set the amount of time to\nrun the test or the number of bytes to transmit, respectively.

pcstat

https://github.com/tobert/pcstat is a tool I\nwrote out of frustration. One question we often have is whether or not a given\nfile is being cached by Linux. Linux itself doesn't export this information in\nan easy way, so pcstat gets it via the mincore(2) system call. The easiest way\nto get it if you have Go installed is \"go get github.com/tobert/pcstat\". This\nwill place a pcstat binary in $GOPATH/bin that you can scp to any Linux server.

pcstat -bname /var/lib/cassandra/data/*/*/*-Data.db\n

\"image

cl-netstat.pl

https://github.com/tobert/perl-ssh-tools\nAnother custom tool. This one logs into your whole cluster over ssh and displays\na cluster-wide view of network traffic, disk IOPS, and load averages. I use this\nall the time. It's a bit of a pain to install, but it's worth it. By default it\nupdates every 2 seconds.

\"image

We really need an entire guide like this one for cassandra-stress. For most\nperformance tuning, a very simple cassandra-stress configuration is sufficient\nfor identifying bottlenecks by pushing a saturation load at the cluster. It is\nimportant to keep in mind that sustained saturation load should never be used to\ndetermine production throughput; by definition it is unsustainable. It is, on\noccasion, useful to find the max saturation load, then dial it back by 10-20% as\na starting point for finding a cluster's maximum sustainable load.

There are two workloads in particular that expose most issues quickly, one using\nsmall objects and the other using large objects.

small objects for finding TXN/s limits

Small objects running at maximum transaction throughput help expose issues with\nnetwork packets-per-second limits, CPU throughput, and as always, GC.\nI've been unable to get enough throughput with small objects to stress 10gig\nnetworks or good SSDs; for those move on to large objects. This is the easiest\ntest to run and is by far the most common, since it requires almost no\nconfiguration of cassandra-stress.

I usually put my cassandra-stress commands in little shell scripts so it's\neasier to edit and I don't have to rely on command history or, horror of\nhorrors, typing. For example, here's what my small-stress.sh looks like:

#!/bin/bash\nexport PATH=/opt/cassandra/bin:/opt/cassandra/tools/bin:$PATH\ncassandra-stress \\\n    write \\\n    n=1000000 \\\n    cl=LOCAL_QUORUM \\\n    -rate threads=500 \\\n    -schema \"replication(factor=3)\" \\\n    -node 192.168.10.12\n

large objects for finding MB/s limits

Many of the times I'' asked to look at a cluster, IO is usually suspect. Small\nobjects tend to run up against CPU or GC bottlenecks before they have a chance\nto do enough IO to show whether I'm tuning in the right direction. Using large\nobjects helps here.

In this example, I'm writing partitions with 32 columns at 2K each for a total\nof 64K per partition. You will probably need to try some different values to get\nthings moving. Sometimes disabling durability on the CF or putting the CL on\ntmpfs is useful to reduce CPU/GC load and move more IO through flushing.

cassandra-stress \\\n    write \\\n    n=1000000 \\\n    cl=LOCAL_QUORUM \\\n    -rate threads=500 \\\n    -col \"size=fixed(2048)\" \"n=fixed(32)\" \\\n    -schema \"replication(factor=3)\" \\\n    -node 192.168.10.12\n

There are three major places to find settings that impact Cassandra's\nperformance: the java command-line (GC, etc.), the schema, and cassandra.yaml.\nProbably in that order. The inaccuracy of some comments in Cassandra configs is\nan old tradition, dating back to 2010 or 2011. The infamous \"100mb per core\"\natrocity dates back a ways, but we're not here to talk about history. What you\nneed to know is that a lot of the advice in the config commentary is misleading.\nWhenever it says “number of cores” or “number of disks” is a good time to be\nsuspicious. I'm n't going to rewrite the whole yaml file here, but instead cover\nthe few settings that should always be checked when tuning.

The Commitlog Bug in 2.1

a.k.a. https://issues.apache.org/jira/browse/CASSANDRA-8729

^ UPGRADE TO >= Cassandra 2.1.9 or DSE 4.7.3 and set:

commitlog_segment_recycling: false\n

Prior to 2.1.9:

The Jira linked above has most of the gritty details. TL;DR, the workaround is\nto set commitlog_segment_size_in_mb > commitlog_total_space_in_mb which\ncauses Cassandra to drop segments after use rather than reuse them.\n'

memtables & flushing

One of the most common tweaks we have to make is bumping\nmemtable_flush_writers on systems with sufficient resources. The advice in the\nstock yaml isn't bad in this case but it isn't very nuanced. Generally you can\nstart at 4 and see what happens. If you're seeing dropped mutations under\nsaturation load, try going as high as 8 but probably not much higher since 8\nblocked flushwriters probably means your disks aren't up to the task and the\nload needs to be scaled back.

I think memtable_flush_writers and memtable_cleanup_threshold should always\nbe set together. The default formula is: 1 / (memtable_flush_writers + 1) so\nif you have a lot of flush writers, your cleanup threshold is going to be very\nlow and cause frequent flushing for no good reason. A safe starting value for a\ncluster with few tables is 0.15 (15% of memtable space). If you have lots of\nactive tables, a smaller value may be better, but watch out for compaction cost.

Note: memtables aren't compressed so don't expect compressed sstable sizes to\nline up.

memtable_allocation_type: offheap_objects

Offheap memtables can improve write-heavy workloads by reducing the amount of\ndata stored on the Java heap. 1-2GB of offheap memory should be sufficient for\nmost workloads. The memtable size whould be left at 25% of the heap as well,\nsince it is still in use when offheap is in play.

An additional performance boost can be realized by installing and enabling\njemalloc. On common distros this is usually a yum/apt-get install away. Worst\ncase you can simply install the .so file in /usr/local/lib or similar.\nInstructions for configuring it are in cassandra-env.sh.

concurrent_{reads,writes,counters}

Like some other options, the recommendations in the comments for these are\nmisleading. Increasing these values allows Cassandra to issue more IO in\nparallel, which is currently the best way to push the Linux IO stack. You may\nalso want to take a look at /sys/block//queue/nr_requests. Start with 128\non huge machines and go as low as 32 for smaller machines.

sstable compression

The CPU tradeoff is almost always a net win compared to iowait or the wasted\nspace on the drives. Turning compression off is sometimes faster. The main\ntunable is chunk_length_kb in the compression properties of a table. The\ndefault chunk length is 128K which may be lowered either at CREATE TABLE time or\nwith ALTER TABLE.

In 2.1, uncompressed tables will use a reader with 64k buffer. If your\nreads are significantly smaller than 64k, using compression to allow Cassandra\nto lower that buffer size will likely be a significant win for you in terms of\nIO wait, latency, and overall read throughput, even if you don't necessarily care\nabout the savings in disk space. In any case, aligning the buffer chunk size to\nbe a multiple of the disk block size (ie: xfs 4k blocks) is optimal.

streaming

Make sure to always set streaming_socket_timeout_in_ms to\na non-zero value. 1 hour is a conservative choice that will prevent the worst\nbehavior.

https://issues.apache.org/jira/browse/CASSANDRA-8611

Java 8

Every cluster I've touched for the last couple months has been running Java 8\nwith no problem. JDK8 has some nice improvements in performance across the\nboard, so some clusters will pick up some additional headroom just by upgrading.\nJava 7 is deprecated by Oracle and is on its way out. That said, if for some\nreason (e.g. inverse-conservative policies) you have to stick with JRE7, at\nleast try to get the last update release.

OpenJDK

There is some remembered pain around OpenJDK that as far as I can tell dates\nback to the initial releases of it in Fedora where it was JDK6 with no JIT and a\nbunch of GNU Classpath and Apache Harmony things bolted on. That didn't last too\nlong before Sun finished OSSing the important parts of Hotspot, making the\nOpenJDK we have today that is, from the server VM's perspective, identical to\nthe Oracle releases of Hotspot. The critical thing to watch out for today with\nOpenJDK is OS packages. Java 8 took a long time to adopt because the early\nreleases were buggy. Do yourself a favor and check the full version of any\nOpenJDK before trusting it. If it's out of date and you still want to use\nOpenJDK, check out the Zulu packages\nproduced by Azul.

heap estimation & GC selection

If you haven't read the bit about offheap from above, please check that out. In\ngeneral, our default heap size of 8GB is a good starting point for\nCassandra-only workloads. When adding in Solr, you will almost always want to\nincrease the heap. I've seen it set anywhere from 8GB to 32GB in production\nsystems. I've tested up to 256GB with G1, and while it works great, it's a waste\nof pressious page cache space. G1 works best at 26-32GB, start in that range if\nyou have the spare RAM. When messing around with GC, you should always enable GC\nlogging. With newer JDK7 builds and any JDK8 build, GC log rotation is built-in\nso there's no reason to have it disabled, especially in production. Some folks\nprefer to leave it off in production by default because\nGC log writes occur during STW\nand are synchronous, which may introduce additional hiccups.

Now that you have GC logging enabled you have a choice: stick with CMS (the\ndevil we know) or switch to G1GC (the devil we barely know). You can get more\nthroughput from CMS, but it takes careful planning and testing to achieve it,\nwhereas with G1, you can get good performance without a lot of tweaking.

useful settings for any (parallel) GC

By default, Hotspot caps GC threads at 8, seemingly because of some legacy\nassumptions combined with unrelated side-effects on SPARC. In any case, if the\ntarget system has more than 8 cores, you may want to allow GC to use all of\nthem. This has been observed to reduce STW durations. I haven't seen any\nnegative side-effects. As the comments say, HT cores don't count. See\nalso: \"EC2 cores are not what you think they are\".

# The JVM maximum is 8 PGC threads and 1/4 of that for ConcGC.\n# Machines with > 10 cores may need additional threads.\n# Increase to <= full cores (do not count HT cores).\n#JVM_OPTS=\"$JVM_OPTS -XX:ParallelGCThreads=16\"\n#JVM_OPTS=\"$JVM_OPTS -XX:ConcGCThreads=16\"\n

Reference processing isn't usually a big deal for Cassandra, but in some\nworkloads it does start to show up in the GC logs. Since we pretty much always\nwant all the parallel stuff offered by the JVM, go ahead and enable parallel\nreference processing to bring down your p99.9's.

# Do reference processing in parallel GC.\nJVM_OPTS=\"$JVM_OPTS -XX:+ParallelRefProcEnabled\"\n

CMS

The 100mb/core commentary in cassandra-env.sh for setting HEAP_NEWSIZE is\nwrong. A useful starting point for CMS is 25% of the heap. For some\nworkloads it may go as high as 50% of the heap, but start at 20-25% and see what\nhappens. Before moving onto testing, add the following settings derived from\nCASSANDRA-8150. There's\nsome good stuff in there, but be careful not to haphazardly combine all the\nsettings from the various comments. They don't always mix well.

TODO: add Pierre's notes about CMS

# [http://blog.ragozin.info/2012/03/secret-hotspot-option-improving-gc.html](http://blog.ragozin.info/2012/03/secret-hotspot-option-improving-gc.html)\nJVM_OPTS=\"$JVM_OPTS -XX:+UnlockDiagnosticVMOptions\"\nJVM_OPTS=\"$JVM_OPTS -XX:ParGCCardsPerStrideChunk=4096\"\n# these will need to be adjusted to the workload; start here\nJVM_OPTS=\"$JVM_OPTS -XX:SurvivorRatio=2\"\nJVM_OPTS=\"$JVM_OPTS -XX:MaxTenuringThreshold=16\"\n# Branson thinks these are cool. TODO: describe what these do.\nJVM_OPTS=\"$JVM_OPTS -XX:+CMSScavengeBeforeRemark\"\nJVM_OPTS=\"$JVM_OPTS -XX:CMSMaxAbortablePrecleanTime=60000\"\nJVM_OPTS=\"$JVM_OPTS -XX:CMSWaitDuration=30000\"\n

CMSScavengeBeforeRemark: triggers a Young GC (STW) before running CMS Remark\n(STW) phase. The expected effect is to reduce the duration of the Remark phase.

CMSWaitDuration: once CMS detects it should start a new cycle, it will wait up\nto that duration (in millis) for a Young GC cycle to happen. The expected effect\nis to reduce the duration of the Initial-Mark (STW) CMS phase.

SurvivorRatio sizes the Eden and the survivors. SurvivorRatio=N means: divide\nthe young generation by N+2 segments, take N segments for Eden and 1 segment for\neach survivor.

MaxTenuringThreshold defines how many young GC an object should survive before\nbeing pushed in the old generation. A too high value increases the Young GC\npause time (because of extra copying). A too low value increases pressure on\nCMS.

G1GC

Recommendation: pick a node, flip its cassandra-env.sh over to the config block\nbelow, then come back here and read while that node generates some GC logs for\nyou to look at.

G1GC is the newest garbage collection backend in Hotspot. It was introduced\nduring Java 6 and gradually improved through Java 7 and seems to be solid for\nproduction as of Java 8u45. I do not recommend using G1 on any JRE older than\nHotspot 8u40. There have been changes to the algorithm between u40 and u45 and I\nexpect more as adoption increases, so the latest release of Java 8 is\nrecommended. Hotspot 9 is expected default to G1GC, so it's time to start\nlearning how it works.

The main benefit of the G1 collector is what they call ergonomics.\nTuning CMS is a black art that requires\na lot of iteration to dial in. G1 is usually good out of the box and can be\ndialed in with just a couple parameters to the JVM. According to the various\ndocs on the web, CMS shoots for 1% or less of application CPU time spent in GC\nwhile G1 allows for up to 10%. It's a good deal at half the price, and the observed\nCPU usage is usually much lower than 10%.

It is critical that you comment out the -Xmn line when switching to G1\nPerhaps my favorite feature in G1 is that the eden size is calculated\nautomatically to try to meet the latency target. It also automatically sets the\ntenuring threashold. It does a decent job and may even be useful to try as a way\nto estimate the -Xmn for CMS (switch to G1, run a load, grep Eden\n/var/log/cassandra/gc.log.0.current).

There are two main settings to use when tuning G1: heap size and MaxGCPauseMillis.

G1 can scale to over 256GB of RAM and down to 1GB (6GB minimum is recommended)\nThe first thing to try in many situations is to bump the heap by a few\nGB and see what happens. Sometimes it helps, sometimes it doesn't. The results\nare usually obvious within a few minutes, so I'll often go from 8GB to 16GB to\n32GB (when available) to see if it helps. Adding more heap space allows G1 to\n\"waste\" more heap on uncollected garbage that may be mixed with tenured data. If\nit is under pressure to reclaim space for eden, you will see significant memory\ncopy time in the GC logs. That's bound by the memory bandwidth of the system and\nthere isn't much we can do about it, so increasing the heap to allow more slack\nis the easy path.

The other tunable is -XX:MaxGCPauseMillis=n. The default in Hotspot 8 is 200ms.\nWhen testing G1 on\nlower-end hardware (mobile CPUs, EC2) it was observed that throughput suffered\ntoo much with the 200ms pause target. Increasing it to 500ms keeps the average\nSTW pause below the default timeouts in cassandra.yaml while allowing for better\nthroughput. The critical thing to keep in mind is that this is a target and\nnothing is guaranteed; STW on fast machines might hover around 120ms and never\neven approach the target. Slower machines may exceed the target occasionally,\nwhich is why your timeouts in cassandra.yaml should allow for some slack.

# Use the Hotspot garbage-first collector.\nJVM_OPTS=\"$JVM_OPTS -XX:+UseG1GC\"\n# Main G1GC tunable: lowering the pause target will lower throughput and vise versa.\n# 200ms is the JVM default and lowest viable setting\n# 1000ms increases throughput. Keep it smaller than the timeouts in cassandra.yaml.\nJVM_OPTS=\"$JVM_OPTS -XX:MaxGCPauseMillis=500\"\n# Have the JVM do less remembered set work during STW, instead\n# preferring concurrent GC. Reduces p99.9 latency.\nJVM_OPTS=\"$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=5\"\n# Start GC earlier to avoid STW.\n# The default in Hotspot 8u40 is 40%.\nJVM_OPTS=\"$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25\"\n# For workloads that do large allocations, increasing the region\n# size may make things more efficient. Otherwise, let the JVM\n# set this automatically.\n#JVM_OPTS=\"$JVM_OPTS -XX:G1HeapRegionSize=32m\"\n

useful GC log highlights

Set up the above config and kick off some load. Start tailing the GC log and\nwait for the eden size to stabilize. It often gets close within a few seconds of\nrunning load, but give it a minute or so to be sure then start looking at the\ndetailed information. Each GC log section is rather large so I'm not going to\ndocument it here. There are three lines that provide most of what we need to\nknow.

[Object Copy (ms): Min: 157.6, Avg: 161.5, Max: 162.2, Diff: 4.6, Sum: 1292.0]\n

Object Copy time is embedded in a larger block of stats. With most of the\nsystems I've examined, this is where the vast majority of the STW time is spent,\nso the trick is to tune the JVM so that it does less copying of objects. As\nmentioned earlier, start with adding heap space and offheap memtables.\nCompaction is particularly pragmatic (as observed through jvisualvm or sjk-plus\nttop) and there doesn't seem to be much we can do about it. Throttling\ncompaction can even make it worse by forcing Cassandra to keep objects in memory\nlonger than necessary, causing promotion which leads to memory compaction which\nis bound by memory bandwidth of the system.

[Eden: 4224.0M(4224.0M)->0.0B(4416.0M) Survivors: 576.0M->448.0M Heap: 6334.9M(8192.0M)->2063.4M(8192.0M)]\n

This is where you can see how much of the heap is being used for eden space. It\nwill go to 0.0B every time this is printed, since with the default logging it\nonly prints it after a STW. The survivors number under G1 rarely go over\n1GB and usually hover in the 200-300MB range. If it goes over 1GB there might be\nsomething wrong in the DB worth investigating. The last part, \"Heap:\"\nshows the total amount of allocated heap space. This will vary the most. If it\nhovers at 90-100% of the total heap, you're probably using Solr and have a lot\nof data and will need a bigger heap.

[Times: user=1.73 sys=0.00, real=0.20 secs]\n

I don't typically use these final numbers for much tuning, but they're good to\nglance at every once in a while to get an idea how much of your CPU is being\nburned for GC. user= represents the amount of CPU time consumed on all cores and\nis usually a multiple of real=. If sys is significant relative to the other\nnumbers, it probably points at contention somewhere in the system (sometimes\ndebuggable with strace -c). Finally, the real= part is wall-clock time and will\ncorrelate with the observable pause.

Always Pre-Touch

# Make sure all memory is faulted and zeroed on startup.\n# This helps prevent soft faults in containers and makes\n# transparent hugepage allocation more effective.\nJVM_OPTS=\"$JVM_OPTS -XX:+AlwaysPreTouch\"\n

Disable Biased Locking

Biased locking is an optimization introduced in Hotspot 1.5 that optimizes\nsingle-writer locks. It's a win in systems that have mostly uncontended\nlocking. Cassandra is a large system with many contended locks in hot paths\nmaking this optimization counter-productive. The difference between having this\nenabled/disabled is difficult to detect unless the system is running close to\nfull capacity.

# Biased locking does not benefit Cassandra.\nJVM_OPTS=\"$JVM_OPTS -XX:-UseBiasedLocking\"\n

https://blogs.oracle.com/dave/entry/biased_locking_in_hotspot

http://www.azulsystems.com/blog/cliff/2010-01-09-biased-locking

http://mechanical-sympathy.blogspot.com/2011/11/biased-locking-osr-and-benchmarking-fun.html

Thread Local Allocation Blocks

TLABs are enabled by default in Cassandra, but the option is mixed in with some\nCMS stuff so it occasionally gets dropped by accident when switching to G1 so\nit's worth calling out as important. With the number of threads in play in a\nCassandra instance, it's worth also enabling TLAB resizing if only to recover\nthe TLAB from threads that rarely wake up or do significant allocation. Right\nnow this is just a theory, but being able to increase the size of TLAB is likely\na big win for Cassandra since a few threads (e.g. compaction) allocate large\namounts of memory making any opportunity to avoid a GC lock a big win. That's\nthe theory and although a statistically significant difference between\n+/-ResizeTLAB could not be found in simple tests, this is a common and practical\noptimization that should be enabled.

# Enable thread-local allocation blocks and allow the JVM to automatically\n# resize them at runtime.\nJVM_OPTS=\"$JVM_OPTS -XX:+UseTLAB -XX:+ResizeTLAB\"\n

other JVM applications

A good chunk of applications using Cassandra are built on the JVM. Quite often\neven our own tools (e.g. cassandra-stress) have simple GC and tuning settings\nthat limit performance. Copying Cassandra's settings is not the answer; many of\nthe things that are good for Cassandra are bad for smaller/simpler apps. That\nsaid, here's the settings I use for G1 with cassandra-stress and many of the\nother tools in the distribution, as well as other JVM apps. It's not universal,\nbut perhaps a better starting point than the defaults.

java -server -ea \\\n  -Xmx8G -Xms1G \\\n  -XX:+UseG1GC \\\n  -XX:+AggressiveOpts -XX:+UseCompressedOops \\\n  -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods \\\n  $MAIN\n

The most visible deferred cost of writing to Cassandra is compaction. I find it\nuseful to describe it in terms of compound interest: you get to write at every\nhigh throughput now (borrowing), but at some point you have to redo all that IO\n(principal) with a fair amount of waste (interest) to maintain acceptable reads\n(credit score).

concurrent_compactors

On SSDs I start at 4 and go up from there if the workload requires it. The\nbiggest problem with huge numbers of compactors is the amount of GC it\ngenerates, so make sure to watch your GC logs or p99 client latency to make sure\nthat additional compactors doesn't ruin your latency.

Whether or not to throttle compactions should follow the same reasoning. With\nCassandra 2.1, compaction properly takes advantage of the OS page cache, so\nread IO isn't as big of a deal as it was on 2.0. Limiting the amount of IO\nallowed for compaction also cuts the amount of GC generated by the compaction\ncode, so it's still a good idea even on very fast storage.

STCS

This is the original deal and is by far the most widely deployed and tested\ncompaction strategy. This should be the default choice when you\ndon't know what to do or have insufficient information. I haven't done much\ntuning of the various knobs available, so please let me know if there's anything\ninteresting I should be looking at.

LCS

Use LCS when you need to fill up disks past 50% or have\nreally tight read SLAs. Otherwise, stick with STCS.

LCS has significant issues with streaming / resulting compaction with node\ndensities > 800GB especially. This is getting addressed in Cassandra 2.2, but is\na real issue when bootstrapping new nodes or data centers currently.

DTCS

http://www.datastax.com/dev/blog/dtcs-notes-from-the-field

Cassandra uses memory in 3 ways: Java heap, offheap memory, and OS page cache\n(a.k.a. buffer cache but only uncool people call it that anymore). You want as\nmuch RAM as you can get, up to around 256GB. For a read-heavy system it might\neven make sense to go into the 512GB-2TB range (all the major server vendors\nhave 2TB servers now), but really you want to scale out rather than up whenever\npossible.

The Page Cache

Cassandra relies heavily on the operating system page cache for caching of data\non disk. Every read that is served from RAM is a read that never gets\nto a disk, which has a systemic effect of reducing load across the board. Every\npage of RAM that is available for caching helps, which is why I'll cheerfully\nrecommend 128GB of RAM even though Cassandra itself only consumes a fraction of\nit directly.

http://queue.acm.org/detail.cfm?id=1814327

Swap (Always Say Never)

Prior to the 2000's, RAM was often the biggest line item on server quotes. I\nremember spending more than $80,000 for 8GB of RAM in a PA-RISC system in ~2001.\nWhen RAM is $10,000/GB and disk is $250/GB, swap makes sense. Even on\nthose systems any swap usage was catastrophic to performance, which is why the\nbusiness was willing to spend the big numbers on RAM. In today's age of < $100\nfor an 8GB DIMM, using swap in any latency-sensitive system is silly.

Always disable swap. In addition, always set /proc/sys/vm/swappiness to 1 just\nin case it gets reenabled by accident. The default value is 60, which tells the\nkernel to go ahead and swap out applications to make room for page cache\nheaddesk. With both settings in place, the system should never swap and that's\none less thing to think about when tracking down latency problems.

Recommendation:

swapoff -a\nsed -i ‘s/^\\(.*swap\\)/#\\1/' /etc/fstab\necho \"vm.swappiness = 1\" > /etc/sysctl.d/swappiness.conf\nsysctl -p /etc/sysctl.d/swappiness.conf\n

numactl & -XX:+UseNUMA

The quickest way to tell if a machine is NUMA is to run \"numactl --hardware\".

One of the big changes to systems in the last decade has been the move from the\nIntel Pentium front-side bus architecture to Non-Uniform Memory Architecture,\na.k.a. NUMA. Really, it's two changes in one: modern x86 CPUs have integrated\nthe memory controller onto the same die. This means that in a multi-socket\nsystem, there are two memory controllers. Rather than making one CPU have to ask\nthe other for all memory, each CPU gets a share of the memory (usually\nsymmetrical), and they only talk to each other when a process executing on one\nCPU needs memory located on the other CPU's memory bus. There are a bunch of\noptimizations in the hardware to make this as painless as possible, but as\nusual, there's still a cost in latency. When an application or thread only uses\nmemory local to the CPU, things go really fast and when that fails, things go\nslower.

By default, the bin/cassandra script will prepend the JVM command with\nnumactl --interleave. This is a good default that will enable decent performance on\nmost systems. That said, there are more options for NUMA systems that may open\nup additional performance. One is to comment out the numactl --interleave in\nbin/cassandra and add -XX:+UseNUMA to cassandra-env.sh. This instructs the JVM\nto handle NUMA directly. The JVM will allocate memory across NUMA domains and,\naccording to docs, will divide GC across domain domains. It does not do any\nthread pinning though (the code exists but is a noop on Linux). I've tested\nUseNUMA with a 256GB heap and it does work, but it's not necessarily\n--interleave.

The fastest option is for multi-JVM setups on NUMA where you can use numactl\n--cpunodebind to lock a JVM to a particular NUMA node so all memory is local and\nthreads are not allowed to execute on remote cores. This is the highest\nperformance option, but does limit the process to one socket, so use it with\ncaution. There are also problems with availability in the face of a server\nfailure, so please be careful if you try this route.

http://frankdenneman.nl/2015/02/27/memory-deep-dive-numa-data-locality/

zone_reclaim_mode, destroyer of p99s

This is usually disabled by default, but if by chance it is enabled, you will\nlikely observe random STW pauses caused by the kernel when zone reclaim fires.

https://www.kernel.org/doc/Documentation/sysctl/vm.txt (at the very bottom)

http://frosty-postgres.blogspot.com/2012/08/postgresql-numa-and-zone-reclaim-mode.html

http://docs.datastax.com/en/cassandra/2.1/cassandra/troubleshooting/trblshootZoneReclaimMode.html

The slowest part of a node is going to be either the disk or the network. Even\nin the age of SSDs, it's difficult to predict which is best. A good 10gig\nnetwork can get below 40µs latency, which can keep up with SATA and SAS. This is\nwhy we're starting to see SSD NAS become usable in the public clouds.

When a transaction is served entirely out of memory, the client txn latency is\nroughly:

memory_txn_latency + network_latency + client_latency\n

Cache misses are always worse:

disk_latency + memory_txn_latency + network_latency + client_latency\n

This is not unique to Cassandra; every durable database with data > RAM has to\ndeal with disks as the wildcard in client latency. Cache miss latency is\ndominated by disk access time. No amount of magic can make that go away (though\nrapid read protection may hide it).

http://tobert.github.io/post/2014-11-13-slides-disk-latency-and-other-random-numbers.html

Solid State Drives (a.k.a. flash drives)

Leaving transport/HBA aside for the moment, SSD is absolutely the preferred\nsolution for every workload. There are a few exceptions where HDD makes sense\nfor economical reasons, but the arguments get shaky as soon as you start looking\nat the vast difference in latency and power consumption. SSDs have no moving\nparts as well, so while they can fail, it's a lot more predictable if you\nmonitor the retired cells over SMART.

http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead

https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/

Most flash on the market today is referred to as MLC, which usually refers to\n2-level cells. This means that each flash cell can hold 2 bits of data. SLC\n(Single Level Cell) is difficult to find these days and is obscenely expensive\nso forget about it. Some Samsung drives are based on TLC (triple) to get 3 bits\nper cell, while SanDisk has some drives at 4 bits per cell. The base\nrecommendation for production workloads is MLC. Some folks are testing TLC for\nhigh-volume/low-write (DTCS) workloads and having some success. YMMV.

https://en.wikipedia.org/wiki/Multi-level_cell

Hard Disk Drives (a.k.a. spinning rust)

Hard drives get the moniker \"spinning rust\" because they are literally a few\ndiscs made out of iron spinning while a mechanical arm moves a sensor across\nthem to detect changes in the magnetic field. They have served us well for\ndecades and are by far the cheapest storage available. They still show up in new\nsystems for a few reasons:

You already know that SSDs are better, so when do HDDs make sense? Since the\npreferred size of a Cassandra <= 2.1 node is still around 4-5TB, the answer is\nusually never. There are exceptions, such as mostly-write workloads + DTCS where\nseeking isn't as big of a problem. Sometimes you already bought the\nmachine or are mitigating against drive failures elsewhere, so they have to be\nmade to work. The trick to getting HDDs to perform as well as possible is to\ntune things for linear IO wherever possible. Sometimes this means deeper queues\nnomerges=0, nr_request=256 and sometimes it means getting more RAM. When HDDs\nare the only option, get as much RAM as you can. Every cache hit in RAM means\nless IO on the drive. It's not a big deal on SSDs where random IO isn't\npenalized, but on an HDD that IO will probably cause a seek which is exactly the\nworst thing to do to an HDD.

Transports: SATA, SAS, NVMe, PCIe, and virtualized

Most of the drives in production today are either SAS or SATA. SAS\nHBAs are preferred over SATA even when using SATA drives. SATA drives work fine\non SAS controllers that can scale a little better. That said, in simple machines,\nusing the onboard AHCI SATA controller is fine. One important difference that's\nuseful to note is that SAS is rated at 1 undetectable error out of\n10^16 bits while SATA drives are typically in the 10^15 range. While Cassandra has\nmultiple levels of mitigation (sstable checksums and replication), this can be a\nuseful way to convince people to move to SAS. Since NL-SAS is basically a\nSATA drive with a SAS controller, they should have the same error correction as\nSAS, but be sure to check the data sheet for the drive.

NVMe is the new kid on the block. It is a standard similar to AHCI for SATA or\nUHCI for USB that specifies a hardware/driver and physical interface for\nPCI-Express (PCIe) flash devices that is optimized for parallelism, unlike\ntraditional block interfaces.

For the next year or two it is really important to verify the kernel NVMe driver\nis up-to-date. The early releases of NVMe for Linux were riddled with bugs and\nmissing features. Most kernels released after 2015-05 should be fine.

https://communities.intel.com/community/itpeernetwork/blog/2015/06/09/nvm-express-linux-driver-support-decoded

PCI-Express cards such as those sold by Intel, FusionIO (now SanDisk), and\nSamsung are still commonly deployed. These are by far the highest-performing\noption, with latencies measured in microseconds.

When running in a virtualized environment, a number of new variables have to be\nconsidered. The most common and easy to address problem is use of virtual IO\ndrivers rather than emulation. Most of the public cloud images are virtio-native\nthese days. Make sure to check. On some private clouds you may find images\nmisconfigured to use an emulated SCSI adapter from the 1990's or even worse, an\nBX440 IDE adapter. These use significantly more CPU and memory bandwidth than\nvirtio drivers that cooperate with the hypervisor to provide decent performance.\nThat said, the best option is IOMMU access to an underlying device, but that's\nfairly rare. It will be more common in the near future as cloud providers roll\nout NVMe.

http://www.techrepublic.com/blog/data-center/how-sas-near-line-nl-sas-and-sata-disks-compare/

https://en.wikipedia.org/wiki/NVM_Express

http://tobert.org/disk-latency-graphs/

Amazon EBS, Google PD, Azure Volumes

Amazon EBS \"standard\" = BAD, AVOID AT ALL COSTS (literally!)

Amazon EBS \"io1\" or “gp2” = (notbad), go for it!.

The new SSD-backed \"general purpose\" EBS has a latency of 1ms or less most of the\ntime and pairs nicely with c4.4xlarge instances. Make sure the instance is EBS\noptimized. Make sure to take a look at the EBS Product Details\npage to see how IO bursting plays out.\nInterestingly, if the volume size exceeds 3.4TB, the volume is automatically\nbumped to 10,000 IOPS which is a great deal compared to io1 volumes and\nespecially i2.2xlarge clusters. io1 volumes are also useful but are more expensive so\nconsider them for the commit log, but most users will be best served by gp2.

SAN/NAS

Just say no. Friends don't let friends and all that. There are few exceptions to\nthis rule. One is the new wave of PCI-Express DAS flash sold by EMC, Netapp, and\nothers. These devices are quite expensive but are popping up all over the\nstorage industry. In some SAN/NAS shops we may be able to leverage partnerships\nwith the big storage vendors to bring in a managed flash device. The key to\nrecognizing appropriate DAS storage is when it can be deployed as one array per\nrack, or one DAS box per availability zone/rack.

discovery

When getting acquainted with a new machine, one of the first things to do is\ndiscover what kind of storage is installed. Here are some handy commands:

IO elevator, read-ahead, IO merge

Folks spend a lot of time worrying about tuning SSDs, and that's great, but on\nmodern kernels these things usually only make a few % difference at best. That\nsaid, start with these settings as a default and tune from there.

When in doubt, always use the deadline IO scheduler. The default IO scheduler is\nCFQ, which stands for \"Completely Fair Queueing\". This is the only elevator that\nsupports IO prioritization via cgroups, so if Docker or some other reason for\ncgroups is in play, stick with CFQ. In some cases it makes sense to use the noop\nscheduler, such as in VMs and on hardware RAID controllers, but the difference\nbetween noop and deadline is small enough that I only ever use deadline.\nSome VM-optimized kernels are hard-coded to only have noop and that's fine.

echo 1 > /sys/block/sda/queue/nomerges # SSD only! 0 on HDD\necho 8 > /sys/block/sda/queue/read_ahead_kb # up to 128, no higher\necho deadline > /sys/block/sda/queue/scheduler\n

I usually start with read_ahead_kb at 8 on SSDs and 64 on hard drives (to line\nup with Cassandra <= 2.2's sstable block size). With mmap IO in <= 2.2 and all\nconfigurations >= 3.0. Setting readahead to 0 is fine on many configurations but\nhas caused problems on older kernels, making 8 a safe choice that doesn't hurt\nlatency.

Beware: setting readahead very high (e.g. 512K) can look impressive from the\nsystem side by driving high IOPS on the storage while the client latency\ndegrades because the drives are busy doing wasted IO. Don't ask me how I know\nthis without buying me a drink first.

TRIM & fstrim

Most SSDs and some virtual disks support some form of TRIM that allows the\noperating system to inform the device when a block is no longer referenced so\nthat it can be erased and returned to the free space pool. This helps the disk\ncontroller do efficient wear leveling and avoid latency spikes when the free\ncell pool gets low.

xfs and ext4 support the 'discard' mount option, but you should not use it. The\nslowest part of an SSD is erasing previously used cells. This can take an\neternity in computer time, tying up command slots and occasionally blocking\napplications. No good. There is an alternative though, and that's\n'fstrim' or the 'wiper.sh'\nscript from the hdparm package. Run one of them around once a week, one node at a time\njust like repair so that the whole cluster doesn't hiccup at once.

partitioning & alignment

Whenever possible, I prefer to use GPT disk labels instead of the classic MBR\npartition tables. GPT offers a number of advantages such as support for drives\nlarger than 4TB and better resiliency through writing at both the beginning and\nend of the drive. In addition, the GPT tools on Linux usually align partitions\nto a 1MB boundary automatically, which avoids any potential block misalignment.

Here is one of the scripts I use for setting up GPT, MDRAID, and xfs. It should\nbe fairly easy to adapt to other environments:\nhttps://gist.github.com/tobert/4a7ebeb8fe9446687fa8

Partition misalignment happens on drives with 4KB sectors, which is the size on\nall SSDs and most hard drives manufactured in the last few years. Some of these\ndevices will emulate 512 byte blocks to support older operating systems, so you\ncan't rely on what Linux tells you. Since it's always safe to use 4K alignment\non 512 byte devices, it's best to always align on 4K boundaries or go with 1\nmegabyte to be safe. Intel's EFI specifications recommend aligning on 128MiB\nboundaries, but that's just silly.

When deploying consumer-grade flash drives, it may make sense to leave some\nspace fallow either by not partitioning part of the drive or by creating a\npartition that will not be used. This free space acts as an additional reservoir\nof spare flash cells for the wear leveling controller in the drive to use and\nmay extend the useful lifetime of the drive.

fio

fio is the tool of choice for benchmarking filesystems and drives.

http://tobert.github.io/post/2014-04-28-getting-started-with-fio.html

http://tobert.github.io/post/2014-04-17-fio-output-explained.html

JBOD

It's just a bunch of disks, how hard can it be? Take a look at\nCASSANDRA-7386,\nCASSANDRA-8329, and\nCASSANDRA-6696

JBOD is almost always the fastest option for storage aggregation when the\nsoftware supports it, and in Cassandra's case it does, but with a few caveats. I've\navoided JBOD configs for a while because of these caveats, but it looks like its\ntime may finally be coming with Cassandra 2.2 and 3.0.

If you're stuck on huge SATA drives, definitely give it a second thought. SATA\ndrives in particular can benefit from JBOD rather than RAID0, since drive\nfailures are likely to be more painful and allowing each drive to have a\nseparate command queue distributes seeks better rather than having all drives do\nall seeks as in RAID0.

http://tobert.github.io/post/2014-06-17-jbod-vs-raid.html

MDRAID

I don't have data to support this, but by my estimation, the majority of\nCassandra clusters in production today are using Linux's MDRAID subsystem. It\nhas been around for a long time and is battle-tested. Given a\nchoice, I will take a simple JBOD SAS controller with MDRAID over hardware RAID\nevery time. It's more flexible and all the tools are fairly easy to use. For\nRAID0, there really isn't much hardware can do to accelerate it outside of\ncaching. Even when using RAID5 and RAID6, software RAID is preferrable since\nparity calculation (RAID[56]) is accelerated on Intel CPUs since Westmere.

RAID0 is common in combination with Cassandra because it provides the simplest\nmanagement combined with good performance and capacity. Drive failures aren't a\nhuge deal since Cassandra is replicating the data.

A typical RAID0 setup looks like:

mdadm --create /dev/md0 --chunk=256 --metadata=1.2 --raid-devices=6 --level=0 /dev/sd[cdefgh]1\n

Most of this is straightforward except for the chunk size. Most of the time the\nchunk should be 64-256 bytes and should always be a power of 4096. The fio\nnumbers are best at 128-256K and that tends to be the size of \"erase blocks\" on\nSSDs, so that's what I usually go with.

Of course, there are other RAID types and sometimes they make sense. It\nsometimes makes sense to use RAID1 for a commit log or RAID10 in situations\nwhere maximum availability is required or servicing hard drives is\ndifficult/expensiver, e.g. remote datacenters with expensive remote hands. RAID5\nand RAID6 don't show up much, but I've used them with Cassandra to good effect.\nYou get some redundancy for the price of lower write speed (reads in modern\nparity RAID are often better than you'd expect). The trick with parity RAID is\nmaking extra sure the filesystem is informed of the stripe width so it can\nallocate space in stripe-sized chunks. Always make sure to check\n/sys/block/md0/md/stripe_cache_size and set it to 16KB or\npossibly higher.

HW RAID

The biggest difficulty with HW RAID is that most of the CLI tools are really,\nreally awful. Another problem is that they present virtual block devices to the\nOS, making it difficult to probe drives directly or read their SMART data.\nThat said, they can be set up to offer decent performance. It is often helpful\nto check the firmware errata of a RAID card before chasing other parts of the\nsystem as HW manufacturers have to ship firmware to the factory far in advance\nof GA, which sometimes results in buggy firmware being shipped.

The other thing to keep an eye out for is write-through v.s. write-back caches\non these cards. The NVRAM on HW RAID cards is their biggest advantage, often\ncoming in sizes of 512MB or even bigger these days. When a battery backup is\npresent on the card, write-back caching can provide incredible speedups. The\nbattery is necessary to keep the NVRAM online during a power failure so any\noutstanding IOs can be flushed to stable storage when the power comes back.\nThese batteries have to be serviced every few years, so some users will opt\nout of the cost and hassle. In that case, it can still be set to write-through\ncaching for some additional performance, but most of the time I'd opt for a JBOD\ncard + MDRAID to with plenty of RAM keep things simple.

LVM / device-mapper

TODO: evaluate dm-cache

TODO: evaluate dm-delay for latency simulation

The Linux kernel includes a block IO virtualization layer called device-mapper.\nThe current LVM system is built on top of this and closely resembles the LVM\nfrom HP-UX. Most of the time, LVM is only in the way on Cassandra systems, but\nit does show up frequently since many enterprises use LVM for all of\ntheir disk management. The Redhat/CentOS/Fedora installers default to installing\nthe OS on an LV, so it's bound to show up.

The critical commands to know are vgdisplay -v and vgscan. vgdisplay will\nshow you all of the volume groups on a system and with -v it will also show you\nthe LVs and PVs. vgscan will scan all the drives in the system looking for LVM\nPV signatures.

LVM also includes mirroring and striping modules based on dm-raid. These can be\nused in place of MDRAID, but given the complexity in LVM,\nmy recommendation is to stick with MDRAID. This may change as time marches on\nand dm-cache becomes a little easier to use.

Since LVM is built on device-mapper, you can find LVs by running ls /dev/mapper/.\nThis is why you'll often see /dev/mapper in the device name in\nmount listings. The /dev/$VG/ paths are symlinks to the devmapper devices.\nAnother useful trick is to use the dmsetup command directly. This gives you\nlow-level access behind LVMs back to examine disk layouts. In particular,\ndmsetup ls is useful.

Cassandra relies on a standard filesystem for storage. The choice of\nfilesystem and how it's configured can have a large impact on performance.

One common performance option that I find amusing is the noatime option. It\nused to bring large gains in performance by avoiding the need to write to inodes\nevery time a file is accessed. Many years ago, the Linux kernel changed the\ndefault atime behavior from synchronous to what is called relatime which means\nthe kernel will batch atime updates in memory for a while and update inodes only\nperiodically. This removes most of the performance overhead of atime, making the\nnoatime tweak obsolete.

Another option I've seen abused a few times is the barrier/nobarrier flag. A\nfilesystem barrier is a transaction marker that filesystems use to tell\nunderlying devices which IOs need to be committed together to achieve\nconsistency. Barriers may be disabled on Cassandra systems to get better disk\nthroughput, but this should NOT be done without full understanding of what\nit means. Without barriers in place, filesystems may come back from a power\nfailure with missing or corrupt data, so please read the mount(8) man page first\nand proceed with caution.

xfs (just do it)

xfs is the preferred filesystem for Cassandra. It is one of most mature\nfilesystems on Earth, having started in SGI Irix, now with well over a decade in\nthe Linux kernel. It offers great performance over time and can be tuned to\nsupport a variety of underlying storage configurations.

mkfs.xfs will try to detect drive and RAID settings automatically. It almost\nalways gets this wrong because the Linux kernel gets it wrong because most\ndrives lie to the operating system in order to support ancient operating systems\nthat hard-coded 512 byte blocks. All of that is to say, when creating new\nfilesystems, always explicitly set the block size and RAID parameters to be sure\nthey're correct.

For partitions or whole drives, setting just the block size should be\nsufficient. Nearly every drive sold in the last few years has a 4K block size.\nSetting a 4K block size on a 512 byte device doesn't hurt much, while setting a\n512 byte block size on a 4K device causes extra work for the drive in the form\nof read-modify write for 512 byte block updates. TL;DR, always set -s size=4096.

mkfs.xfs -s size=4096 /dev/sdb1\n

And on a RAID device (adjust to the local configuration):

mkfs.xfs -s size=4096 -d su=262144 -d sw=6 /dev/md0\n

This is a potential SSD optimization but the data so far is inconclusive. It\ndoesn't seem to hurt anything though, so I'm mentioning it in hopes that someone\nelse will figure out if it's worth the effort. The idea is to set the stripe\nwidth to the erase block size of the underlying SSD, usually 128K (256 * 512)\nand then set the stripe unit (a.k.a. chunk size) to 4K (8 * 512) to match the\nblock size.

mkfs.xfs -f -s size=4096 -d sunit=8 -d swidth=256 /dev/sdb1\n

If you're setting the sunit/swidth, it's worth passing the same values through\nto mount via mount -o or /etc/fstab. The man page says these only need to be set\nwhen changing the geometry of a RAID device, but when they're not set the kernel\nreports the wrong values for them, so to be safe always set them in /etc/fstab.

ext4 (if you must)

The ext4 filesystem is evolved from the ext line of filesystems in the Linux\nkernel. It is almost as fast as ext2 was, and much faster than ext3. ext2 and\next3 filesystems can be upgraded in-place to ext4 and you should do so,\nespecially for ext3. While ext2 is a bit faster than ext4 due to the lack\nof a journal, it is not recommended since it will block reboots on fsck after\npower failures.

Choose ext4 when the local policy demands it and follow the same RAID alignment\nguidance as xfs.

ZFS (if you love yourself and don't need commercial support)

ZFS-on-Linux has been around for a few years now and is quite stable. I've\ndeployed Cassandra on ZFS and it's a beautiful fit. The big downside is that\nthere's no commercial support available. One feature that works particularly\nwell with Cassandra is ZFS's SLOG and L2ARC devices, which allow you to use an\nSSD for journaling and caching in front of slower drives. It also offers inline\ncompression which may be handy for getting better compression ratios than those\nbuilt into Cassandra.

http://zfsonlinux.org/

btrfs

btrfs has a reputation for being unreliable and should not be deployed to\nproduction systems without extensive testing to see if it's safe for your\nworkload. I've run btrfs in production with Cassandra in the past and it worked\ngreat, particularly on EC2 ephemeral disks with LZO compression enabled. Your\nmileage will certainly vary, so only choose btrfs if you're willing to risk\nsome data and spend the time testing.

It often surprises me how little discussion there is around network design and\noptimization. Cassandra is completely reliant on the network, and while we do a\ngood job of not trusting it, a little extra work in setting things up can make\nthe experience much smoother and provide better performance and availability.

Kernel tuning for network throughput is in the Linux section of this doc.

NIC selection (1g/10g/vNIC/etc.)

Prior to Cassandra 2.1, my guidance around networking was \"use 1gig,\nwhatever\". With 2.1, however, it's quite a bit easier to push machines to the\nlimit of the network. Saturating 1g interfaces is fairly easy with large write\nworkloads. 10gig is now the recommendation for high-performance clusters.

When you're stuck dealing with virtual machines, avoid emulated NICs at\nall costs. These usually show up as a Realtek or Intel e1000 adapter in the\nguest operating system and the performance is abysmal. On KVM and\nVirtualBox it should be \"virtio-net\", for Xen, it's \"xen-net\" (IIRC), and on\nVMware it should be “vmxnet3”. While the virtual NICs are much better than\nemulated NICs, the best option is often referred to as vNICs. These are usually\n10gig cards that can negotiate with the hypervisor to create shards of the NIC\nthat can be mapped directly into the guest operating system's kernel as a\nhardware device. This bypasses a lot of memory copying and CPU time and allows\nfor nearly 100% bare metal performance inside a VM. In EC2 this is known as\nenhanced networking, which should always be enabled when available.

packet coalescing & EC2

If you're hitting a performance limit in EC2 and don't have enhanced networking\nenabled, you're probably hitting the secret packets-per-second limit in EC2.\nThere are two ways to get around this. Either enable enhanced networking (which\nrequires VPC), or enable message coalescing in Cassandra (available in 2.1.5 or\nDSE 4.7.0).

http://www.datastax.com/dev/blog/performance-doubling-with-message-coalescing

Today's x86 CPUs are all multi-core and most of the Intel chips offer something\ncalled hyper-threading that makes the core count appear to double. More on that\nbelow. A system with more than one physical CPU installed is said to be\nmulti-socket, in that the motherboard has multiple CPU sockets with associated\nmemory banks (see NUMA above). Just to make sure the terminology is straight: a\nnode is a single motherboard with one or more CPU sockets with memory banks. A\nCPU may have many processing cores.

While Cassandra runs fine on many kinds of processors, from Raspberry Pis to\nMacbooks to high-end servers, all production loads should use a CPU that supports\nECC memory to avoid silent memory corruption.

When choosing CPUs, the #1 most important feature to select for is cache. An\nL1 cache hit is 0.5 nanoseconds. L2 is 7ns. Reading from RAM takes 100ns. In\nreality, RAM is even slower than that in a multi-core world, since cores often\nneed to synchronize to make sure they're not mutating the same area of memory.\nThe more cache there is on the CPU, the less often this happens.

There are usually one or two CPU models in the sweet spot of price/performance.\nThe way to find it is to look at the distribution of cache sizes first and find\nthe cache size just below that of the most expensive CPU. For example, if the\ntop-end Xeon has 32MB of cache and costs $2000 each, there's probably a\ndifferent CPU for around $600 that has 24MB of cache with a clock speed right\naround halfway between the slowest and fastest. This where the best value is.\nYou're almost always better off buying more systems to spread the work out\nrather than paying the markup on the fastest CPU available.

a word on hyperthreading

A hyperthread is a virtual core or \"sibling\" core that allows a single core to\npretend as if it's 2 cores. They exist to get more work out of the silicon by\npipelining multiple tasks in parallel for the same backing silicon. Many\ndatabases advise that this be turned off, but overall it's a benefit for\nCassandra.

Recommendation: enable HT

EC2 note: the newer generations of EC2 instances seem to be claiming\nhyperthreading cores as real cores. If you take a look at /proc/cpuinfo on an\ni2.2xlarge, you will see 8 cores assigned to the system. If you look a little\ncloser, you can see the \"sibling id\" field indicates that half of those cores\nare indeed hyperthreading cores, so you in effect only have 4 cores worth of\nsilicon in those VMs.

C-states & frequency scaling

See also: \"powertop\" under observation tools.

Over the last few years, the cost of power for datacenters has become a more and\nmore prominent consideration when buying hardware. To that end, even Xeon\nprocessors now have power management features built into them and a lot of the\ntime this stuff is enabled out of the box. The impact on Cassandra is that when\na processor goes into power saving mode, there is a latency cost for waking it\nback up. Sometimes it's a few microseconds, sometimes it's in the milliseconds.\nThis is fine for a lot of applications that don't use the CPU all the time or\nare not latency-sensitive, it can significantly impact throughput and even the\nstability of the clock when misconfigured.

On RHEL6, CentOS6, and other older LTS distros, the default idle driver for\nIntel chips is called \"intel_idle\". This driver is very aggressive about\nputting the processor to sleep and regularly causes client-visible latency\nhiccups. To disable it, add \"intel_idle.max_cstate=0 processor.max_cstate=0\nidle=mwait\" to the kernel command line then reboot. This is not necessary on\nRHEL7 and similarly modern distros. The idle driver can be verified with\ncat /sys/devices/system/cpu/cpuidle/current\\_driver. There is also a nuclear option\navailable on Linux in the form of booting with idle=poll. This is absolutely\nnot recommended for general use; it makes the CPU run at 100% 24x7 which may\nshorten the lifetime of the hardware and wastes a lot of power. That said, it\ncan be handy to use it when you're in doubt and want to entirely eliminate CPU\nsleeping as a source of latency.

Another thing I've seen recently is machines that get configured with frequency\nscaling on by default. Frequency scaling is great on laptops where minimum power\nconsumption is more important than throughput, but it has particularly nasty\nside-effects for Cassandra. The first is that with the CPUs running at lower\nclock speeds, latency will be higher. Another is that performance will be\ninconsistent. Lastly, and most nastily, it seems to destabilize the tsc clock on\nthe system which may cause time drift.

# make sure the CPUs run at max frequency\nfor sysfs_cpu in /sys/devices/system/cpu/cpu[0-9]*\ndo\n     echo performance > $sysfs_cpu/cpufreq/scaling_governor\ndone\n

http://jpbempel.blogspot.gr/2015/09/why-bios-settings-matter-and-not-size.html

clock sources

The vast majority of Cassandra instances run on x86 CPUs, where\nthere are multiple clock sources available. The fastest and most common is\ncalled \"tsc\" which stands\nfor Time Stamp Counter. This is a register on x86 CPUs, so it is very fast. It\nhas a major downside in that it isn't guaranteed to be stable. This is part of\nthe reason why x86 machines have so much clock drift, making NTP a requirement\nfor every Cassandra node. There are alternatives available, such as HPET, ACPI,\nand the various paravirtual clocks (kvm, xen, hyperv). The problem with these\nclocks is that they sit on a bus and take an order of magnitude or more time to\nread results compared to tsc. For some applications, like Cassandra, that hammer\non the gettimeofday() syscall to get the system time, it can have a direct\nimpact on performance.

Some of the clouds are starting to move over to using paravirtual clocks,\npresumably to reduce the amount of clock drift in VMs. The xen paravirtual clock\nin particular has been observed by Netflix to cause performance problems, so\nit's a good idea to switch back to tsc, then double-check that NTP is working.

echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource\n

Source: http://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html

sysctl.conf

The primary interface for tuning the Linux kernel is the /proc virtual\nfilesystem. In recent years, the /sys filesystem has expanded on what /proc\ndoes. Shell scripts using echo and procedural code are difficult to manage\nautomatically, as we all know from handling cassandra-env.sh. This led the\ndistros to create /etc/sysctl.conf, and in modern distros, /etc/sysctl.conf.d.\nThe sysctl command reads the files in /etc and applies the settings to the\nkernel in a consistent, declarative fashion.

The following is a block of settings I use almost every server I touch. Most\nof these are safe to apply live and should require little tweaking from site to\nsite. Note: I have NOT tested these extensively with multi-DC, but most of them\nshould be safe. Those items that may need extra testing for multi-DC have\ncomments indicating it.

http://tobert.github.io/post/2014-06-24-linux-defaults.html

# The first set of settings is intended to open up the network stack performance by\n# raising memory limits and adjusting features for high-bandwidth/low-latency\n# networks.\nnet.ipv4.tcp_rmem = 4096 87380 16777216\nnet.ipv4.tcp_wmem = 4096 65536 16777216\nnet.core.rmem_max = 16777216\nnet.core.wmem_max = 16777216\nnet.core.netdev_max_backlog = 2500\nnet.core.somaxconn = 65000\nnet.ipv4.tcp_ecn = 0\nnet.ipv4.tcp_window_scaling = 1\nnet.ipv4.ip_local_port_range = 10000 65535\n# this block is designed for and only known to work in a single physical DC\n# TODO: validate on multi-DC and provide alternatives\nnet.ipv4.tcp_syncookies = 0\nnet.ipv4.tcp_timestamps = 0\nnet.ipv4.tcp_sack = 0\nnet.ipv4.tcp_fack = 1\nnet.ipv4.tcp_dsack = 1\nnet.ipv4.tcp_orphan_retries = 1\n# significantly reduce the amount of data the kernel is allowed to store\n# in memory between fsyncs\n# dirty_background_bytes says how many bytes can be dirty before the kernel\n# starts flushing in the background. Set this as low as you can get away with.\n# It is basically equivalent to trickle_fsync=true but more efficient since the\n# kernel is doing it. Older kernels will need to use vm.dirty_background_ratio\n# instead.\nvm.dirty_background_bytes = 10485760\n# Same deal as dirty_background but the whole system will pause when this threshold\n# is hit, so be generous and set this to a much higher value, e.g. 1GB.\n# Older kernels will need to use dirty_ratio instead.\nvm.dirty_bytes = 1073741824\n# disable zone reclaim for IO-heavy applications like Cassandra\nvm.zone_reclaim_mode = 0\n# there is no good reason to limit these on server systems, so set them\n# to 2^31 to avoid any issues\n# Very large values in max_map_count may cause instability in some kernels.\nfs.file-max = 1073741824\nvm.max_map_count = 1073741824\n# only swap if absolutely necessary\n# some kernels have trouble with 0 as a value, so stick with 1\nvm.swappiness = 1\n

On vm.max_map_count:

http://linux-kernel.2935.n7.nabble.com/Programs-die-when-max-map-count-is-too-large-td317670.html

limits.conf (pam_limits)

The DSE and DSC packages install an /etc/security/limits.d/ file by default that\nshould remove most of the problems around pam_limits(8). Single-user systems\nsuch as database servers have little use for these limitations, so I often turn\nthem off globally using the following in /etc/security/limits.conf. Some\nusers may already be customizing this file, in which case change all of the\nasterisks to cassandra or whatever the user DSE/Cassandra is running as.

* - nofile     1000000\n* - memlock    unlimited\n* - fsize      unlimited\n* - data       unlimited\n* - rss        unlimited\n* - stack      unlimited\n* - cpu        unlimited\n* - nproc      unlimited\n* - as         unlimited\n* - locks      unlimited\n* - sigpending unlimited\n* - msgqueue   unlimited\n

chrt

The Linux kernel's default policy for new processes is SCHED_OTHER. The\nSCHED_OTHER policy is designed to make interactive tasks such as X windows and\naudio/video playback work well. This means the scheduler assigns tasks very\nshort time slices on the CPU so that other tasks that may need immediate service\ncan get time. This is great for watching cat videos on Youtube, but not so great\nfor a database, where interactive response is on a scale of milliseconds rather\nthan microseconds. Furthermore, Cassandra's threads park themselves properly.\nSetting the scheduling policy to SCHED_BATCH seems more appropriate and can\nopen up a little more throughput. I don't have good numbers on this yet, but\nobservations of dstat on a few clusters have convinced me it's useful and\ndoesn't impact client latency.

chrt --batch 0 $COMMAND\nchrt --batch 0 --all-tasks --pid $PID\n

You can inject these into cassandra-env.sh or\n/etc/{default,sysconfig}/{dse,cassandra} by using the $$ variable that returns\nthe current shell's pid. Child processes inherity scheduling policies, so if you\nset the startup shell's policy, the JVM will inherit it. Just add this line to\none of those files:

chrt --batch 0 --pid $$\n

taskset & isolcpus & irqbalance

Sometimes a machine ends up spending more time processing IO requests\n(interrupts) than it does getting application (Cassandra) work done. This often\nmanifests as high system CPU time and a large amount of context switches and\ninterrupts observed in vmstat/dstat. There are two major approaches to handling\nthis. One is to reserve a core or two for interrupt processing. This is common\nin real-time use cases such as music production or high-frequency trading. The\nother is to evenly distribute interrupts over the cores in a system. One of the\ndifficult-to-observe benefits of a reserved CPU core is that the kernel's code\ncan stay hot in cache on that core. The reserved core may never surpass 10%\nutilization, but the latency benefits are sometimes worth it.

taskset

The easiest way to get started on a running system is with the taskset utility.\nTaskset can be used to tell the Linux scheduler which CPUs are available to a\nprocess or thread. The results of taskset are usually observable within a couple\nseconds. After moving load off one of the cores, it may be necessary to manually\nmove interrupts over to the core.

taskset -apc 2-7 $CASS_PID\ntaskset -c 2-7 ./cassandra -f\ntaskset -pc 2-7 $$ # in /etc/{default,sysconfig}/{dse,cassandra}\n

isolcpus

If you can reboot, you can reserve cores for the kernel by using the isolcpus=\nkernel command line option. In my tests, the kernel automatically schedules its\ntasks on the reserved CPU. You may need to do some additional IRQ management to\nget it right, but it's worth the effort when it works out. In order to enable it\nyou will need to edit the grub configuration in /etc/default/grub on Debian or\n/etc/sysconfig/grub on Redhat. For quick tests you can edit /boot/grub/grub.conf\n(grub2) or /boot/grub/menu.lst (grub1 & pvgrub (EC2)) and add it to the end of\nthe kernel options. This will probably get reverted on the next kernel\nupgrade, so make sure to do it in the way prescribed by the Linux distribution.\nMy test machine's /proc/cmdline looks like this:

BOOT_IMAGE=../vmlinuz-linux root=PARTUUID=91012260-6834-425a-b488-9dd9f537a294 rw isolcpus=0-1 initrd=../initramfs-linux.img\n

Make sure irqbalance is disabled or configured to wire interrupts to the\nselected CPU. On NUMA machines it's important to make sure the reserved core is\non the socket that manages the PCIe devices. Core 0 is almost always the right\nchoice.

irqbalance

Some of the advice you'll find on the internet says to disable irqbalance for\nperformance. This was good advice for early versions of it that were prone to\nrebalancing storms, but it isn't bad these days. If in doubt, disable the\nirqbalance service and run \"irqbalance --oneshot\" once at boot time and forget\nabout it.

dirty tricks with renice, ionice, and taskset

This is one of my favorite dirty tricks, combined with renice/ionice. If you've\never noticed the \"-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42\" settings\nin Cassandra's JVM arguments, they're there so that compaction can be set to a\nlower priority than other threads in Cassandra. This is visible in the default\nhtop configuration in the nice field.

\"image

Linux has always used a 1:1 threading model and even uses the global pid space\nto assign ids to threads, so they're still visible if you know how to look for\nthem. htop displays thread ids by default and standard top can do it if you hit\nH. With ps, the -L flag makes them show up. We can take advantage of this to do\nsome prioritization and pinning outside of Cassandra so we don't have to wait\naround for features to make their way into DSE.

for tid in $(ps -eLo tid,args,nice |awk '/java.*4$/{print $1}')\ndo\n  taskset -pc 5-6 $tid     # pin to cores 5 and 6\n  renice 20 $tid           # set the lowest nice priority\n  ionice -c 2 -n 7 -p $tid # set IO to the lowest best-effort priority\ndone\n

After that, your htop will look more like this (load is done but compaction is still going):

\"image

Note: ionice requires the cfq IO scheduler, which is usually not the case in\nproperly tuned systems. It is one of the main reasons for sticking with CFQ\noutside of Docker with io limits.

I did not come up with all of this on my own. The list of folks I've learned from\neven in just the last two years is excessively long. Thank you, all.

","id":"8777d174-861f-5ec5-9d01-586f14aa4bfa","title":"Amy's Cassandra 2.1 tuning guide","origin_url":"https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html","url":"https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html","wallabag_created_at":"2016-12-22T13:56:48+00:00","published_at":"2015-09-24T00:00:00+00:00","published_by":"['']","reading_time":72,"domain_name":"tobert.github.io","preview_picture":"https://tobert.github.io/pages/image_1.png","tags":["cassandra","troubleshooting and tuning","performance"],"description":"Amy's Cassandra 2.1 Tuning Guide (2015)I really appreciate all the folks who have told me that this guide helped them\nin some way. I'm happy to hear that.I've pushed this small update to change my nam..."},{"content":"

Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building reliable applications.

Accord is in active development and still a feature branch in the Apasche Cassandra® Repo. You will find bug. What we ask is that you help with a contribution of a bug report.

You can use the Github discussions bug report forum for this or use the Planet Cassandra Discord channel for accord listed below. A bug report should have the folowing:

If you have suggestions about syntax or improving the overall developer expirience, we want to hear about that to! Add it as a suggestion or feature request using Github discussions or let us know in the Planet Cassandra Discord.

Now, on to the fun!

docker pull pmcfadin/cassandra-accord\ndocker run -d --name cassandra-accord -p 9042:9042 pmcfadin/cassandra-accord
brew tap rustyrazorblade/rustyrazorblade\nbrew install easy-cass-lab
/\n├── docker/              # Docker configuration and setup\n├── easy-cass-lab/      # Multi-node testing environment\n├── examples/           # Implementation examples\n│   ├── banking/       # Financial transaction examples\n│   ├── inventory/     # Stock management examples\n│   └── user-mgmt/     # User operations examples\n└── docs/              # Guides and documentation\n

Our documentation includes:

  1. Choose your deployment option:\n
    2. \n
  2. Follow the Quick Start Guide
    3. \n
  3. Explore example implementations
    4. \n
  4. Connect with our Discord community
    5. \n
  5. Feedback! Github Discussions
    6. \n
BEGIN TRANSACTION\n    LET fromBalance = (SELECT account_balance \n                      FROM ks.accounts \n                      WHERE account_holder='alice');\n    IF fromBalance.account_balance >= 20 THEN\n        UPDATE ks.accounts \n        SET account_balance -= 20 \n        WHERE account_holder='alice';\n        UPDATE ks.accounts \n        SET account_balance += 20 \n        WHERE account_holder='bob';\n    END IF\nCOMMIT TRANSACTION;

Apache License 2.0

","id":"b239385a-84ae-5556-b454-b52325f69e75","title":"GitHub - pmcfadin/awesome-accord: Repository of all kinds of things to help you get up and running with ACID transactions on Apache Cassandra®","origin_url":"https://github.com/pmcfadin/awesome-accord","url":"https://github.com/pmcfadin/awesome-accord","wallabag_created_at":"2025-01-16T16:28:31+00:00","published_at":null,"published_by":"['pmcfadin']","reading_time":1,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/3e477fb2dd2b1ded1c5b53477f4848297badc75ece00c5b49bad1476fdb76167/pmcfadin/awesome-accord","tags":["acid","open.source","cassandra","accord"],"description":"Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building..."}],"tagSets":[{"tag":"cassandra","articles":[{"content":"

Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building reliable applications.

Accord is in active development and still a feature branch in the Apasche Cassandra® Repo. You will find bug. What we ask is that you help with a contribution of a bug report.

You can use the Github discussions bug report forum for this or use the Planet Cassandra Discord channel for accord listed below. A bug report should have the folowing:

If you have suggestions about syntax or improving the overall developer expirience, we want to hear about that to! Add it as a suggestion or feature request using Github discussions or let us know in the Planet Cassandra Discord.

Now, on to the fun!

docker pull pmcfadin/cassandra-accord\ndocker run -d --name cassandra-accord -p 9042:9042 pmcfadin/cassandra-accord
brew tap rustyrazorblade/rustyrazorblade\nbrew install easy-cass-lab
/\n├── docker/              # Docker configuration and setup\n├── easy-cass-lab/      # Multi-node testing environment\n├── examples/           # Implementation examples\n│   ├── banking/       # Financial transaction examples\n│   ├── inventory/     # Stock management examples\n│   └── user-mgmt/     # User operations examples\n└── docs/              # Guides and documentation\n

Our documentation includes:

  1. Choose your deployment option:\n
    2. \n
  2. Follow the Quick Start Guide
    3. \n
  3. Explore example implementations
    4. \n
  4. Connect with our Discord community
    5. \n
  5. Feedback! Github Discussions
    6. \n
BEGIN TRANSACTION\n    LET fromBalance = (SELECT account_balance \n                      FROM ks.accounts \n                      WHERE account_holder='alice');\n    IF fromBalance.account_balance >= 20 THEN\n        UPDATE ks.accounts \n        SET account_balance -= 20 \n        WHERE account_holder='alice';\n        UPDATE ks.accounts \n        SET account_balance += 20 \n        WHERE account_holder='bob';\n    END IF\nCOMMIT TRANSACTION;

Apache License 2.0

","id":"b239385a-84ae-5556-b454-b52325f69e75","title":"GitHub - pmcfadin/awesome-accord: Repository of all kinds of things to help you get up and running with ACID transactions on Apache Cassandra®","origin_url":"https://github.com/pmcfadin/awesome-accord","url":"https://github.com/pmcfadin/awesome-accord","wallabag_created_at":"2025-01-16T16:28:31+00:00","published_at":null,"published_by":"['pmcfadin']","reading_time":1,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/3e477fb2dd2b1ded1c5b53477f4848297badc75ece00c5b49bad1476fdb76167/pmcfadin/awesome-accord","tags":["acid","open.source","cassandra","accord"],"description":"Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building..."},{"content":"

Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:

Visual Flow application is divided into the following repositories:

Check the official guide.

Visual flow is an open-source software licensed under the Apache-2.0 license.

","id":"0470ca53-9271-53a1-98cb-20f2a6ae5380","title":"GitHub - ibagroup-eu/Visual-Flow: Visual-Flow main repository","origin_url":"https://github.com/ibagroup-eu/Visual-Flow","url":"https://github.com/ibagroup-eu/Visual-Flow","wallabag_created_at":"2024-12-02T13:34:31+00:00","published_at":null,"published_by":"['ibagroup-eu']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/9187fdecad3a37939c1971bcdec19ffed4090307ee508b009f47c7bcd49a7f8d/ibagroup-eu/Visual-Flow","tags":["mongo","nocode","elasticsearch","open.source","cassandra","data.pipeline","elastic","aws.s3","etl","low.code","postgres"],"description":"Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:Can integrate data from heterogeneous sources:\nA..."},{"content":"

\"GitHub \"Go

\"cql-proxy\"

cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.

The cql-proxy sidecar enables unsupported CQL drivers to work with DataStax Astra. These drivers include both legacy DataStax drivers and community-maintained CQL drivers, such as the gocql driver and the rust-driver.

cql-proxy also enables applications that are currently using Apache Cassandra or DataStax Enterprise (DSE) to use Astra without requiring any code changes. Your application just needs to be configured to use the proxy.

If you're building a new application using DataStax drivers, cql-proxy is not required, as the drivers can communicate directly with Astra. DataStax drivers have excellent support for Astra out-of-the-box, and are well-documented in the driver-guide guide.

Use the -h or --help flag to display a listing all flags and their corresponding descriptions and environment variables (shown below as items starting with $):

$ ./cql-proxy -h\nUsage: cql-proxy\nFlags:\n  -h, --help                                              Show context-sensitive help.\n  -b, --astra-bundle=STRING                               Path to secure connect bundle for an Astra database. Requires '--username' and '--password'. Ignored if using the\n                                                          token or contact points option ($ASTRA_BUNDLE).\n  -t, --astra-token=STRING                                Token used to authenticate to an Astra database. Requires '--astra-database-id'. Ignored if using the bundle path\n                                                          or contact points option ($ASTRA_TOKEN).\n  -i, --astra-database-id=STRING                          Database ID of the Astra database. Requires '--astra-token' ($ASTRA_DATABASE_ID)\n      --astra-api-url=\"https://api.astra.datastax.com\"    URL for the Astra API ($ASTRA_API_URL)\n      --astra-timeout=10s                                 Timeout for contacting Astra when retrieving the bundle and metadata ($ASTRA_TIMEOUT)\n  -c, --contact-points=CONTACT-POINTS,...                 Contact points for cluster. Ignored if using the bundle path or token option ($CONTACT_POINTS).\n  -u, --username=STRING                                   Username to use for authentication ($USERNAME)\n  -p, --password=STRING                                   Password to use for authentication ($PASSWORD)\n  -r, --port=9042                                         Default port to use when connecting to cluster ($PORT)\n  -n, --protocol-version=\"v4\"                             Initial protocol version to use when connecting to the backend cluster (default: v4, options: v3, v4, v5, DSEv1,\n                                                          DSEv2) ($PROTOCOL_VERSION)\n  -m, --max-protocol-version=\"v4\"                         Max protocol version supported by the backend cluster (default: v4, options: v3, v4, v5, DSEv1, DSEv2)\n                                                          ($MAX_PROTOCOL_VERSION)\n  -a, --bind=\":9042\"                                      Address to use to bind server ($BIND)\n  -f, --config=CONFIG                                     YAML configuration file ($CONFIG_FILE)\n      --debug                                             Show debug logging ($DEBUG)\n      --health-check                                      Enable liveness and readiness checks ($HEALTH_CHECK)\n      --http-bind=\":8000\"                                 Address to use to bind HTTP server used for health checks ($HTTP_BIND)\n      --heartbeat-interval=30s                            Interval between performing heartbeats to the cluster ($HEARTBEAT_INTERVAL)\n      --idle-timeout=60s                                  Duration between successful heartbeats before a connection to the cluster is considered unresponsive and closed\n                                                          ($IDLE_TIMEOUT)\n      --readiness-timeout=30s                             Duration the proxy is unable to connect to the backend cluster before it is considered not ready\n                                                          ($READINESS_TIMEOUT)\n      --idempotent-graph                                  If true it will treat all graph queries as idempotent by default and retry them automatically. It may be\n                                                          dangerous to retry some graph queries -- use with caution ($IDEMPOTENT_GRAPH).\n      --num-conns=1                                       Number of connection to create to each node of the backend cluster ($NUM_CONNS)\n      --proxy-cert-file=STRING                            Path to a PEM encoded certificate file with its intermediate certificate chain. This is used to encrypt traffic\n                                                          for proxy clients ($PROXY_CERT_FILE)\n      --proxy-key-file=STRING                             Path to a PEM encoded private key file. This is used to encrypt traffic for proxy clients ($PROXY_KEY_FILE)\n      --rpc-address=STRING                                Address to advertise in the 'system.local' table for 'rpc_address'. It must be set if configuring peer proxies\n                                                          ($RPC_ADDRESS)\n      --data-center=STRING                                Data center to use in system tables ($DATA_CENTER)\n      --tokens=TOKENS,...                                 Tokens to use in the system tables. It's not recommended ($TOKENS)

To pass configuration to cql-proxy, either command-line flags, environment variables, or a configuration file can be used. Using the docker method as an example, the following samples show how the token and database ID are defined with each method.

docker run -p 9042:9042 \\\n  --rm datastax/cql-proxy:v0.1.5 \\\n  --astra-token <astra-token> --astra-database-id <astra-datbase-id>
docker run -p 9042:9042  \\\n  --rm datastax/cql-proxy:v0.1.5 \\\n  -e ASTRA_TOKEN=<astra-token> -e ASTRA_DATABASE_ID=<astra-datbase-id>

Proxy settings can also be passed using a configuration file with the --config /path/to/proxy.yaml flag. This can be mixed and matched with command-line flags and environment variables. Here are some example configuration files:

contact-points:\n  - 127.0.0.1\nusername: cassandra\npassword: cassandra\nport: 9042\nbind: 127.0.0.1:9042\n# ...

or with a Astra token:

astra-token: <astra-token>\nastra-database-id: <astra-database-id>\nbind: 127.0.0.1:9042\n# ...

All configuration keys match their command-line flag counterpart, e.g. --astra-bundle is astra-bundle:, --contact-points is contact-points: etc.

Multi-region failover with DC-aware load balancing policy is the most useful case for a multiple proxy setup.

When configuring peers: it is required to set --rpc-address (or rpc-address: in the yaml) for each proxy and it must match is corresponding peers: entry. Also, peers: is only available in the configuration file and cannot be set using a command-line flag.

Here's an example of configuring multi-region failover with two proxies. A proxy is started for each region of the cluster connecting to it using that region's bundle. They all share a common configuration file that contains the full list of proxies.

Note: Only bundles are supported for multi-region setups.

cql-proxy --astra-bundle astra-region1-bundle.zip --username token --password <astra-token> \\\n  --bind 127.0.0.1:9042 --rpc-address 127.0.0.1 --data-center dc-1 --config proxy.yaml
cql-proxy ---astra-bundle astra-region2-bundle.zip --username token --password <astra-token> \\\n  --bind 127.0.0.2:9042 --rpc-address 127.0.0.2 --data-center dc-2 --config proxy.yaml

The peers settings are configured using a yaml file. It's a good idea to explicitly provide the --data-center flag, otherwise; these values are pulled from the backend cluster and would need to be pulled from the system.local and system.peers table to properly setup the peers data-center: values. Here's an example proxy.yaml:

peers:\n  - rpc-address: 127.0.0.1\n    data-center: dc-1\n  - rpc-address: 127.0.0.2\n    data-center: dc-2

Note: It's okay for the peers: to contain entries for the current proxy itself because they'll just be omitted.

There are three methods for using cql-proxy:

  1. \n

    Build cql-proxy.

    \n
    go build
    \n
    2. \n
  2. \n

    Run with your desired database.

    \n
    3. \n
  1. \n

    Run with your desired database.

    \n
    2. \n

If you wish to have the docker image removed after you are done with it, add --rm before the image name datastax/cql-proxy:v0.1.5.

Using Kubernetes with cql-proxy requires a number of steps:

  1. \n

    Generate a token following the Astra instructions. This step will display your Client ID, Client Secret, and Token; make sure you download the information for the next steps. Store the secure bundle in /tmp/scb.zip to match the example below.

    \n
    2. \n
  2. \n

    Create cql-proxy.yaml. You'll need to add three sets of information: arguments, volume mounts, and volumes. A full example can be found here.

    \n
    3. \n
  1. \n

    Create a configmap. Use the same secure bundle that was specified in the cql-proxy.yaml.

    \n
    kubectl create configmap cql-proxy-configmap --from-file /tmp/scb.zip
    \n
    2. \n
  2. \n

    Check the configmap that was created.

    \n
    kubectl describe configmap cql-proxy-configmap\n  Name:         cql-proxy-configmap\n  Namespace:    default\n  Labels:       <none>\n  Annotations:  <none>\n  Data\n  ====\n  BinaryData\n  ====\n  scb.zip: 12311 bytes
    \n
    3. \n
  3. \n

    Create a Kubernetes deployment with the YAML file you created:

    \n
    kubectl create -f cql-proxy.yaml
    \n
    4. \n
  4. \n

    Check the logs:

    \n
    kubectl logs <deployment-name>
    \n
    5. \n

Drivers that use token-aware load balancing may print a warning or may not work when using cql-proxy. Because cql-proxy abstracts the backend cluster as a single endpoint this doesn't always work well with token-aware drivers that expect there to be at least \"replication factor\" number of nodes in the cluster. Many drivers print a warning (which can be ignored) and fallback to something like round-robin, but other drivers might fail with an error. For the drivers that fail with an error it is required that they disable token-aware or configure the round-robin load balancing policy.

","id":"228f2fac-87de-5dab-b7ff-0e0c6d40dcbf","title":"GitHub - datastax/cql-proxy: A client-side CQL proxy/sidecar.","origin_url":"https://github.com/datastax/cql-proxy","url":"https://github.com/datastax/cql-proxy","wallabag_created_at":"2024-11-01T17:26:01+00:00","published_at":null,"published_by":"['datastax']","reading_time":8,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/c2528e3426d98910ed27819e048b4c1081fab2ed2c7adbea6e6a3b1872deb30a/datastax/cql-proxy","tags":["migration","proxy","cassandra","cql"],"description":" cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.The cql-proxy sidecar enables unsu..."},{"content":"
\n

Zero Downtime Migration (ZDM) Proxy is an open-source component developed in Go and based on client-server architecture. It enables you to migrate from one Apache Cassandra® cluster to another without downtime or code changes in the application client.

For details on ZDM Proxy, see zdm-proxy GitHub.

When using ZDM Proxy, the client connects to the proxy rather than to the source cluster. The proxy connects both to the source cluster and the target cluster. It sends read requests to the source cluster only, while write requests are forwarded to both clusters.

For details on how ZDM Proxy works, see Introduction to Zero Downtime Migration.

Connect to your Aiven for Apache Cassandra service using cqlsh, for example.

cqlsh --ssl-u avnadmin -p YOUR_SECRET_PASSWORD cassandra-target-cluster-name.a.avns.net 12345

You can expect to receive output similar to the following:

Connected to a1b2c3d4-1a2b-3c4d-5e6f-a1b2c3d4e5f6 at cassandra-target-cluster-name.a.avns.net:12345
[cqlsh 6.1.0 | Cassandra 4.0.11 | CQL spec 3.4.5 | Native protocol v5]

In your target service, create the same keyspaces and tables you have in your source Apache Cassandra cluster.

create keyspace KEYSPACE_NAME with replication ={'class':'SimpleStrategy', 'replication_factor':3};
create table KEYSPACE_NAME.TABLE_NAME (n_id int, value int, primary key (n_id));

Download the ZDM Proxy's binary from ZDM Proxy releases.

wget https://github.com/datastax/zdm-proxy/releases/download/v2.1.0/zdm-proxy-linux-amd64-v2.1.0.tgz
tar xf zdm-proxy-linux-amd64-v2.1.0.tgz

Check if the binary has been downloaded successfully using ls in the relevant directory. You can expect to receive output similar to the following:

LICENSE  zdm-proxy-linux-amd64-v2.1.0.tgz  zdm-proxy-v2.1.0
  1. \n

    Specify connection information by setting ZDM_TARGET_* and ZDM_ORIGIN_* environment variables using the export command.

    \n

    note

    ORIGIN refers to the source service.

    \n
    2. \n
  2. \n

    Run the binary.

    \n
    3. \n
exportZDM_ORIGIN_CONTACT_POINTS=localhost
exportZDM_ORIGIN_USERNAME=cassandra
exportZDM_ORIGIN_PASSWORD=cassandra
exportZDM_ORIGIN_PORT=1234
exportZDM_TARGET_CONTACT_POINTS=cassandra-target-cluster-name.a.avns.net
exportZDM_TARGET_USERNAME=avnadmin
exportZDM_TARGET_PASSWORD=YOUR_SECRET_PASSWORD
exportZDM_TARGET_PORT=12345
exportZDM_TARGET_TLS_SERVER_CA_PATH=\"/tmp/ca.pem\"
exportZDM_TARGET_ENABLE_HOST_ASSIGNMENT=false
# ZDM_ORIGIN_ENABLE_HOST_ASSIGNMENT=false  # (may be needed, see note)
./zdm-proxy-v2.1.0

ENABLE_HOST_ASSIGNMENT

Make sure you set the ZDM_TARGET_ENABLE_HOST_ASSIGNMENT variable. Otherwise, ZDM Proxy tries to connect to one of internal addresses of the cluster nodes, which are unavailable from outside. If this occurs to your source cluster, set ZDM_ORIGIN_ENABLE_HOST_ASSIGNMENT=false.

To connect to ZDM Proxy, use, for example, cqlsh. Provide connection details and, if your source or target require authentication, specify target username and password.

Check more details on using the credentials in Client application credentials.

The port that ZDM Proxy uses is 14002, which can be overridden.

  1. \n

    Connect using ZDM Proxy.

    \n
    cqlsh -u avnadmin -p YOUR_SECRET_PASSWORD localhost 14002
    \n

    You can expect to receive output similar to the following:

    \n
    Connected to CLUSTER_NAME at localhost:14002
    [cqlsh 6.1.0 | Cassandra 4.1.3 | CQL spec 3.4.6 | Native protocol v4]
    \n
    2. \n
  2. \n

    Check data in the table.

    \n
    select * from KEYSPACE_NAME.TABLE_NAME;
    \n

    You can expect to receive output similar to the following:

    \n
    n_id | value
    ------+-------
    1|42
    2|44
    3|46
    (3 rows)
    \n
    3. \n
  3. \n

    Insert more data into the table to test how ZDM Proxy handles write request.

    \n
    insert into KEYSPACE_NAME.TABLE_NAME (n_id, value) values (4, 48);
    insert into KEYSPACE_NAME.TABLE_NAME (n_id, value) values (5, 50);
    \n
    4. \n
  4. \n

    Check again data inside the table.

    \n
    select * from KEYSPACE_NAME.TABLE_NAME;
    \n

    You can expect to receive output similar to the following:

    \n
    n_id | value
    ------+-------
    5|50
    1|42
    2|44
    4|48
    3|46
    (5 rows)
    \n
    5. \n
  1. \n

    Connect to the source:

    \n
    cqlsh localhost 1234
    \n

    You can expect to receive output similar to the following:

    \n
    Connected to SOURCE_CLUSTER_NAME at localhost:1234
    [cqlsh 6.1.0 | Cassandra 4.1.3 | CQL spec 3.4.6 | Native protocol v5]
    \n
    2. \n
  2. \n

    Check data in the table:

    \n
    select * from KEYSPACE_NAME.TABLE_NAME;
    \n

    You can expect to receive output similar to the following:

    \n
    n_id | value
    ------+-------
    5|50
    1|42
    2|44
    4|48
    3|46
    (5 rows)
    \n

    ZDM Proxy has forwarded both the write request and the read request to the source cluster. As a result, all the values are there: both newly-added ones (50 and 48) and previously added ones (42, 44, and 46).

    \n
    3. \n
  1. \n

    Connect to the target service.

    \n
    cqlsh --ssl-u avnadmin -p YOUR_SECRET_PASSWORD cassandra-target-cluster-name.a.avns.net 12345
    \n

    You can expect to receive output similar to the following:

    \n
    Connected to a1b2c3d4-1a2b-3c4d-5e6f-a1b2c3d4e5f6 at cassandra-target-cluster-name.a.avns.net:12345
    [cqlsh 6.1.0 | Cassandra 4.0.11 | CQL spec 3.4.5 | Native protocol v5]
    \n
    2. \n
  2. \n

    Check data in the table.

    \n
    select * from KEYSPACE_NAME.TABLE_NAME;
    \n

    You can expect to receive output similar to the following:

    \n
    n_id | value
    ------+-------
    5|50
    4|48
    (2 rows)
    \n

    50 and 48 are there in the target table since ZDM Proxy has forwarded the write request to the target service. 42, 44, and 46 are not there since ZDM Proxy has not sent the read request to the target service.

    \n
    3. \n
","id":"c4220cd6-6736-5c34-870e-18564cff721d","title":"Migrate to Aiven for Apache Cassandra® with no downtime | Aiven docs","origin_url":"https://aiven.io/docs/products/cassandra/howto/zdm-proxy","url":"https://aiven.io/docs/products/cassandra/howto/zdm-proxy","wallabag_created_at":"2024-11-01T17:25:08+00:00","published_at":null,"published_by":null,"reading_time":4,"domain_name":"aiven.io","preview_picture":"https://aiven.io/docs/images/site-preview.png","tags":["migration","proxy","cassandra","aiven"],"description":"\nZero Downtime Migration (ZDM) Proxy is an open-source component developed in Go and based on client-server architecture. It enables you to migrate from one Apache Cassandra® cluster to another withou..."}]},{"tag":"performance","articles":[{"content":"
\"image\"

Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system used by the largest tech companies in the world. There are numerous reasons behind its popularity, including no single point of failure, exceptional horizontal scaling with a data layout designed as a perfect fit for time-series data.

However, despite these perks, like any other system, Cassandra is prone to performance issues. This makes monitoring imperative. And it all starts with knowing what to measure. In this article, we will explain the key Cassandra performance metrics you should monitor to make sure everything is up and running at all times.

What Is Cassandra and How Does It Work?

Let’s keep it short – Apache Cassandra is a distributed NoSQL database designed to provide fault-tolerant and highly available architecture with performance in mind.

As a distributed system Cassandra is built out of nodes. A node is a single instance of Apache Cassandra that can operate on its own. Multiple nodes can form a cluster – a distributed system holding common data and responding to query requests. Cassandra works in a master-less architecture where each node communicates in a peer to peer fashion using a protocol known as Gossip. The gossip protocol is designed so that each node is informed about the state of all other nodes and a single node performs gossip communication with up to three other nodes every second.

The cluster can be divided into data centers and racks, just like the real-life data centers are divided. In Cassandra terminology, a data center is designed to hold multiple racks and a single rack holds a complete replica of the data.

\"image\"

Cassandra Cluster Logical Overview

When it comes to the data, Cassandra stores it in tables that are organized the same way as in any other database – in rows and columns. A single table is called a column family. The tables themselves are grouped into keyspaces, where a keyspace usually holds logically similar data – for example, from a business perspective. The keyspace is also used for data replication, and the replication itself is configured on a keyspace level.

Getting back to the tables. Each table defines a primary key that is built of the partition key and the clustering columns. Cassandra uses the partition key to index the data. All data that share a common partition key make a single data partition – a basic unit for data retrieval and storage. The clustering columns are optional.

Needless to say, Apache Cassandra is a complicated, distributed system and it’s not uncommon for users to encounter operation problems and difficulties. Everything breaks eventually, from the low-level bare metal components, up to the high-level software. It is not unusual for users to deal with network issues and CPU utilization problems, especially on very large clusters. Cassandra is written in Java and uses both off-heap and heap memory, which means that as the volume of data grows, you may hit issues with the garbage collector. Finally, because of the amount of data that you will process you may need to deal with the hard disk space and performance of your I/O subsystem. All of these can be avoided by keeping an eye for the relevant metrics with the help of a good Cassandra monitoring tool.

How Is Cassandra Performance Measured?

The most complex, distributed systems provide a set of metrics that you should take care of, monitor, and alert on to ensure that your system is healthy and working well. Apache Cassandra is no different. It provides a plethora of performance metrics which we can divide into three categories:

  • Dedicated Apache Cassandra metrics that describe how the system and its parts perform.
  • Java Virtual Machine metrics that tell you about the execution environment on which Apache Cassandra is running.
  • Operating system metrics describing the metrics related to the bare metal servers, virtual machines, or containers, depending on the environment that you are using.

Dedicated Cassandra Performance Metrics

When monitoring Apache Cassandra clusters, is the metrics that the distributed data store exposes via the JMX interface. There are many Cassandra performance metrics exposed in the JMX and having visibility into most of them is a good idea. You never know what can be useful when troubleshooting.

Nodes

One of the most important Cassandra metrics is the number of nodes that are currently available and connected to form a cluster. The ability to store the data and respond to queries is directly related to the availability of nodes.

Compaction Metrics

Compaction is the operation of merging multiple smaller instances of SSTable into one bigger SSTable that contains all the data from the smaller tables. Because of that, it can be very expensive and resource-consuming. Having visibility into compaction performance is critical for long-term observability – the Cassandra monitoring tool of your choice needs to provide the number of compactions and the number of compacted bytes.

During compaction, until the process ends, the total disk space used may be double that before the compaction. Because of that, you should consider leaving about 50% of space free to account for compactions and, of course, set up appropriate alerts to inform you when the amount of free disk space is close to a level where compaction could fail.

Read and Write Performance Metrics

The next set of metrics is dedicated to clients and the read and write side of the operations. You should measure the number of reads happening in a given period, the request latency, and the number of timeouts and failures. Your Cassandra monitoring tool should provide the top-level view and allow for slicing and dicing through the data showing you the aggregated view, per node view, per keyspace view, and per table view. The same goes for write operations.

You should see the number of write requests happening and write latency. Local writes and reads may also be important when troubleshooting.

Table Metrics

Table metrics are also essential. The ones you should pay close attention to are partition size, tombstone scans, and the number of SSTables per read.

Partition Size

Partition size is crucial for cluster performance. Cassandra uses it as a unit of data storage, replication, and retrieval, thus directly dictating the performance of your Cassandra tables. The ideal partition size varies but is usually below 100MB and not less than 10 – 20MB.

Tombstones

Cassandra produces tombstones when you delete the data. They are markers of the deleted data. Data in Cassandra is immutable by design, and because of that, it can only be physically removed from the SSTable during compactions. Because of that, you should keep an eye on how they affect your disk space.

SSTables Per Read

Similar to tombstones, the number of SSTables per read is related to the immutability of the data in Cassandra. A single table can be built of multiple SSTables, which are written sequentially. A single read operation can result in reading multiple SSTables to retrieve the relevant data. The more SSTables Cassandra needs to read to return the data, the more resources are required to complete the read operation. This is why you should minimize the number whenever possible.

Other Metrics

As we mentioned earlier, other Apache Cassandra performance metrics can be helpful and you should consider monitoring them.

Caches

There are two types of caches in Cassandra – the key cache and the row cache. Cassandra uses the key cache to store the location of row keys in memory so that the rows can be accessed without the need to hit the disk. The row cache stores the rows themselves in memory. By using the caches, Cassandra reduces the need to read the data from the disk and trades the memory usage for performance.

You need to monitor the key cache requests and row cache requests, which tell how many requests to a given cache type were made, and the key cache hit ratio and the row cache hit ratio, which show the percentage of results retrieved from the cache instead of the disk.

Threadpool

Cassandra is designed to handle the high load, withstand backpressure, and perform asynchronous tasks. Monitoring various thread pools is crucial for understanding Cassandra’s performance and bottlenecks. Each thread pool exposes the number of active, pending, and blocked tasks. Accumulated, pending, and blocked tasks usually tell about performance issues and the need for more processing power or different data and query architecture.

Bloom Filter

In the read path, Cassandra merges the data stored on a disk inside the SSTables with the data stored in memory. To minimize the amount of checking for data existence in the SSTables on the disk Cassandra uses a data structure called bloom filter.

The bloom filter is a probabilistic data structure that can tell Cassandra that the data is definitely not in a given file or that the data may be present in a given file. The key metrics to monitor here are the amount of space used by bloom filters, the number of false positives, and the ratio. You can reduce the number of false positives by assigning more memory to the bloom filters.

Java Virtual Machine Metrics

Apache Cassandra is a JVM-based application that comes with all the usual JVM pros and cons. From the developer’s perspective, memory management is easier and requires less hassle – you just use an object and forget about it, letting the JVM do the cleaning up. But that means that something has to clean up all the unused objects in memory. This is where the Java Garbage Collection comes in and the metrics that come with it.

A proper Cassandra monitoring tool should provide metrics that allow you to check and troubleshoot issues with the Java Virtual Machine, such as JVM memory utilization and garbage collection count and time. You can read more about them in our guide about JVM metrics.

Operating System Metrics

You can’t ignore Operating System metrics either. Information such as CPU utilization, memory usage, and disk usage is essential and can play a major role when it comes to Cassandra performance.

CPU Utilization

Your CPU is used for data processing and query handling. The more spare CPU cycles you have on a given node, the data and queries it can process. The user part of the CPU usage will show you your Cassandra process needs, while the wait can point to a bottleneck in I/O or network. As with every Java application, CPU cycles are also needed for garbage collection, so keep that in mind when planning.

Memory Usage

Memory usage is crucial for every JVM-based application. The newest version of Cassandra leverages both off-heap and heap memory. This means that you not only need to set the heap size of your Cassandra nodes correctly but also have enough off-heap memory for keeping your cluster performance at its best.

Disk Usage

Disk and I/O are crucial – Cassandra keeps its data on the disk, and each query may require a substantial number of I/O operations to return the results. You need to be sure that your hardware can handle your data retrieval needs. You also need to be sure that you have enough space to hold your data and handle the compaction process.

Monitor Cassandra Performance with Sematext

\"monitoring

Sematext Cloud and its Apache Cassandra monitoring integration provide all that you need to monitor your distributed database. Everything is within a single view available without distractions:

  • The overview report gives you a perfect start point for your metrics, painting a picture of the whole cluster.
  • A dedicated Cassandra report that provides an in-depth view of all relevant metrics related to the distributed database.
  • The OS report provides necessary operating system metrics such as CPU and memory utilization and visibility into your network traffic.
  • Finally, the JVM metrics give the full view of the Java Virtual Machine, such as metrics related to garbage collection and per-heap space memory utilization.

Using the dedicated split-view, you can correlate all the available metrics with other metrics, logs, and real user monitoring data, making Sematext a perfect visibility tool.

Sematext allows you to set up alerts on any metric or log the event and supports both threshold-based and anomaly-based alerts for full flexibility. You don’t have to watch your metrics over and over. Once you configure your alerts, you can sleep well, and Sematext will let you know if something is wrong.

If you want to see how Sematext stacks against similar solutions, read our article about the best Cassandra monitoring tools available today.

Get Started with Cassandra Monitoring

As the distributed database Apache Cassandra can quickly become an operational challenge without visibility into what is happening from a global perspective as well as on a node level. You need to have full visibility from top to bottom, but that is not enough. You need to be sure that your monitoring system can notify you when an issue happens and also predict issues before your customers notice them.

One of the tools that will give you all of that is Sematext’s Apache Cassandra monitoring integration. String monitoring your Cassandra cluster by creating the Sematext Cloud account and then the Cassandra monitoring App. Don’t forget to create a Logs App as well to ship you Cassandra logs for a full observability experience.

You might also like

Products

Services

About

Contact

\n© Sematext Group. All rights reserved\n
Terms Of Service · Privacy Policy

\"Software\n\"Software\n\"GetApp\n\"Crozdesk\n\"Crozdesk\n\"Crozdesk
","id":"070dd491-a966-5c75-86e3-0327d7804b22","title":"How Do You Monitor Cassandra Performance: Key Metrics to Measure","origin_url":"https://sematext.com/blog/cassandra-monitoring/","url":"https://sematext.com/blog/cassandra-monitoring/","wallabag_created_at":"2021-11-08T17:30:27+00:00","published_at":"2021-10-04T10:55:25+00:00","published_by":"['Rafal Kuć']","reading_time":11,"domain_name":"sematext.com","preview_picture":"https://sematext.com/wp-content/uploads/2021/10/critical-cassandra-metrics-to-monitor.jpg","tags":["monitoring","cassandra","performance"],"description":"Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system..."},{"content":"

This is our third post in our series on performance tuning with Apache Cassandra. In our first post, we discussed how we can use Flame Graphs to visually diagnose performance problems. In our second post, we discussed JVM tuning, and how the different JVM settings can have an affect on different workloads.

In this post, we’ll dig into a table level setting which is usually overlooked: compression. Compression options can be specified when creating or altering a table, and it defaults to enabled if not specified. The default is great when working with write heavy workloads, but can become a problem on read heavy and mixed workloads.

\n

Before we get into optimizations, let’s take a step back to understand the basics of compression in Cassandra. Once we’ve built a foundation of knowledge, we’ll see how to apply it to real world workloads.

\n

How it works

\n

When we create a table in Cassandra, we can specify a variety of table options in addition to our fields. In addition to options such as using TWCS for our compaction strategy, specifying gc grace seconds, and caching options, we can also tell Cassandra how we want it to compress our data. If the compression option is not specified, LZ4Compressor will be used, which is known for it’s excellent performance and compression rate. In addition to the algorithm, we can specify our chunk_length_in_kb, which is the size of the uncompressed buffer we write our data to as an intermediate step before writing to disk. Here’s an example of a table using LZ4Compressor with 64KB chunk length:

\n
create table sensor_data ( \n    id text primary key, \n    data text) \nWITH compression = {'sstable_compression': 'LZ4Compressor', \n                    'chunk_length_kb': 64};\n
\n

We can examine how well compression is working at the table level by checking tablestats:

\n
$ bin/nodetool tablestats tlp_stress\nKeyspace : tlp_stress\n\tRead Count: 89766\n\tRead Latency: 0.18743983245326737 ms\n\tWrite Count: 8880859\n\tWrite Latency: 0.009023213069816781 ms\n\tPending Flushes: 0\n\t\tTable: sensor_data\n\t\tSSTable count: 5\n\t\tOld SSTable count: 0\n\t\tSpace used (live): 864131294\n\t\tSpace used (total): 864131294\n\t\tOff heap memory used (total): 2472433\n\t\tSSTable Compression Ratio: 0.8964684393508305\n\t\tCompression metadata off heap memory used: 140544\n
\n

The SSTable Compression Ratio line above tells us how effective compression is. Compression ratio is calculated by the following:

\n
compressionRatio = (double) compressed/uncompressed;\n
\n

meaning the smaller the number, the better the compression. In the above example our compressed data is taking up almost 90% of the original data, which isn’t particularly great.

\n

How data is written

\n

I’ve found digging into the codebase, profiling and working with a debugger to be the most effective way to learn how software works.

\n

When data is written to / read from SSTables, we’re not dealing with convenient typed objects, we’re dealing with streams of bytes. Our compressed data is written in the CompressedSequentialWriter class, which extends BufferedDataOutputStreamPlus. This writer uses a temporary buffer. When the data is written out to disk the buffer is compressed and some meta data about it is recorded to a CompressionInfo file. If there is more data than available space in the buffer, the buffer is written to, flushed, and the buffer starts fresh to be written to again (and perhaps flushed again). You can see this in org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.java:

\n
@Override\npublic void write(byte[] b, int off, int len) throws IOException\n{\n    if (b == null)\n        throw new NullPointerException();\n    // avoid int overflow\n    if (off < 0 || off > b.length || len < 0\n        || len > b.length - off)\n        throw new IndexOutOfBoundsException();\n    if (len == 0)\n        return;\n    int copied = 0;\n    while (copied < len)\n    {\n        if (buffer.hasRemaining())\n        {\n            int toCopy = Math.min(len - copied, buffer.remaining());\n            buffer.put(b, off + copied, toCopy);\n            copied += toCopy;\n        }\n        else\n        {\n            doFlush(len - copied);\n        }\n    }\n}\n
\n

The size of this buffer is determined by chunk_length_in_kb.

\n

How data is read

\n

The read path in Cassandra is (more or less) the opposite of the write path. We pull chunks out of SSTables, decompress them, and return them to the client. The full path is a little more complex - there’s a a ChunkCache (managed by caffeine) that we go through, but that’s beyond the scope of this post.

\n

During the read path, the entire chunk must be read and decompressed. We’re not able to selectively read only the bytes we need. The impact of this is that if we are using 4K chunks, we can get away with only reading 4K off disk. If we use 256KB chunks, we have to read the entire 256K. This might be fine for a handful of requests but when trying to maximize throughput we need to consider what happens when we have requests in the thousands per second. If we have to read 256KB off disk for ten thousand requests a second, we’re going to need to read 2.5GB per second off disk, and that can be an issue no matter what hardware we are using.

\n

What about page cache?

\n

Linux will automatically leverage any RAM that’s not being used by applications to keep recently accessed filesystem blocks in memory. We can see how much page cache we’re using by using the free tool:

\n
$ free -mhw\n              total        used        free      shared     buffers       cache   available\nMem:            62G        823M         48G        1.7M        261M         13G         61G\nSwap:          8.0G          0B        8.0G\n
\n

Page cache can be a massive benefit if you have a working data set that fits in memory. With smaller data sets this is incredibly useful, but Cassandra was built to solve big data problems. Typically that means having a lot more data than available RAM. If our working data set on each node is 2 TB, and we only have 20-30 GB of free RAM, it’s very possible we’ll serve almost none of our requests out of cache. Yikes.

\n

Ultimately, we need to ensure we use a chunk length that allows us to minimize our I/O. Larger chunks can compress better, giving us a smaller disk footprint, but we end up needing more hardware, so the space savings becomes meaningless for certain workloads. There’s no perfect setting that we can apply to every workload. Frequently, the most reads you do, the smaller the chunk size. Even this doesn’t apply uniformly; larger requests will hit more chunks, and will benefit from a larger chunk size.

\n

The Benchmarks

\n

Alright - enough with the details! We’re going to run a simple benchmark to test how Cassandra performs with a mix of read and write requets with a simple key value data model. We’ll be doing this using our stress tool, tlp-stress (commit 40cb2d28fde). We will get into the details of this stress tool in a later post - for now all we need to cover is that it includes a key value workload out of the box we can leverage here.

\n

For this test I installed Apache Cassandra 3.11.3 on an AWS c5d.4xlarge instance running Ubuntu 16.04 following the instructions on cassandra.apache.org, and updated all the system packages using apt-get upgrade. I’m only using a single node here in order to isolate the compression settings and not introduce noise from the network overhead of running a full cluster.

\n

The ephemeral NVMe disk is using XFS and mounted it at /var/lib/cassandra. I set readahead using blockdev --setra 0 /dev/nvme1n1 so we can see the impact that compression has on our disk requests and not hide it with page cache.

\n

For each workload, I put the following command in a shell script, and ran tlp-stress from a separate c5d.4xlarge instance (passing the chunk size as the first parameter):

\n
$ bin/tlp-stress run KeyValue -i 10B -p 10M --populate -t 4 \\\n  --replication \"{'class':'SimpleStrategy', 'replication_factor':1}\" \\\n  --field.keyvalue.value='book(100,200)' -r .5  \\\n  --compression \"{'chunk_length_in_kb': '$1', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}\" \\\n  --host 172.31.42.30\n
\n

This runs a key value workload across 10 million partitions (-p 10M), pre-populating the data (--populate), with 50% reads (-r .5), picking 100-200 words from of one of the books included in the stress tool (--field.keyvalue.value='book(100,200)'). We can specify a compression strategy using --compression.

\n

For the test I’ve used slightly modified Cassandra configuration files to reduce the effect of GC pauses by increasing the total heap (12GB) as well as the new gen (6GB). I spend a small amount of time on this as optimizing it perfectly isn’t necessary. I also set compaction throughput to 160.

\n

For the test, I monitored the JVM’s allocate rate using the Swiss Java Knife (sjk-plus) and disk / network / cpu usage with dstat.

\n

Default 64KB Chunk Size

\n

The first test used the default of 64KB chunk length. I started the stress command and walked away to play with my dog for a bit. When I came back, I was through about 35 million requests:

\n

\"stress

\n

You can see in the above screenshot our 5 minute rate is about 22K writes / second and 22K reads/ second. Looking at the output of dstat at this time, we can see we’re doing between 500 and 600MB / second of reads / second:

\n

\"DStat

\n

Memory allocation fluctuated a bit, but it hovered around 1GB/s:

\n

\"sjk

\n

Not the most amazing results in the world. Of the disk reads, some of that throughput can be attributed to compaction, which we’ll always have to contend with in the real world. That’s capped at 160MB/s, leaving around 400MB/s to handle reads. That’s a lot considering we’re only sending 25MB across the network. That means we’re doing over 15x the disk I/O than our network I/O. We are very much disk bound in this workload.

\n

4KB Chunk Size

\n

Let’s see if the 4KB chunk size does any better. Before the test I shut down Cassandra, cleared the data directory, and started things back up. I ran the same stress test above using the above shell script, passing 4 as the chunk size. I once again played fetch with my dog for a bit and came back after around the same time as the previous test.

\n

Looking at the stress output, it’s immediately obvious there’s a significant improvement:

\n

\"stress\"

\n

In almost every single metric reported by the metric library the test with 4KB outperforms the 64KB test. Our throughput is better (62K ops / second vs 44K ops / second in the 1 minute rate), and our p99 for reads is better (13ms vs 24ms).

\n

If we’re doing less I/O on each request, how does that impact our total disk and network I/O?

\n

\"dstat

\n

As you can see above, there’s a massive improvement. Disk I/O is significantly reduced from making smaller (but more) requests to disk, and our network I/O is significantly higher from responding to more requests.

\n

\"sjk

\n

It was initially a small surprise to see an increased heap allocation rate (because we’re reading WAY less data into memory), but this is simply the result of doing a lot more requests. There are a lot of objects created in order to satisfy a request; far more than the number created to read the data off disk. More requests results in higher allocation. We’d want to ensure those objects don’t make it into the Old Gen as we go through JVM tuning.

\n

Off Heap Memory Usage

\n

The final thing to consider here is off heap memory usage. Along side each compressed SSTable is compression metadata. The compression files have names like na-9-big-CompressionInfo.db. The compression metadata is stored in memory, off the Cassandra heap. The size of the offheap usage is directly proportional to the amount of chunks used. More chunks = more space used. More chunks are used when a smaller chunk size is used, hence more offheap memory is used to store the metadata for each chunk. It’s important to understand this trade off. A table using 4KB chunks will use 16 times the memory as one using 64KB chunks.

\n

In the example I used above the memory usage can be seen as follows:

\n
Compression metadata off heap memory used: 140544 \n
\n

Changing Existing Tables

\n

Now that you can see how a smaller chunk size can benefit read heavy and mixed workloads, it’s time to try it out. If you have a table you’d like to change the compression setting on, you can do the following at the cqlsh shell:

\n
cqlsh:tlp_stress> alter table keyvalue with compression = {'sstable_compression': 'LZ4Compressor', 'chunk_length_kb': 4};\n
\n

New SSTables that are written after this change is applied will use this setting, but existing SSTables won’t be rewritten automatically. Because of this, you shouldn’t expect an immediate performance difference after applying this setting. If you want to rewrite every SSTable immediately, you’ll need to do the following:

\n
nodetool upgradesstables -a tlp_stress keyvalue\n
\n

Conclusion

\n

The above is a single test demonstrating how a tuning compression settings can affect Cassandra performance in a significant way. Using out of the box settings for compression on read heavy or mixed workloads will almost certainly put unnecessary strain on your disk while hurting your read performance. I highly recommend taking the time to understand your workload and analyze your system resources to understand where your bottleneck is, as there is no absolute correct setting to use for every workload.

\n

Keep in mind the tradeoff between memory and chunk size as well. When working with a memory constrained environment it may seem tempting to use 4KB chunks everywhere, but it’s important to understand that it’ll use more memory. In these cases, it’s a good idea to start with smaller tables that are read from the most.

","id":"b16bdb40-b8c3-54e5-8afa-60ee43153f3d","title":"Apache Cassandra Performance Tuning - Compression with Mixed Workloads","origin_url":"https://thelastpickle.com/blog/2018/08/08/compression_performance.html","url":"https://thelastpickle.com/blog/2018/08/08/compression_performance.html","wallabag_created_at":"2019-12-02T18:51:11+00:00","published_at":null,"published_by":"['']","reading_time":11,"domain_name":"thelastpickle.com","preview_picture":"https://thelastpickle.com/android-chrome-192x192.png","tags":["cassandra","troubleshooting and tuning","performance"],"description":"This is our third post in our series on performance tuning with Apache Cassandra. In our first post, we discussed how we can use Flame Graphs to visually diagnose performance problems. In our second..."},{"content":"

There are multiple dimensions where Cassandra performance can be tuned. Some of them are described below:
Write Operations:
Commit log and data dirs (sstables) should be on different disks. Commit log uses sequential write however, if SSTables share the same drive with commit log , I/O contention between commit log & SSTables may deteriorate commit log writes and SSTable reads.
 
Read Operations:
A good rule of thumb is 4 concurrent_reads per processor core. May increase the value for systems with fast I/O storage.
 
Cassandra Compaction Contention:
Reduce the frequency of memtable flush by increasing the memtable size or preventing too pre-mature flushing. Less frequent memtable flush results in fewer SSTables files and less compaction. Fewer compaction reduces SSTables I/O contention, and therefore improves read operations. Bigger memtables absorb more overwrites for updates to the same keys, and therefore accommodating more read/write operations between each flushes.
 
Memory Cache:
Do not increase Cassandra cache size unless there is enough physical memory (RAM). Avoid memory swapping at any cost.
 
Row Cache:
The row cache holds the entire content of a row in memory. It provides data caching instead of reading data from the disk. good if column’s data is small so the cache is big enough to hold most of the hotspot data. Bad if column’s data is too large so the cache is not big enough to hold most of the hotspot data. It’s bad for high write/read ratios. By default, it is off. If hit ratio is below 30%, row cache should be disabled.
 
Key Cache Tuning:
The key cache holds the location of data in memory for each column family. Its Effective if there are hot data spot & cannot use row cache effectively because of the large column size. By default, Cassandra caches 200000 keys per column family. Use absolute number for keys_cached instead of percentage.
 
JVM:
Minimum and Maximum Java Heap Size should be half of available physical memory. Size of young generation heap should be 1/4 of Java Heap. Do NOT increase the size without confirming there are enough available physical memory- Always reserves memory for OS File cache.
A detailed understanding of Apache Cassandra is available in this blog post for your perusal!

","id":"2f1edaee-2329-5d58-a6d0-6dce016c28f4","title":"Tuning Cassandra Performance","origin_url":"https://intellipaat.com/blog/tutorial/cassandra-tutorial/tuning-cassandra-performance/","url":"https://intellipaat.com/blog/tutorial/cassandra-tutorial/tuning-cassandra-performance/","wallabag_created_at":"2019-12-02T18:50:11+00:00","published_at":"2019-09-05T08:00:00+00:00","published_by":"['']","reading_time":1,"domain_name":"intellipaat.com","preview_picture":"https://intellipaat.com/blog/wp-content/uploads/2020/09/Certification-in-Bigdata-Analytics-IITG.jpg","tags":["cassandra","troubleshooting and tuning","performance"],"description":"There are multiple dimensions where Cassandra performance can be tuned. Some of them are described below:Write Operations:Commit log and data dirs (sstables) should be on different disks. Commit log u..."},{"content":"

In this topic, i will  cover the basics of general Apache Cassandra performance tuning: when to do performance tuning, how to avoid and identify problems, and methodologies to improve.

When do you need to tune performance ?

optimizing:

when things work but could be better. we want to get better performance.

Troubleshooting:

fixing a problem that impact performance could actually be broken, could just be slow in clusters, something broken can manifest as slow performance.

What are some examples of performance related complaints an admin might receive regarding Cassandra ?

Performance-related complaints:

Latency, Throughput and the U.S.E Method:

Bad methodology
how not to approach performance-related problems?

streetlight anti-method\n\trandom change anti-method\n\tblame someone else anti-method

In performance tuning, what are we trying to improve ?

latency – how long a cluster,node,server or I/O subsystem takes to respond to a request
throughput – how many transactions of a given size (or range) a cluster,node or I/O subsystem can complete in a given timeframe ?
how many operations by seconds our cluster node are processing ?
But we can’t forget COST !!

How are latency and throughput related ?

theoretically, they are independent of each other.
however, change in latency can have a proportional effect on throughput

What causes a change in latency and throughput ?

Understanding performance tuning:utilization,saturation,errors,availability

What is utilization ? saturation ? errors ? availability ?

Utilization – how heavily are the resources being stressed.
Saturation – the degree to which a resource has queued work it cannot service
Errors – recoverable failure or exception events in the course normal operation
Availability – whether a given resource can service work or not

GOAL:

What is the first step in achieving any performance 
tuning goal ?
setting a goal!
what are some examples of commonly heard cassandra performance tuning goals?
reads should be faster
writes to table x should be faster
the cluster should be able to complete x transactions per second

what should a clearly defined performance goal take into account ?

Writing SLA (service level agreement)
we need to know :
what is the type of operation or query ?
read or write workload
select, insert,update or delete
we need to understand latency: expressed as percentile
rank e.g 95th percentile read latency is 2 ms
throughput: operations per second
size: expressed in average bytes
We have to think about duration: expressed in minutes or hours
scope: keyspace, table, query
Example of SLA: “the cluster should be able to sustain
20000 2KB read operations per second from table X for
two hours with a 95th percentile read latency of 3 ms.”

After setting a goal, how can achievement of a goal verified?

timing tooks in your application
query tracing
jmeter test plan
customizable cassandra-stress

Time in computer performance tuning:

how long is a millisecond ?
why do we care about milliseconds ?

Common latency timings in cassandra:

  reads from main memory should take between 36 and 130 microseconds
\n  reads from an SSD should take between 100 microseconds and 12 milliseconds
\n  reads from a Serial Attached SCSI rotational drive should take between 8 milliseconds and 40 milliseconds
\n  reads from a SATA rotational drive take more than 15 milliseconds
\n

Example:
Workload characterization:
classroom use case and cassandra story middle sized financial firm uses cassandra to manage distributed data 42 million stock quotes driven by a particular set of queries

what queries drove this data model ?

retrieve information for a specific stock trade by trade ID find all information about stock trades for a specific stock ticker and range timestamps find all information about stock trades that occurred on a specific date over a short period of time

How do you characterize the workload ?

what is the load being placed on your cluster
calling application or API
remote IP address

Who is causing the load ?

code path or stack trace

Why is the load being called ?

What are the load characteristics
throughput
Direction(read/write)
include variance
keyspace and column family
How do you characterize your workload ?
how is the load changing over time and is there
a daily pattern ?
is your workload read heavy or write heavy?
how big is your data ?
how much data on each node (bytes on node=data density)?
does active data fit in buffer cache ?

Performance impact of Data Model: How does the data model affect performance ?

poorly shaped rows (too narrow or too wide (we have partition to large)
hotspots (particular areas with a lot of reads/writes)
poor primary or secondary indexes
too many tombstones (lot of delete)

So data model considerations:

understand how primary key affects performance
take a look at query patterns and adjust how tables are modeled
see how replication factor and/or consistency level impact performance
change in compaction strategy can have a positive (or negative) impact
parallelize reads/writes if necessary
look at how moving infrequently accessed data can improve performance
see how per column family cache is having an impact
what is the relationship between the data model and cassandra’s read path optimizations (key/row cache,bloom filters, index) ?
nesting data (allows for greater degree of flexibility in the column family structure.) (keep all data to the same partition to satisfy a given query) but it can be easy to find model to keep most active data sets in cache, frequently accessed data which are in cache can improve performance.

Methodologies:

active performance tuning: we focus in the particular problem and we verify if it is fixed – suspect there’s a problem, isolate problem using tools, determine if problem is in cassandra ,environment or both.
verify problems and test for reproducibility,fix problems using tuning strategies, test, test and test again
Passive performance tuning : regular system “sanity checks”: looking some giving threshold,something we adjust for growth regularly monitor key health areas in cassandra/environment.
identify and tune for future growth/scalability.
apply tuning strategies as needed.
we have to use the USE Method as tool for troubleshooting
this method gives us the methodology to look for on all components of the system not only one. it is the strategy defined by Brendan Gregg http://www.brendangregg.com/USEmethod/use-linux.html.

Also performs a health check of various system components to identity bottlenecks and errors
separated by components, type and metric to narrow scope and find location of problem.

what are two things performance tuning to improve ? latency and throuput.
what are two types of performance tuning methodologies ? active and passive

what tool can be used to get a performance baseline ? jmeter or cassandra-stress

Cassandra-stress:

Interpreting the output of cassandra-stress

Each line reports data for the interval between the last elapsed time and current elapsed time, which is set by the –progress-interval option (default 10 seconds).

[hduser@base ~]$ cassandra-stress write -node 192.168.56.71 
\nINFO  04:11:53 Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
\nINFO  04:11:56 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
\nINFO  04:11:56 New Cassandra host /192.168.56.71:9042 added
\nINFO  04:11:56 New Cassandra host /192.168.56.72:9042 added
\nINFO  04:11:56 New Cassandra host /192.168.56.73:9042 added
\nINFO  04:11:56 New Cassandra host /192.168.56.74:9042 added
\nConnected to cluster: Training_Cluster
\nDatatacenter: datacenter1; Host: /192.168.56.71; Rack: rack1
\nDatatacenter: datacenter1; Host: /192.168.56.72; Rack: rack1
\nDatatacenter: datacenter1; Host: /192.168.56.73; Rack: rack1
\nDatatacenter: datacenter1; Host: /192.168.56.74; Rack: rack1
\nCreated keyspaces. Sleeping 1s for propagation.
\nSleeping 2s...
\nWarming up WRITE with 50000 iterations...
\nFailed to connect over JMX; not collecting these stats
\nWARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level analysis only
\nRunning with 4 threadCount
\nRunning WRITE with 4 threads until stderr of mean < 0.02
\nFailed to connect over JMX; not collecting these stats
\ntype,      total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
\ntotal,           402,     403,     403,     403,    10,4,     5,9,    31,4,    86,2,   132,7,   132,7,    1,0,  0,00000,      0,      0,       0,       0,       0,       0
\ntotal,           979,     482,     482,     482,     8,1,     6,3,    17,5,    45,9,    88,3,    88,3,    2,2,  0,06272,      0,      0,       0,       0,       0,       0
\ntotal,          1520,     530,     530,     530,     7,5,     6,4,    14,8,    24,3,   103,9,   103,9,    3,2,  0,07029,      0,      0,       0,       0,       0,       0
\ntotal,          1844,     321,     321,     321,    11,9,     6,4,    33,0,   234,5,   248,5,   248,5,    4,2,  0,06134,      0,      0,       0,       0,       0,       0
\ntotal,          2229,     360,     360,     360,    11,3,     5,4,    43,6,   127,6,   145,4,   145,4,    5,3,  0,06577,      0,      0,       0,       0,       0,       0
\ntotal,          2457,     199,     199,     199,    20,2,     5,9,    82,2,   125,4,   203,6,   203,6,    6,4,  0,11009,      0,      0,       0,       0,       0,       0
\ntotal,          2904,     443,     443,     443,     8,9,     7,0,    23,0,    37,3,    56,1,    56,1,    7,4,  0,09396,      0,      0,       0,       0,       0,       0
\ntotal,          3246,     340,     340,     340,    11,7,     7,4,    41,6,    87,0,   101,0,   101,0,    8,5,  0,08625,      0,      0,       0,       0,       0,       0
\ntotal,          3484,     235,     235,     235,    16,6,     7,2,    76,1,   151,8,   152,1,   152,1,    9,5,  0,09208,      0,      0,       0,       0,       0,       0
\ntotal,          3679,     196,     196,     196,    19,5,     8,1,    86,0,   156,5,   174,8,   174,8,   10,5,  0,09960,      0,      0,       0,       0,       0,       0
\ntotal,          4083,     369,     369,     369,    11,1,     7,3,    38,3,    90,8,   114,9,   114,9,   11,6,  0,09041,      0,      0,       0,       0,       0,       0
\ntotal,          4411,     320,     320,     320,    11,4,     7,4,    39,7,    61,1,    93,6,    93,6,   12,6,  0,08422,      0,      0,       0,       0,       0,       0
\ntotal,          4683,     227,     227,     227,    17,1,     6,0,    90,1,   153,3,   199,8,   199,8,   13,8,  0,08478,      0,      0,       0,       0,       0,       0
\ntotal,          5131,     445,     445,     445,     8,9,     7,6,    19,3,    26,8,    50,8,    50,8,   14,8,  0,07997,      0,      0,       0,       0,       0,       0
\ntotal,          5661,     521,     521,     521,     7,5,     5,4,    17,5,    63,1,    89,2,    89,2,   15,8,  0,07788,      0,      0,       0,       0,       0,       0
\ntotal,          6179,     512,     512,     512,     7,7,     6,2,    16,7,    39,3,    59,1,    59,1,   16,8,  0,07578,      0,      0,       0,       0,       0,       0
\ntotal,          6427,     245,     245,     245,    15,9,     6,1,    56,3,    94,0,   180,6,   180,6,   17,8,  0,07567,      0,      0,       0,       0,       0,       0
\ntotal,          6831,     394,     394,     394,    10,2,     5,6,    39,0,    90,4,   111,7,   111,7,   18,9,  0,07129,      0,      0,       0,       0,       0,       0
\ntotal,          7071,     235,     235,     235,    16,9,     7,4,    58,3,   149,7,   235,3,   235,3,   19,9,  0,07150,      0,      0,       0,       0,       0,       0
\ntotal,          7532,     455,     455,     455,     8,7,     6,1,    17,2,    90,5,   142,2,   142,2,   20,9,  0,06840,      0,      0,       0,       0,       0,       0
\ntotal,          7890,     353,     353,     353,    10,9,     7,3,    35,4,    80,6,   149,5,   149,5,   21,9,  0,06532,      0,      0,       0,       0,       0,       0
\ntotal,          8172,     288,     288,     288,    13,6,     7,1,    45,6,    89,1,    89,7,    89,7,   22,9,  0,06374,      0,      0,       0,       0,       0,       0
\ntotal,          8355,     171,     171,     171,    22,7,     9,1,    82,5,   137,7,   151,5,   151,5,   24,0,  0,06656,      0,      0,       0,       0,       0,       0
\ntotal,          8614,     235,     235,     235,    16,9,     8,0,    56,4,    98,7,   139,5,   139,5,   25,1,  0,06622,      0,      0,       0,       0,       0,       0
\ntotal,          9027,     402,     402,     402,     9,6,     8,5,    20,7,    26,5,    30,8,    30,8,   26,1,  0,06346,      0,      0,       0,       0,       0,       0
\ntotal,          9496,     463,     463,     463,     8,5,     7,6,    17,4,    23,2,    30,1,    30,1,   27,1,  0,06139,      0,      0,       0,       0,       0,       0
\ntotal,          9912,     408,     408,     408,     9,6,     7,5,    23,4,    33,0,    38,0,    38,0,   28,1,  0,05903,      0,      0,       0,       0,       0,       0
\ntotal,         10275,     359,     359,     359,    11,0,     8,8,    26,4,    33,7,    44,7,    44,7,   29,1,  0,05693,      0,      0,       0,       0,       0,       0
\ntotal,         10528,     251,     251,     251,    15,6,     9,5,    45,4,   176,1,   295,9,   295,9,   30,1,  0,05602,      0,      0,       0,       0,       0,       0
\ntotal,         10711,     181,     181,     181,    21,1,     7,8,    53,8,   340,8,   396,8,   396,8,   31,1,  0,05597,      0,      0,       0,       0,       0,       0
\ntotal,         10947,     233,     233,     233,    17,3,     9,3,    55,5,    94,7,   123,7,   123,7,   32,1,  0,05584,      0,      0,       0,       0,       0,       0
\ntotal,         11130,     177,     177,     177,    21,0,    10,6,    86,7,   115,6,   151,9,   151,9,   33,2,  0,05706,     
\n...
\n
\nResults:
\nop rate                   : 388 [WRITE:388]
\npartition rate            : 388 [WRITE:388]
\nrow rate                  : 388 [WRITE:388]
\nlatency mean              : 10,2 [WRITE:10,2]
\nlatency median            : 7,1 [WRITE:7,1]
\nlatency 95th percentile   : 25,9 [WRITE:25,9]
\nlatency 99th percentile   : 71,2 [WRITE:71,2]
\nlatency 99.9th percentile : 150,3 [WRITE:150,3]
\nlatency max               : 396,8 [WRITE:396,8]
\nTotal partitions          : 63058 [WRITE:63058]
\nTotal errors              : 0 [WRITE:0]
\ntotal gc count            : 0
\ntotal gc mb               : 0
\ntotal gc time (s)         : 0
\navg gc time(ms)           : NaN
\nstdev gc time(ms)         : 0
\nTotal operation time      : 00:02:42
\nSleeping for 15s
\n
\n
\nData\t                            Description
\n--------------------------------------------------------------------------------------------------
\ntotal\t                           :Total number of operations since the start of the test.
\ninterval_op_rate\t           :Number of operations performed per second during the interval 
\n(default 10 seconds).
\ninterval_key_rate\t           :Number of keys/rows read or written per second during the interval 
\n(normally be the same as interval_op_rate unless doing range slices).
\nlatency\tAverage latency            : for each operation during that interval.
\n95th                               :95% of the time the latency was less than the number displayed in the column 
\n(Cassandra 1.2 or later).
\n99th                               :99% of the time the latency was less than the number displayed in the column 
\n(Cassandra 1.2 or later).
\nelapsed\tNumber of seconds          :elapsed since the beginning of the test.
\n

Cassandra tuning:

successfull performance tuning is all about understanding.
understanding what some of metrics means.
how the software which we are tuning is architecture ?
in the distributed system we have to know how the software of one node
is working together with software of others nodes ?
that is for the next section come from :
how different pieces of cassandra can be put together ?

we can be talk in about cassandra, data model.
how we can get metric and how they can be performing.

The next section, we talk about environment tuning: JVM and operating system
The next section will be focussing on disk tuning and compaction tuning

Examine cluster and node health and tuning:

Discuss table design and tuning: successfull performance tuning is all about understanding.
understanding what kind of some metrics means ? how the software which we are
tuning is architecture ? with the distributed system, we have to how the software in one node works together with the software with others nodes?
some metrics we will expose
we are talking about cassandra, data model then we will talk about the environment tuning as the JVM and the operating system, also we will discuss disk tuning and compaction tuning

let’s talk about cluster and node tuning:what activities are in a cassandra cluster
happen between nodes ?
what happening on the network ? when we dig into some performance problem, we will look into all nodes not only one:
answers are: coordinator,gossip,replication,repair,read repair,bootstrapping,
node removal, node decommissioning.
we have to dig not only one node but additional node

Deeling in one node, whay does a cassandra node do ? read performance, write,
monitor,participate in the clusters, maintain consistency.
internally cassandra has lot of things to do, there is a architecture that
how is it organize ? how does cassandra organize all of that work ?
the answer of this is SEDA (staged event driven architecture), so we have
several thread pool and the messaging service to know how the queue works.

What is a thread pool ?
example, we have thread pool:workers:6

                                                                                              worker thread
\n  task          queue:max pending tasks:7                worker thread
\n  task task  task task  task  task task task task =>     worker thread 
\n  task                                task               worker thread
\n  blocked tasks
\n

what are cassandra’s thread pools ?

read readstage:32

write (mutationstage,flushwriter,memtablepostflusher,countermutation,
migrationstage)
monitor (memorymeter:1,tracing)
participate in cluster (requestresponsestage:%CPUs,pendingrangecalculator:1,gossipstage:1)
maintain consistency(commitlogarchiver:1,miscstage:1 snapshoting/replicate data after node remove)
antientropystage:1 repair consistency – merkle tree build,internal responsestage:#CPUs,HintedHanoff:1,readrepairstage:#CPUs)
we will know , for example the single thread on monitor (memorymeter:1)
which one has the configurable number of threads ? its read (readstage:32*)

the utility like nodetool tpstats which can give us metrics inside into , how much work each these pools are doing ? how many appending, active blocks:
active – number of messages pulled off the queue,currently being processed by a thread.
pending – number of messages in queue waiting for a thread.
completed – number of messages completed.
blocked -when a pool reaches its max thread count it will begin queuing.
until the max size is reached. when this is reached it will block until there is room in the queue.
total blocked/all time blocked – total number of messages that have been blocked.

cassandra thread pools: multi-threaded stages:
readstage (affected by main memory,disk) -perform local reads
mutationstage (affected by CPU,main memory,disk) – perform local insert/update,
schema merge,commit log replay,hints in progress)
requestresponsestage (affected by network,other nodes) – when a response to a request is received this is the stage used to execute any callbacks that were created with the original request.
flushwriter (affected by CPU,disk) -sort and write memtables to disk
hintedhandoo (one thread per host being sent hints,affected by disk,network,others nodes) –
sends missing mutations to others nodes
memorymeter (several separate threads) – measure memory usage and live ratio of a memtable.
readrepairstage (affected by network,others nodes) -perform read repair
countermutation (formerly replicateOnWriteStage) -performs counter writes on non-coordinator nodes and replicates after a local write
internalresponsestage -responds to non-client initiated messages, including bootstrapping and schema checking.

single-threaded stages:
gossipstage (affected by network) – gossip communication
antientropystage(affected by network,other nodes) – build merkle tree and repair consistency
migrationstage(affected by network,other nodes) – make schema changes
miscstage (affected by disk,network,pthers nodes) – snapshotting,replicating data after node remove
memtablepostflusher (affected by disk) -operations after flushing the memtable.
Discard commit log files that have all data in them persisted to sstables.
flush non-column family backed secondary indexes
tracing -for query tracing
commitlogarchiver (formally commitlog_archiver) -back up or restore a commit log.

Messages types:

handled by the readstage thread pool
(read: read data from cache or disk,range_slice: read a range of data,paged_range:
read part of a range of data)
)

handled by the mutationstage thread pool: (mutation:write (insert or update) data,
counter_mutation:changes counter columns,read_repair:update out-of-sync data discovered during a read)

handled by readresponsestage:(request_response:respond to a coordinator,_trace:
used to trace a query (enable tracing or every(trace probability) queries)
binary:deprecated

Question:why are some messages types “droppable” ?
if the message set in one of the queue to long and the to long will be decide by the timeout by the cassandra.yaml, it will be dropped.
why cassandra will do that ? why cassandra does not do my work ?
if the node has a suffisant resource to do the job, it does not dropped it.
in practice, if i have a read to give to cassandra and the timeout is elapsed and it timeout. what does the coordinator in this situation do ? it depends, maybe it can be satisfy by the consistency level to another node, then the consistency level gives the query or the return the timeout then back to the client if it is the to high consistency level.
In the read, there is not the deal, usally we can reexecute the query.

however we have another mutation (insert/delete/update) they can be to the queue to long (2 seconds) then no thing is done,
so what happen ? the coordinator can be experienced the
timeout, may be it store the hint, may be it can be back to the driver to be retry, we
don’t know, but the important thing some recovery actions can be taking in this point.
still that node who has a write doesn’t perform, that thing like read repair or repair command can back result to the client
why are some messages types “droppable”
some messages can be dropped to prioritize activities in cassandra when high resource contention occurs.
the number of these messages dropped is in the nodetool tpstats output
if you see dropped messages, you should investigate.
Question:What is the state of your cluster ? we have nodetool to gather information on our cluster or OpsCenter to gather information on our cluster.
nodetool gives us informations what happen right now, it is cummulatitve
but we do not know what happen last second,it is good for a general assessment
and also for comparaison between nodes (nodetool -h node0 tpstats, nodetool -h node0 compactionstats)

nodetool -h node0 netstats (we have to look for repair statistics) read repair
is when we are looking for discrepancy and if we find discrepancy between nodes, we will fixed by increasing cch (blocking), another read repair is the background, if we run the query probably consistency level 1 and we return no response, if we get a discrepancy. then we will increase the cch (background).

too wide: We have partition too large
too narrow:partition is to small and our query takes row on all bunch of partition
hotspots: we have a customer who decides to design partition by country code then 90% of all rows are in US and the rest on all others countries
poor primary and secondary indexes:
too many tombstones: we have a workload doing a lot of deleting and then reading data around the same partition, those things can have a huge impact on read performance.

OpsCenter

Changing logging levels with nodetool setlogginglevel:
nodetool getlogginglevels: use to get the current runtime logging levels

root@ds220:~# nodetool getlogginglevels
\nLogger Name                                                                 Log Level
\nROOT                                                                        INFO
\nDroppedAuditEventLogger                                                     INFO
\nSLF4JAuditWriter                                                            INFO
\ncom.cryptsoft                                                               OFF
\ncom.thinkaurelius.thrift                                                    ERROR
\norg.apache.lucene.index                                                     INFO
\norg.apache.solr.core.CassandraSolrConfig                                    WARN
\norg.apache.solr.core.RequestHandlers                                        WARN
\norg.apache.solr.core.SolrCore                                               WARN
\norg.apache.solr.handler.component                                           WARN
\norg.apache.solr.search.SolrIndexSearcher                                    WARN
\norg.apache.solr.update                                                      WARN
\n

nodetool setlogginglevel: used to set logging level for a service can be used instead of modifying the logback.xml file possible levels:

ALL
\nTRACE
\nDEBUG
\nINFO
\nWARN
\nERROR
\nOFF
\n

we can increase the level logging for example for namespace only to DEBUG or even TRACE

Data Model Tuning:

One off the key component of cassandra is the data model.
for performance tuning, we have the workload which place on the database. it is in the given place on cassandra table, that where the data model come .then we have software tuning (OS,JVM, Cassandra) and Hardware.
in many cases, updating hardware is an option.

When we need to diagnose the data model, cassandra provides many tools to know how our table is performing, how the query is performing.
Here is one nodetool cfstats, additionaly nodetool cfhistograms

when we identify the query which we may suspect, we have CQL tracing but we also have nodetool settraceprobability
nodetool cfstats example:
\"cfstats1\"
This one below is the aggregate number of the keyspace not for this particular table:
\"cfstats2\"
Then below the aggregate number of the table stock only:
\"cfstats3\"
Here we have 13 sstable on the disk for this table.
How much data we are store on the disk with the metrics space used? in this case we have live bytes and the total bytes is the same, that means in this particular table, we don’t have delete, ever update only insert ,however, if I do some delete or update, we will see the difference between live and total space used, the total number will be bigger than the live.
Off heap memory used (1928102) shows us how many memory this table are using. We have also other couple of metrics as bloom filter space used, bytes: 7088, and Bloom filter off heap memory used, bytes: 6984, we also have Index summary off heap memory used: 390, and compression metadata off heap memory used, bytes : 1920728, and memTable data size, bytes (36389573)
Conclusion: If I look at these numbers:
1928102 + 7088 + 6984 + 1920728 +36389573=40252475 bytes (39 MB)

I can know how much memory of ram are consuming by this table.

SSTable compression ratio: 0.173624: if I enable the compression like here. We have 17 % of compression of this table.
Number of keys: 1664 this is counting just the partition, so if we have a simple table schema , only have a partition key no cluster column , this number (1664) will be a rough number of rows on a given server.
These all metrics can be different on every server, so we have to go on each server to look at.

Memtable cell count (126342), memTable data size, bytes (36389573) and memTable switch count (107). MemTable data size (36389573) , this is a data which is storing in memory currently and they will go to flush on the disk. For memTable switch count, every time I flush memTable data to disk, this metric will increase. Memtable cell count indicates how many cell we have in memory.if we divide that by column count looking in schema definition (example 5 columns), 126342/5 = 25268,4, we can know approximativement the number of rows because Cassandra stores cell else the rows.

Local read count, local read latency, local write count and local write latency show us metrics read, write for this particular table. Local read count and local write count give us the indication to know if we have read heavy or write heavy. For our example, we have write heavy (4352651 greater than 942517).

Pending tasks: 0 this is counting any tasks pending, we saw that earlier on the tpstats.

Bloom filter false positives, bloom filter false ratio, bloom filter space used, bytes, bloom filter off heap memory used,bytes: the main metric here is the false ratio (0.0000) that takes the number of bloom filter false positives (48566) divided by the number of local read count (942517) , that can give us the false ratio. What did we pay to get 48566 of bloom filter false positives to get 942517 local read count? We paid these memory ram bloom filter space used, bytes: 7088 plus bloom filter off heap memory used, bytes: 6984, total of these memories: 7088 + 6984 =14072 bytes approximatively: 14 KB of RAM to get false positives to avoid unnecessary disk seek. If it is unacceptable, we can tune it by paying more memory and to get lower false positives and consequently less fewer unnecessary less disk seek, so read performance will go UP.

Index summary off heap memory used, bytes: 390: Is a memory structure we use to help us to jump to correct place in the partition index in memory. In this case we paid 390 bytes of memory that is tunable also but in Cassandra 2.1 we cannot adjust it automatically.

Compression metadata off heap memory used, bytes: 1920728 bytes, it is the amount of ram consume for compression. We can look at also.

Compacted partition minimum bytes, compacted partition maximum bytes, compacted partition mean bytes: very useful metrics for getting indication for how we have setup our partition. What I mean by that ? if we look at compacted partition minimum bytes (35426 bytes = 35 KB) and compacted partition mean bytes (122344216 bytes =116.68 MB) and we look at the maximum bytes (557074610 bytes =531,28 MB), we have the idea of the distribution of the size of the partition. If we look at the number of means bytes (122344216 bytes =116.68 MB), there is a huge discrepancy between the min and the max, probably we have the good indication of the hotspot of the data , the partition is abnormally much larger than the other partitions.

Average live cells per slice, average tombstones per slice: these metrics are only useful on production.
Average live cells per slice (last five minutes): 2.0: it has been reading on average 2.0 and 0.0 for tombstones, that good. If it is for example 2.0 for tombstones, that means for every 2.0 last five minutes for read, probably it pick at 2.0 tombstones that gives 50% over read performance to tombstones. Unfortunately, this number of tombstones cannot be the same. That the good indication that we have exception large delete of data.
For all these metrics we have some tunable.

Notes:
The bloom_filter_fp_chance and read_repair_chance control two different things. Usually you would leave them set to their default values, which should work well for most typical use cases.
bloom_filter_fp_chance: controls the precision of the bloom filter data for SSTables stored on disk. The bloom filter is kept in memory and when you do a read, Cassandra will check the bloom filters to see which SSTables might have data for the key you are reading. A bloom filter will often give false positives and when you actually read the SSTable, it turns out that the key does not exist in the SSTable and reading it waste the time. The better the precision used for the bloom filter, the fewer false positives it will give (but the more memory it will need).
From the documentation:
0 Enables the unmodified, effectively the largest possible, Bloom filter
1.0 Disables the Bloom Filter
The recommended setting is 0.1. A higher value yields diminishing returns.
So a higher number gives a higher chance of a false positive (fp) when reading the bloom filter.
read_repair_chance: controls the probability that a read of a key will be checked against the other replicas for that key. This is useful if your system has frequent downtime of the nodes resulting in data getting out of sync. If you do a lot of reads, then the read repair will slowly bring the data back into sync as you do reads without having to run a full repair on the nodes. Higher settings will cause more background read repairs and consume more resources, but would sync the data more quickly as you do reads.
See documentation on this link: http://docs.datastax.com/en/cql/3.1/cql/cql_reference/tabProp.html
You can read also this document:
http://www.datastax.com/dev/blog/common-mistakes-and-misconceptions
Here is the description of that table:
\"desc-table1\"
Below we have tunable parameters for this table:
\"desc-table2\"
These parameters influence directly the metrics we saw earlier.
Example:
bloom_filter_fp_chance=0.010000, that impact how many false positives we got and how much memory were are going to give up to low or high.
caching=’KEYS_ONLY’: In this case it is key only, we just go to cache only key but we have the ability to cache rows cache , so we will increase the memory utilization of this table, that also have the big impact.
dclocal_read_repair_chance=0.100000 and read_repair_chance=0.00000: these parameters influence the no blocking shows in tpstats.
gc_grace_seconds=864000: it impacts the tombstones, how long we can hold on the tombstones but it impacts the last line we saw earlier (average tombstones per slice (last five minutes), so it is tunable.
index_interval=128: in Cassandra 2.1, a max and min interval, those will impact index summary off memory used, bytes saw earlier. That is tunable.
Populate_io_cache_on_flush=’false’: in Cassandra 2.0 it allows us to populate if we want to flush to disk and say.
MemTable_flush_period_in_ms=0 if we want to flush on the disk by scheduling, most people do that.
Compression= {‘sstable_compression’:’LZ4compressor’}: that can be impact the compression ratio we saw early. If we want, we can tune off the compression. Why do we change that? May be we have the high compression ratio probably trading off CPU cycle, we can change it.
Speculative_retry=’99.0PERCENTILE’: this is not impacting in the tpstats output. When we are performing a read, we have additional replicat could be used to feel the request, Cassandra will wait for this long 99.0 % , if it is the quorum read each replicat will go to ask others replicat to get the data , Cassandra will wait this long 99.0 PERCENTILE in terme in milliseconds.

NODETOOL CFHISTOGRAMS:

Let‘s go to node0 cfhistograms: in Cassandra 2.0, it is pretty long, it gives us very fine bucket. Also we have a new format for percentile sstables writes latency ,read latency , partition size , cell count
\"cfhistogram1\"
\"cfhistogram2\"
How do we read us?
sstables per read:
\"cfhistogram3\"
31261 read will be satisfy by only one sstable but we have 240663 read operations will be satisfy by adding another sstable, that two disk seeks so the performance is there, then when we go down here, we have 397454 read operations by adding another sstables, we have three sstables, that three disk seeks. When we go, the slow the performance get, we have the performance problem,we need to get inside to know how it is performing.
Generally if we have this problem, it usually the function of compaction that getting behind. Compaction is the low priority task, it contains disk io, we don’t do that when we have a huge activity on the cluster.
Consequence: we have to read on the sstables to satisfy more read.
We see here on the five sstables we have 272481 read operations to satisfy read that is the problem.
Next is the write latency:
\"cfhistogram4\"
\"cfhistogram5\"
We saw in cfstats, local read latency: 0.000 ms and local write latency: 0.000 ms.
In write latency, us is the microseconds, here we see 2415947 writes operations will be completed in 50 us (microseconds) , this is the buck, my write operations completed in 50 us, so we have a long tail , then when we come down on write latency, the max was 126934 us:1
For the read latency, 50 us (microseconds) of read operations has 12976, this is a buck for read operations .this is proof that with Cassandra write latency is much much lower than read latency.
\"cfhistogram6\"
Another trend is the notion of two bumps:
Here is the first bump:
\"cfhistogram7\"
Second bump:
\"cfhistogram8\"
This is an indication of read from ram for the first bump and read from disk for the second bump.
Partition size:
So we have fine great output for partition size, these data generally is representative of cfstats by the metrics follow (compacted partition minimum bytes, compacted partition maximum bytes and compacted partition mean bytes).
We can see here we have one partition with 42Kb and at the end my largest has 25109160 bytes (24MB). 0, 1, 2 mean we create the bucket.
\"cfhistogram9\"

Cell count per partition:
Sometimes people have large volume of data in each cell this definitively influence number of partition size but we have a large data and the small number of cells (0, 1, 2) . Combining information below with which we saw on the partition size, we can see how the data is lay out on the term of data model. That is looking at the table schema

\"cfhistogram10\"
So this line means 3 of my partition have 4866323 cells

\"cfhistogram11\"

Quiz:
Which of the following information does CQL display for a query trace? Elapsed time
Which of the following components would you not find statistics for in a nodetool cfstats output? row cache
nodetool cfhistograms shows per-keyspace statistics for read and write latency? false only for the table.

Environment Tuning:

Some we have to tune others components. For example:
The most common bottlenecks in Cassandra? Performance checkpoint, things that slowing down the performance of the system.
The most common bottlenecks are:
Inadequate hardware.
Poorly configured JVM parameters: Cassandra runs on top on this virtual machine.
High CPU utilization: Particularly for write workload which we can see the Cassandra CPU bond, CPU pick..
Insufficient or incorrect memory cache tuning: This referring for read. This point is very outside of Cassandra.
I like to drive this point.
Question: What is the best way to cope with inadequate node hardware in a Cassandra cluster?
Lot of people ask, we have a machine then we want the performance then what can we do?
Upgrade the hardware. This is the bad new.
If we don’t get adequate equipment for the job, it is not a good idea to do the job.
We could say to add another node if the existing machine not performing.

What are some of the technologies upon which a Cassandra node depends?
Java, JVM, JNA, JMX and a bunch of others stuff that starts with “j”

JVM:

Cassandra is the java program, we will know how the jvm works, so we can tune it.
When we can for tuning, JVM has a lot options.
If we run this command:

java -XX:+PrintFlagsFinal

We will see all of the options.
JVM and Garbage Collection (GC): what is Garbage collection? Java gives the developer to allocate and deallocate memory. Java can complain all of the resources on the machine, lot of ram, CPU, IO.
JVM generational Heap:
When we start the java process, jvm will allocate a big chunk of ram and that is the chunk will be managed can be half. Inside that chunk this is a heap, it divides in several pieces, we have the new gen, old gen and perm gen. after java 8, perm gen change. What is does? It stores all of classe definitions.
Important thing that Cassandra boot load all code in most part of perm gen which could not change in size.

\"jvm1\"
With Cassandra, we can do anything in perm gen.
But in New gen, we have Eden and survival spaces. So, when we create a new objects any kind of objects, that object will be created in Eden. Let say we created it inside the function then it will be created in the Eden and as the space of eden fills Up with objects allocations that can the garbage collection kik send , that the first garbage collector.
This piece (new gen) is called parallel new.
Important thing is if Eden fills up, we need to look for garbage, we will find and drop all objects which will not references and deallocate them. However, there will be others objects which are always references, they can be move to survival area in New gen.
And after the survival, this process could be promoted to Old gen.
There are lot of options to do how many time the objects can be on Eden or to be move to survival and also to Old gen. Lot of options.
This are (old gen), can be fill up with the options.
Take a look at this.
I have in my test cluster, 4 nodes working in different virtual machine.

nodetool status
\n[hduser@base cassandra]$ nodetool status
\nDatacenter: datacenter1
\n=======================
\nStatus=Up/Down
\n|/ State=Normal/Leaving/Joining/Moving
\n--  Address        Load       Tokens       Owns    Host ID                               Rack
\nUN  192.168.56.72  40.21 MB   256          ?       5ddb3532-70de-47b3-a9ca-9a8c9a70b186  rack1
\nUN  192.168.56.73  50.88 MB   256          ?       ea5286bb-5b69-4ccc-b22c-474981a1f789  rack1
\nUN  192.168.56.74  48.63 MB   256          ?       158812a5-8adb-4bfb-9a56-3ec235e76547  rack1
\nUN  192.168.56.71  48.52 MB   256          ?       a42d792b-1620-4f41-8662-8e44c73c38d4  rack1
\n

Now we can do the command:

cassandra-stress write -node 192.168.56.71

Result:

INFO 23:56:42 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter 
\nname with DCAwareRoundRobinPolicy constructor) INFO 23:56:42 New Cassandra host /192.168.56.71:9042 added INFO 23:56:42 New Cassandra 
\nhost /192.168.56.72:9042 added INFO 23:56:42 New Cassandra host /192.168.56.73:9042 added INFO 23:56:42 New Cassandra host /192.168.56.74:9042 
\nadded Connected to cluster: Training_Cluster Datatacenter: datacenter1; Host: /192.168.56.71; Rack: rack1 Datatacenter: datacenter1; 
\nHost: /192.168.56.72; Rack: rack1 Datatacenter: datacenter1; Host: /192.168.56.73; Rack: rack1 Datatacenter: datacenter1; Host: /192.168.56.74; 
\nRack: rack1 Created keyspaces. Sleeping 1s for propagation. Sleeping 2s... Warming up WRITE with 50000 iterations... Failed to connect over JMX; 
\nnot collecting these stats WARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level 
\nanalysis only Running with 4 threadCount Running WRITE with 4 threads until stderr of mean < 0.02 Failed to connect over JMX; not collecting 
\nthese stats type, 
\ntotal ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb 
\ntotal, 2086, 2086, 2086, 2086, 1,9, 1,5, 4,2, 7,0, 46,4, 58,0, 1,0, 0,00000, 0, 0, 0, 0, 0, 0 
\ntotal, 4122, 2029, 2029, 2029, 2,0, 1,6, 4,8, 8,0, 14,0, 15,3, 2,0, 0,02617, 0, 0, 0, 0, 0, 0 
\ntotal, 6171, 2029, 2029, 2029, 1,9, 1,5, 5,1, 7,6, 12,0, 13,6, 3,0, 0,02038, 0, 0, 0, 0, 0, 0 
\ntotal, 8466, 2288, 2288, 2288, 1,7, 1,4, 4,2, 6,1, 11,9, 14,4, 4,0, 0,02715, 0, 0, 0, 0, 0, 0

We can run the program called jvisualvm
If we have the JDK installed this java visual vm
\"java1\"

We can see the available plugins on the Tools windows and activate some plugins like VisualVM-Glassfish, visual GC:
\"java2\"
We can see the Eden space:
\"java3\"
\"java4\"
In the long time, Cassandra recommended the CMS collector. In java 7, G1 exist and in java 8 it is very good.so depending of the version of Cassandra which you are running, it will be CMS or G 1.
CMS and G1 have both old generation and the permanent generation.
\"java5\"
The difference is in G1 we have several contigus chunk of memory like this:
\"java6\"
\"java7\"
G1 is very very well with very large heap. We generally recommend 8 GB heap for Cassandra.
When the old gen fill up we can have the capacity and we do a merging garbage collection, this pause time could last for second. That bring the pausis. What is the notion of pausis ?when we are doing the garbage collection in CMS or G1, some part of have to do the stop the work pause, that where they stop our program running at all and in eden and survivor space to find any unnecessary objects so we can clean them up.
How long this pause is last? is the function of couple of different things
How many objects are still in live?
The number of CPU available to the jvm is also the big determine. How long the garbage collection pause there
Additionally CMS offers one of the other thing called the heap fragmentation. Any way CMS defragment those is to do that the full stop the wall pause by the serial collector which is another garbage collection but it is single threaded. That the extreme long pause come from.
For G1 the only option we have is the target pause time and the minimum is 12 hundred milliseconds.
We have a couple of tools available:

1. java visual vm

2. OpsCenter
\"java8\"

3. Jconsole and jvisualvm

\"java9\"
4. And the last is jstat

jstat -gccause 1607 5000 (1607 is the process id for Cassandra)

Notes:

The most significant impact for Java virtual machine on Cassandra performance is Garbage collection.
The G1 collector is the preferred choice for garbage collection over CMS.
Metaspace does NOT exist in the new generation part of the JVM heap memory.

JVM Tools and Tuning Strategies:

If we have the server with 126 GB of RAM, it starts Cassandra by allocated 8 GB heap, what are it doing of the rest of the memory?
Page cache.
What is page caching? This is useful for Cassandra improving read performance. it can cache data that people are accessing frequently and get it up quicker than in the disk.

[hduser@base ~]$ free -m
\n             total       used       free     shared    buffers     cached
\nMem:          5935       4271       1664         13        104       1189
\n-/+ buffers/cache:       2977       2957 
\nSwap:         2047          0       2047 
\n

Here we have 5935 MB of ram but 4271 MB are used, so only 1189 MB are cached.

How does Cassandra utilize page caching?

It also make a write efficient but it cannot improve a lot write.
How do you triage root cause for out of memory (OOM) errors? If we don’t have enough memory.
It can be java errors not only Cassandra.
All of the challenge can be to buffering data in case of writing. It tries to write data in disk

Quiz:

What is the benefit of using the page cache? Reads are more efficient, Writes are more efficient, and Repairs are more efficient
Memory used by the page cache will not be available to other programs until explicity freed. ? true
Which of the following is most likely to cause an out-of-memory error? Client-side joins
CPU
CPU intensive:
Writes (INSERT, UPDATE, DELETE), encryption, compression, Garbage collection is CPU-intensive.
The more CPU which you can give to garbage is the faster garbage collection will run.
If you want to activate the compression or the encryption, you have to monitor the CPU utilization with the tools like dstat or opscenter.
What to do ?
Add nodes
Use nodes that have more and faster CPUs
However, if you have saturation of the CPU, we have a couple of options:
1. Turn off encryption, turn off compression
2. Add nodes
3. Alternatively upgrade theses nodes with more CPU
Quiz:
Which of the following operations would significantly benefit from faster CPUs? Writes and Garbage collection
OpsCenter is a tool that can be used to monitor CPU usage ? true
What would be the best course of action to resolve issues with CPU saturation? Add more nodes.

Disk Tuning:

In this section we are going to talk about disk tuning and compaction.
Question: How do disk considerations affect performance?
When we operate in the database like Cassandra where we have active dataset that have a large available RAM, disk. So, we have to take these things into account.
SSD: spinning, or rotation, disks must move a read/write mechanical head to the portion of the disk that is being written to or read.
Cassandra is architecture around rotation drive with sequential write and sequential read. However, SSD is very faster. If you have a latency application, SSD is crutial.

Some of the tuning in the Cassandra.yaml file that affect disk are:
Configuring disks in the Cassandra.yaml file:
1. Disk_failure_policy: what should occur if a data disk fails? Not for performance.
• By default, we do stop which Cassandra can detect some kinds of corruption, shut down gossip and thrift.
• Best-effort -stop using failed disk and respond using remaining sstables on others disks -obsolete data can be returned if the consistency level is one
• Ignore -ignore fatal errors and let requests fail
2. Commit_failure_policy: – What should occur if a commit log disk fails?
• Stop -same as above
• Stop_commit – shutdown commit log, let writes collect but continue to serve reads
• Ignore -ignore fatal errors and let batches fail
3. Concurrent_reads -typically set to 16 * number of drives: how many threads we can allocate for reads pool
4. Trickle_fsync -good to enable on SSDs but very bad for rotation drive. In SSD, it will do big flash

Tools to diagnose relative disks issue:

Using Linux sysstat tools to discover disk statistics:
System activity reporter (sar) -can get information about system buffer activity, system calls, block device, overall paging, semaphore and memory allocation, and CPU utilization.
Flags identify the item to check:

sar  -d for disk 
\nsar  -r for memory 
\nsar  -S for swap space used 
\nsar  -b for overall I/O activities 
\ndstat  -a Leatherman tool for Linux -versatile replacement 
\nfor sysstat (vmstat, iostat, netstat, nfsstat, and ifstat) 
\nwith colors
\nFlags identify the item to check:                      
\ndstat      -d             for disk 
\ndstat      -m             for memory, etc
\n

Can also string flags to get multiple stats:


\ndstat    -dmnrs
\n

Step to install dstat on RHEL/CentOS 5.x/6.x and fedora 16/17/18/19/20:
Installing RPMForge Repository in RHEL/CentOS
For RHEL/CentOS 6.x 64 Bit

sudo rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
\nsudo yum repolist
\nsudo yum install dstat
\n

If we want to use linux script to monitor our statistics, we can use cron jobs like:

sudo dstat  -drsmn --output /var/log/dstat.txt 5 3 >/dev/null
\n
#!/bin/bash
\ndstat -lrvn 10  --output /tmp/dstat.csv -CDN 30 360 mutt -a /tmp/dstat.csv  -s \"dstat report\"  me@me.com >/dev/null

Output can be displayed on webpage for monitoring
Output could be piped into graphical programs, like Gnumeric, Gnuplot, and Excel for visual displays.

Another tool is: nodetool cfhistograms to discover disk issues:

For SSD disk io disk performing, this tool can tell us something. The problem could be the JVM, the size of the RAM, there are a lot of thing it could be but disk is one of the things it could be.
With cfhistograms, there are two groups of bumps. It could be 1 of three things, usually we see two bumps.
One relative to read coming from RAM and big one coming from disk. Sometimes, it could be anything else like a lot of compaction. Compaction can cause disk contention and this contention can cause the read disk goes UP. because the compaction uses the disk by reading.
So, how do we deep that? How to fix it?
We have the utility Throttle down compaction by reducing the compaction_throughput_mb_per_sec.
Using nodetool proxyhistograms to discover disk issues: will show the full request latency recorded by the coordinator.

Using CQL TRACING :
To distinguish between slow disk response and slow query: slow disk response will be evident in how long it takes to access each drive.
If your queries need to look for SSTables on too many partitions to complete, you will see this in the trace
These issues will have different patterns.
Here we can see lot off informations like looking on the tombstoned, etc. We can know if our latency on the query is coming from the disk.and it shows us the source on which machine is experiencing those longer latencies.
Where is tracing information stored?
events table gives us lot of relative details for this particular query.
What role does disk readahead play in performance? We read a head a couple of blocks and that tunable, how many blocks to read a head? The problem is in Cassandra we don’t know exactly how much data we want to read a head
We recommend people to use readahead value of 8 for SSDs
Command to do that is:

blockdev -setra 8  
\n

QUIZ:

nodetool cfhistograms shows the full request latency recorded by the coordinator.false
Which of the following statements is NOT true about the readahead setting?
Which of the following tools can be used to monitor disk statistics? iostat, dstat, sysstat, sar.

Disk Tuning:

Compaction

How does compaction impact performance?

Compaction is the most io intensive operation in Cassandra cluster. So, some of the choice we make around is our compaction strategies and how we throllet, some things happen on the disk, how the big impact on the disk utilization in our cluster.
ction Strategy DTCS  for the time series data model.
Adjust compaction is to look at the impact of min/max SSTable thresholds.
Understand the connecUnderstand compaction strategies: there are 3 currently (SizeTiered Compaction Strategy STCS, DateTiered Compaction Strategy DTCS, Leveled Compaction Strategy LTCS)
For write and intensive workload, SizeTiered Compaction Strategy STCS can be the best.
Leveled Compaction Strategy is generaly recommended for read and workload only if we are using SSDs.
DateTiered Compation between number of SSTables and compaction as it affects performance
See what options are available for compaction to improve performance
How do tombstones affect compaction?
Compaction evicts tombstones and removes deleted data while consolidating multiple SSTables into one. More tombstones means more time spent during compaction of SSTables.
Once a column is marked with a tombstone, it will continue to exist until compaction permanently deletes the cell. Note that if a node is down longer than gc_grace_seconds and brought back online, it can result in replication of deleted data -zombie!
To prevent issues, repair must be done on every node on a regular basis.

Best practices: nodes should be repaired every 7 days when gc_grace_seconds is 10 days (the default setting)

How data modelling affects tombstones?

If a data model requires a lot of of deletes within a partition of data then a lot of tombstones are created. Tombstones identify stale data awaiting deletion – which data will have to be read until it is removed by compaction.
More effective data modelling will alleviate this issue. Ensure that your data model is more likely to delete whole partitions, rather than columns from a partition.
The data model has a significant impact on performance. Careful data modelling will avoid the pitfalls of rampant tombstones that affect read performance.
Tombstones are normal writes but will not otherwise affect write performance.
If you know you do a lot of delete you discover to do a long delete which will affect read performance we probably need to do something to fix it, example changing a data model or the way our workload use this data model.

Using nodetool compactionstats to investigate issues:

This tool can be used to discover compaction statistics while compaction occurs. Reports how much still needs compacting and the total amount of data getting compacted.
But by using CQL tracing to investigate issues, we can see how many nodes and partitions are accessed.
The number of tombstones will be shown.
The read access time can be observed as decreasing after a compaction is complete. It can also be seen to take longer while a compaction is in progress.
Why is a durable queue an anti-pattern that can cause compaction issues?
A lot people for another reason like to use Cassandra for durable queue because of this problem of tombstones, this is a use anti pattern. Generally what happen, only reading and delete on the same place then read performance can grow up. If you try to corrige a queue use a queue, use something like KAFKA that the perfect durable queue. Cassandra is not a tool to use as a queue.

How do disk choices affect compactions issues?

Compaction is the most disk IO with intensive operation of Cassandra performance.so having a good disks has a good positive affect on it. Conversely, have a slow disks can have a very detrimental effect on it. When you have compaction going very slow, you can increase SSTable for read then read performance can suffer.
Use nodetool cfhistograms to look at the read performance.
QUIZ:
Which of the following compaction strategies should be used for read-heavy workloads, assuming certain hardware conditions are met? Leveled Compaction
What tool, command, or setting can be used to investigate issues with tombstones? CQL tracing
Compaction can potentially utilize not only a significant amount of disk I/O, but also disk space as well.true

Disk Tuning: Easy Wins and Conclusion:

To end this paper, we need to revisit:
• Performance tuning methodology
• Outline easy performance tuning wins
• Outline Cassandra or environment anti-patters

How does this all fit together?

1. We need to understand performance and Cassandra at the high level. That is general performance tuning techniques and some of the terminology as well as how Cassandra itself works.
2. Collect performance data on the following things to know where to look for data and what that data means in term of tuning or isolation problem:
• workload and data model
• cluster and nodes
• operating system and hardware
• disk and compaction strategies.
3. Parsing the information gathered and begin formulating a plan:
• Based on metrics collected, where are the bottlenecks?
• What tools are available to fix issues that come up?
4. Apply solutions to any/all areas required and test solutions:
• Using tools and knowledge gained, apply solutions, test solutions applied and start cycle again as needed.

Question: What was that performance tuning methodology again?

We have:
Active performance tuning – suspect there’s a problem?
• Determine if problem is in Cassandra, environment or both.
• Isolate problem using tool provided.
• Verify problems and test for reproductibility.
• Fix problems using tuning strategies provided.
• Test, test, and test again.
• Verify that your “fixes” did not introduce additional problems.
Passive performance tuning – regular system “sanity checks”
• Regularly monitor key health areas in Cassandra / environment using tools provided.
• Identify and tune for future growth/scalability.
• Apply tuning strategies as needed.
• Periodically apply the USE Method for system health check.

Easy Cassandra performance tuning wins:

• Increase flushwriters, if blocked: Flushing memTable to sstables, if we look in nodetool tpstats tool ,we will see flushwriters regularly getting blocked and we only have one flushwriters which is common on system which has only one disk, we can increase it to 2 that can resolve the problem.
• Decrease concurrent compactors: we see lot of people to set their concurrent compactors to high, we recommend at 2 watch for CPU saturation. And if saturated, we can drop it to 1 which will make compaction single thread that is by default.
• Increase concurrent reads and writes appropriately: Write is very affecting by CPU then read is very affecting by disk (the kind of disks, the number of disks available) so adjust concurrent reads and writes appropriately.
• Nudge Cassandra to leverage OS cache to read based workloads: nudge means more RAM so the more we can read in RAM, better will be the performance.
• In cloud environment, sometimes we need to increase phi_convict_threshold for cloud deployments or those with bad network connectivity.
• Increase compaction_throughput if disk I/O is available and compactions are falling behind: take the default for 15 MB/s, if we have lot of disk IO available, just increase this, then the compaction will complete quickly.
• Increase streaming_throughput to increase the pace of streaming:the default is 200MB/s when we bring the node online and the last repair but if we want to bring the node online faster, this is the parameter which we can increase.
• In terme of tuning data model: if we have a time series data modelling, we can learn a lot of by reading the link: http://planetcassandra.org/blog/getting-started-with-time-series-data-modeling/
• Avoid creating more than 500 tables in Cassandra: if it is empty, these tables take at least 1MB of space on heap.
• Keep wide rows under 100 MB or 100000 columns: we remember the in memory compaction by default has 15 MB, that the reason, it is a bad idea to have wide rows below 100 MB.
• Leverage wide rows instead of collections for high granularity items: sometimes, people have partition that contains lot of list, for that, it is recommended to use clustering column.
• Avoid data modelling hotspots by choosing a partition key that ensures read/write workload is spread across cluster: try to find the right partition key, not to large partition key not to small.
• Avoid tombstone build up by leveraging append only techniques.
You can read also these documents about tombstones:
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html
http://www.jsravn.com/2015/05/13/cassandra-tombstones-collections.html
You can view this video about tombstones:
https://www.youtube.com/watch?v=olTsTxpBFqc&feature=youtu.be&t=270
• Use DESC sort to minimize impact of tombstones, if I do a descending sort by trying to get recent data.
You can read also this document:
http://www.sestevez.com/range-tombstones/
• Use inverted indexes to help where data duplication or nesting is not appropriate.
• Use DataStax drivers to ensure coordinator workload is spread evenly across cluster.
• Use the token Aware load balancing policy: allows you to go directly to the data of the data by avoiding to go to the coordinator.
• Use Prepared Statements (where appropriate), if you do a query a lot of time, you will see performance gain if you use that.
• Put OpsCenter’s database on a dedicated cluster.
• Size the cluster for peak anticipated workload: for example for load balancing or bench marking.
• Use a 10G network between nodes to avoid network bottlenecks.
• On JVM hand memery size, the number one is RAM and the CPU if we want Garbage Collection to run faster, so ensure there is adequate RAM to keep active data in memory.
• Understand how heap allocation affects performance.
• Look at how the key cache affects performance in memory.
• Understand bloom filters and their impact in memory: we can tune it on the table by the false positive bloom filter to eliminate unnecessary disk seek.
• Disable swap: swap can cause problem and it will be very very difficult to reproduce, we have to disable it by the command:

sudo swapoff –a

• Remove all swap files on /etc/fstab by:

sed -i 's/^\\(.*swap\\)/#\\1/'  /etc/fstab

• Look at the impact of memtables on performance: by default, memtables take records by heap then the more memtables we have, the more flushing them on the disk, then the more disk IO, so understanding, how many we have, how many they are flushing

What are some Cassandra/environment anti-patterns?

1. Network attached storage. Bottlenecks include: when you put Cassandra on the SAN, it like you are putting it on the top on storage system also Cassandra has sequential pattern of reads and writes. Advice is don’t use SAN plus it will be cheaper.
Use Cassandra on the SAN will increase network latency to all of operations.
• Router latency
• Network Interface Card (NIC)
• NIC in the NAS device
2. Shared network file systems.
3. Excessive heap space size: that can cause the JVM pause time very high because running it memory takes time.
• Can impair the JVM’s ability to perform fluid garbage collection.
4. Load balancers: don’t put load balancers between applications in Cassandra because Cassandra has the load balancer built into the drivers.
5. Queues and queue-like datasets: don’t use Cassandra like a queue.
• Deletes do not remove rows/columns immediately.
• Can cause overhead with RAM/disk because of tombstones.
• Can affect read performance if data not modelled well.

","id":"09f92fe1-9ee7-57a9-90d5-775990c3f999","title":"Cassandra Operations and Performance Tuning","origin_url":"https://dmngaya.wordpress.com/2015/10/10/cassandra-performance-and-tuning/","url":"https://dmngaya.wordpress.com/2015/10/10/cassandra-performance-and-tuning/","wallabag_created_at":"2019-12-02T18:42:19+00:00","published_at":"2015-10-10T07:44:25+00:00","published_by":"['']","reading_time":53,"domain_name":"dmngaya.wordpress.com","preview_picture":"https://i0.wp.com/dmngaya.wordpress.com/wp-content/uploads/2016/07/wallpaper-nature-hd-2.jpg?fit=1200%2C675&ssl=1","tags":["cassandra","troubleshooting and tuning","performance"],"description":"In this topic, i will  cover the basics of general Apache Cassandra performance tuning: when to do performance tuning, how to avoid and identify problems, and methodologies to improve.When do you need..."}]},{"tag":"spark","articles":[{"content":"
\"Arunkumar\"
3 min read
·
May 13, 2018

--

Why: We had a lot of very useful data in our Warehouse and wanted to take advantage of those data in some of our production service to enhance the user’s experience. So we choose to server them from Cassandra for all it’s pros which I’m am not going to get into in this blog.

First stage we went about writing a spark-cassandra exporter. It’s pretty simple and only a couple of line,

This works and took around ~ 30 mins to write ~150 Million rows. But once our services went live we saw the read latencies going a bit high during the bulk insertion time.

Latencies during Cassandra row writes

The spark-cassandra-connector that we are using here had a few configs that can be used to tune the writes here. Tried a bunch of tuning along the line of reducing concurrent and reducing throughput_mb_per_sec. They helped a bit but still there’s a clear increase in read latency.

Cassandra has sstableloader and we thought of testing it for this case. And so changed the code to use and saw that there’s barely any notable read latency during this task (only a slight increase in the 99 percentile, caused by the IO waits).

Latencies during Cassandra SSTable loads

Also if you see the networks graph, the traffic is only on “network in” as now we are generating SSTables in spark and then pushing those tables directly to cassandra. The last spike in below network graph is from SSTable method and the rest are from batched writes.

Network Traffic (Row writes vs SSTable load)

Now let’s get into how to do that in code,

  • Using CQLSSTableWriter build the SSTables per partition
  • We need to define the create and insert statements, but it’s easy to build that from the spark dataframe
  • And stream SSTable to Cassandra script. We pick a random Cassandra server and stream the SSTable to it. Host is chosen at random for a better load balancing of network traffic.
  • And finally the code that run’s it all,
  • As the no. of partitions Cassandra’s suggestion is several tens of megabytes large to minimize the cost of compacting, we use max of 256 MB per SSTable. “sizeInMB” can be calculated from HDFS.
  • Let say the size is 60GB, we will have 256 SSTables of size 256MB each.
  • Set this config “mapreduce.output.bulkoutputformat.streamthrottlembits” to throttle traffic to Cassandra.

Fyi,

  • SSTables has to be at-least several tens of megabytes in size to minimize the cost of compacting the partitions on the server side.
  • This methods increase IO wait since it’s writing directly to Disk and not memory like in Cassandra writes. Depending on the size of data and throughput, you need a SSD with high IOPS.

We’ve been using this method in production for over 6 months now, writing around ~ 300 million rows in < 30 mins without any issue to the read latencies.

Full example code can be found here, https://github.com/therako/sparkles/blob/master/src/main/scala/util/cassandra/SSTableExporter.scala

","id":"c019c512-0802-5940-8d36-d83e1db71e43","title":"Spark and Cassandra’s SSTable loader","origin_url":"https://medium.com/@rako/spark-and-cassandras-sstable-loader-ed55f6e67d17","url":"https://medium.com/@rako/spark-and-cassandras-sstable-loader-ed55f6e67d17","wallabag_created_at":"2024-11-01T17:13:45+00:00","published_at":"2018-06-08T01:43:43+00:00","published_by":"['Arunkumar']","reading_time":2,"domain_name":"medium.com","preview_picture":"https://miro.medium.com/v2/resize:fit:1200/1*bXczQ0OE6A9iB2X1yGOONA.png","tags":["sstable","cassandra","spark"],"description":"Arunkumar·Follow3 min read·May 13, 2018--Why: We had a lot of very useful data in our Warehouse and wanted to take advantage of those data in some of our production service to enhance the user’s exper..."},{"content":"

{{ message }}

\n

/ cassandra-analytics Public

Apache cassandra

cassandra.apache.org/

License

Apache-2.0 license

15 stars 11 forks Branches Tags Activity

Star

Notifications You must be signed in to change notification settings

\n

You can’t perform that action at this time.

\n\n
","id":"1268426a-7484-51b6-b1bf-0376998f5285","title":"GitHub - apache/cassandra-analytics: Apache cassandra","origin_url":"https://github.com/apache/cassandra-analytics","url":"https://github.com/apache/cassandra-analytics","wallabag_created_at":"2024-09-04T12:36:32+00:00","published_at":null,"published_by":"['apache']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/74cd461eac02c9222d5f3057f5e6023dab10987c0f9b29f0f7b59a8f6ed1cf6d/apache/cassandra-analytics","tags":["analytics","cassandra","spark"],"description":"{{ message }}\n / cassandra-analytics PublicNotifications You must be signed in to change notification settings\nFork 11\n\n Star 15\n\nApache cassandracassandra.apache.org/License Apache-2.0 license 15 sta..."},{"content":"

Author: Cédrick Lunven

\"DataStax\"
\"Building
Published in
·
9 min read
·
May 27, 2022

--

Knowing how to construct event-driven architectures is a crucial skill for developers as enterprises are relying on real-time data to drive business growth. In this post, we show you how to build a full event-driven toolkit with highly-scalable technologies like Apache Kafka™, Apache Spark™, and Apache Cassandra®.

Event-driven architectures (EDAs) are software patterns that enable organizations to detect “events”, significant changes in a state or an update, and respond to them in real-time or near real-time. In contrast to the traditional “request/response” architecture, applications built with EDAs provide faster response times, a more seamless user experience, and better scalability without blocked thread waiting.

DataStax recently collaborated with Rahul Singh, CEO of Anant Corporation and creator of Cassandra.Link, a knowledge base of all things Cassandra, to produce a 3-part series on building an Event Driven Toolkit for Apache Cassandra®, Apache Spark™, and Apache Kafka™.

In Part 1, we discussed how REST relates to event-driven systems and created a REST API for Cassandra. In Part 2, we explored different ways to event source information for a particular event with Kafka and connected it to Cassandra with Kafka Connect and Kafka Streams. In this post, we connect Kafka to Cassandra with Spark Streaming and process the data once it comes into Cassandra.

Kafka and Cassandra are a dynamic-duo in microservice architectures. Kafka fits naturally as a distributed queue for event-driven landscapes and acts as a buffer layer to transport messages to the database and surrounding technologies.

Cassandra scales linearly by just adding more nodes, making it an excellent persistent data storage choice for microservices applications. When combined with Spark Streaming, an equally scalable, high-throughput, and fault-tolerant streaming processing system, it creates a robust event-driven toolkit.

This content has been built before the integration of Apache Pulsar™, another queuing system in DataStax Astra DB, Cassandra-as-a-service platform. Everything that you’ll do in this post can be implemented in the same way with Pulsar.

By the end of this post, you’ll have a basic understanding of Spark Streaming in the context of Cassandra and Kafka, and be an expert at running streaming jobs and Spark batches. Ultimately, you’ll build a full event-driven data toolkit.

About Cassandra API for Leaves platform

Figure 1. Leaves Platform.

In the previous workshop, we created a Cassandra API GitPod for Anant Corporation’s Leaves platform which is used to generate their Cassandra.link website. Leaves is a knowledge-curation platform with an admin screen, a MySQL and PHP advanced view, and a mirror of the data in Cassandra and Solr.

Anant Corporation built the Cassandra API to make this platform scalable and serverless. The front-end of this application employs JAMstack and Netlify, a web hosting and automation platform to accelerate development productivity by hosting interfaces that can both talk to APIs but also generate a full website using Gatsby. All 1500+ pages on Casssandra.link get generated but run off an API.

The plan for Anant Corporation is to stop managing the API themselves and migrate it to Astra DB where they can take advantage of a wide range of APIs, such as API in GraphQL or API Ingress. Anant also wants to make the platform event driven. After data comes in event-driven and has been updated, they will process the data and analyze the Cassandra.link website to see correlations using machine learning.

There are currently two APIs–Cassandra.API, and a parity between Leaves API Python and Leaves API Node. Anant Corporation was one of the first users of Astra DB and back then, they needed to create another API separate from the Cassandra API to run some custom scrapping. Watch this YouTube video for a detailed breakdown of the two APIs.

If you didn’t catch the previous workshop, you can create the Cassandra API from scratch with this step-by-step guide or initialize the ready-made GitPod on GitHub. The first part of this series focused on using Kafka as a broker, a registry, and a REST proxy. Now, we’ll cover the stream process that transfers data from Kafka into Cassandra, and a batch process that reads from Cassandra, processes it, and puts it back into Cassandra using Spark Streaming.

Before we do, let’s understand more about how REST, microservices, and Event Driven Architectures relate to each other.

REST vs. Microservices vs. Event Driven Architecture

Microservices are loosely coupled services with a database for each service. This can be implemented synchronously using REST or asynchronously using AMQP protocol, Kafka, or Pulsar. Microservices run their services as an independent application, and these autonomous functions connect to REST APIs, which work to configure larger applications.

Generally, each microservice requires its own database to avoid any resource sharing and coupling between the services. But this isn’t true on Cassandra, because Cassandra can scale to hundreds of thousands of services and servers. A keyspace or a table on Cassandra works as its own kind of microservice that scales independently.

On Astra DB especially, you won’t have to worry about using a different database, keyspace, or table per microservice because it can scale infinitely. When you want to add new functionality, simply create another table instead of a whole database and have one cluster that contains several microservices powered by different keyspaces.

The DataStax Enterprise ecosystem also offers additional features, such as bringing data into Cassandra and retrieving it out as JSON or adding data using DSE Graph 6.8 and retrieving data using Cassandra Query Language (CQL).

Figure 2. How each keyspace on Cassandra works as a microservice.

In short, Cassandra is awesome for microservices because you can split your microservices by data center, key space, table, or query. When you execute a query against Cassandra, the load will be distributed among the nodes for you so there’s no coupling among different microservices. DataStax’s Software Engineer, Jeff Carpenter, explains this in more detail in this book.

Event sourcing and Command and Query Responsibility Segregation (CQRS) are software patterns that people implement with event-driven applications and microservices. CQRS is a method to scale systems so when an event or an update comes, the processor saves that data to different places–one for the event itself, and another for where the data’s going to be queried.

For example, if you were using DataStax but you don’t have a built-in search with DSE search, and you needed to materialize that data in both Cassandra and ElasticSearch, your event processor would take the event and save it in both places.

With CQRS, updating and retrieving data are seen as two different types of requests. CQRS uses commands to update data, and queries to read data. Figure 3 illustrates the architecture of an event-driven CQRS application. The commands write updates to Kafka in the corresponding topics while Kafka Streams creates projections via aggregations or joins of the data in the topics on the query side. Using event sourcing, we can send one event, process it, and save that data into several different places.

Figure 3. Architecture of a CQRS application. Source: IBM.

Before we get into the hands-on exercises, let’s learn more about the technologies that you’ll be working with.

What is Apache Spark?

Figure 4. Apache Spark ecosystem.

Apache Spark is a unified analytics engine built on top of Apache Spark Core. it includes a collection of technologies, such as:

  • Spark SQL: a hive-compliant language for structured data processing on Spark. Coupled with Spark Streaming, you can perform joins, and create batches with different events and queues with Spark SQL.
  • Spark Streaming: a scalable, high-throughput, fault-tolerant stream processing of life data streams. Spark Streaming transforms real-time data from various sources like Kafka, Flume, and Amazon Kinesis, using complex algorithms and delivers the processed data to file systems, databases, and live dashboards.
    There are two kinds of streaming–basic Spark Streaming and Structured Spark Streaming. Structured streaming gives you a schema so you can run Spark SQL transformations on the data coming in from the event before you send it through. You can also express your live datastream as a static table.
  • Machine Learning Library (MLlib): built-in library to make practical machine learning scalable and easy. There are MLlib extensions that you can run on top of Spark.
  • GraphX: an API for graphs and graph-parallel computation on top of Spark.

As a unified analytics engine, Spark can talk to any datastore that you’re looking to connect your data with. In other words, you can import data to Spark, export data to different systems, and present this data in a similar dataset format.

When properly configured, a single Spark job can run on a computer, or hundreds of computers. Due to this scalability, Spark fits in nicely as Cassandra and Kafka’s best friend in scalable data processing. On DataStax Enterprise, you can have both Kafka and Spark on the same node and scale both of them at the same time.

Figure 5 illustrates a Spark cluster architecture which consists of a driver called SparkConnect, a cluster manager that allocates resources, and different worker nodes called “Executor”, “Cache”, and “Tasks”. This YouTube video explains the architecture in detail.

Figure 5. Spark cluster architecture.

Spark is well-known for large-scale processing, machine learning, and analytics. Internet powerhouses like Uber, Netflix, eBay, and Coniva use it for the following:

  • Streaming Extract, Transform, Load (ETL)
  • Data enrichment
  • Trigger event detection
  • Complex session analysis
  • Machine learning
  • Fog computing

What is Astra DB?

Astra DB is a data platform as a service in the cloud built on the infinitely scalable Apache Cassandra with a selection of tools, such as APIs, to help you build applications on top of Cassandra. Astra eliminates operations and reduces deployment time from months to minutes as everything, from provisioning to backups, is fully automated. What’s more, you can instantly create a Cassandra database through Astra DB for free for 5 GB forever–no credit cards required.

Astra DB secures your data with the most advanced security available for Cassandra. Through auto-configured developer tools that you can deploy with a few clicks, Astra DB greatly simplifies app development.

Astra can run on any cloud and spin instances on any region you like. On top of the database, there are tools like REST GraphQL, CQL Console, DataStax Studio, and Data Loader. If you want to work with Kubernetes, our K8ssandra initiative on Astra DB is all you need to connect Cassandra with Kubernetes. With the service broker, you can tell K8ssandra to spin your Cassandra instance directly into Astra DB.

Hands-on exercise overview

Now that you’re familiar with the technologies, let’s get started on the hands-on workshop. You’ll first read data from Kafka in a structured stream, select information, and materialize a new dataset in Cassandra. Then, you’ll run a batch job to take all the data from Cassandra, crunch it, and save it back into another table.

Follow along with this YouTube tutorial, and get codes from this GitHub repository. You won’t need to download or run any codes on your computer; everything you need to run a Spark job is serverless. Click on the links below to get started!

  1. Create a Cassandra database on Astra DB
  2. Open Cassandra.API in GitPod
  3. Start and setup Apache Kafka
  4. Consume data from Kafka and write to Cassandra
  5. Run Apache Spark jobs against Astra DB

Conclusion

In this post we delved into the basics of the scalable and fault-tolerant event-driven architecture–Kafka, Cassandra, and Spark–and how you can process heavy real-time data and analyze them in databases and live dashboards.

Once you’ve mastered the basics, you can try posting your data on different analytics platforms. For more Cassandra and Kafka workshops, check out DataStax Developers on YouTube. If you have any questions about Cassandra, post them on our DataStax Community — the Cassandra stack overflow.

Follow the DataStax Tech Blog on Medium for more developer stories. Follow DataStax Developers on Twitter for the latest news about our developer community.

Resources

  1. Astra DB: the multi-cloud database-as-a-service
  2. Create your Cassandra database on Astra DB
  3. Apache Cassandra®
  4. Apache Spark™
  5. Apache Kafka™.
  6. Apache Pulsar™
  7. Anant Corporation
  8. Kafka Connect
  9. Kafka Streams
  10. Spark Streaming
  11. DataStax Enterprise
  12. DataStax Graph Documentation
  13. YouTube Workshop Part 1: Build a REST API with Apache Cassandra
  14. YouTube Workshop Part 2: Cassandra.API CRUD UI
  15. YouTube Workshop Part 3: Running a Spark Job on Apache Cassandra
  16. GitHub: Cassandra API
  17. GitHub: Cassandra in Real-Time
  18. Definitive Guide for Apache Cassandra 4.0
","id":"522aa199-ac8c-5bc5-80d2-7f365ffa6f9a","title":"Build an Event-Driven Architecture with Apache Kafka, Apache Spark, and Apache Cassandra","origin_url":"https://medium.com/building-the-open-data-stack/build-an-event-driven-architecture-with-apache-kafka-apache-spark-and-apache-cassandra-6f0fc0c87e42","url":"https://medium.com/building-the-open-data-stack/build-an-event-driven-architecture-with-apache-kafka-apache-spark-and-apache-cassandra-6f0fc0c87e42","wallabag_created_at":"2024-08-03T13:51:08+00:00","published_at":"2022-05-27T12:03:40+00:00","published_by":"['DataStax']","reading_time":10,"domain_name":"medium.com","preview_picture":"https://miro.medium.com/v2/da:true/resize:fit:1200/0*auI9qoRPiVH04PBH","tags":["cassandra","event.driven","spark","kafka"],"description":"Author: Cédrick LunvenDataStax·FollowPublished inBuilding Real-World, Real-Time AI·9 min read·May 27, 2022--Knowing how to construct event-driven architectures is a crucial skill for developers as ent..."},{"content":"

{{ message }}

\n

/ Data_streaming_project Public

Data streaming project with robust end-to-end pipeline, combining tools such as Airflow, Kafka, Spark, Cassandra and containerized solution to easy deployment.

0 stars 0 forks Activity

Star

Notifications

\n

You can’t perform that action at this time.

\n\n
","id":"1929ac10-cd1e-50bf-98ed-7526acc11fe4","title":"GitHub - andreia-negreira/Data_streaming_project: Data streaming project with robust end-to-end pipeline, combining tools such as Airflow, Kafka, Spark, Cassandra and containerized solution to easy deployment.","origin_url":"https://github.com/andreia-negreira/Data_streaming_project","url":"https://github.com/andreia-negreira/Data_streaming_project","wallabag_created_at":"2023-12-02T16:46:47+00:00","published_at":null,"published_by":"['andreia-negreira']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/ffade618f679782ffd42041fbd835f696d595d847bc87cdcc11ce3674094c81f/andreia-negreira/Data_streaming_project","tags":["python","cassandra","spark","airflow","kafka","postgres","docker"],"description":"{{ message }}\n / Data_streaming_project PublicNotifications\nFork 0\n\n Star 0 \n\nData streaming project with robust end-to-end pipeline, combining tools such as Airflow, Kafka, Spark, Cassandra and conta..."}]}]}},"staticQueryHashes":[]}