{"componentChunkName":"component---src-templates-article-single-page-js","path":"/post/monitoring-apache-cassandra-tm-made-simple","result":{"pageContext":{"obj_id":"344cc497-9d5b-5ea8-b392-f5271d516517","node":{"content":"

To learn more about the DataStax open-source project, Metric Collector for Apache Cassandra and to try a demo, visit us on GitHub.

Apache Cassandra is a resilient system for users to build applications on, but many operators see Cassandra as a bit of a black box. It’s not that Cassandra doesn’t have hundreds of metrics to consume, it does (over 300 metric series per table!). The fact is visualizing and getting a unified view of the cluster combined with OS-level metrics and application metrics is not an easy thing for Cassandra users to set up.  

What is the Metrics Collector for Apache Cassandra?

To help solve this problem, DataStax released a new open source project called the Metric Collector for Apache Cassandra (MCAC for short).  This project provides a drop-in solution to solve this monitoring gap for Apache Cassandra. Here’s how it works.

MCAC is built on the widely used collectd agent but with a novel twist. Collectd is a metric collection agent that is well adopted and integrates well with all kinds of external metrics systems like, prometheus, graphite, stackdriver, and others. While collectd can scrape JMX metrics out of the box, JMX scraping can be quite slow and works best with only a subset of metrics. Not to mention many people don’t want to maintain and configure the metric agent on every node.  

We use MCAC to power the health tab in Astra and is bundled with our Kubernetes operator for Apache Cassandra. 

Why MCAC is different

To solve this problem MCAC comes as a single bundle with our java agent and a linux portable collectd build all in one. Just add the agent to the cassandra-env.sh, it brings up collectd and ships every metric in Cassandra to collectd via a unix-socket. It works on all Apache Cassandra versions from 2.2 -> 4.0. 

By shipping the metrics this way efficiently it is able to export hundreds of thousands of series per node with little/no impact on C* performance.

Not only does it send the metrics, but it is specially designed to work well with prometheus out of the box, like histograms are tailored for aggregation by prometheus and labels are automatically converted on ingest. This means you can slice and dice metrics across DCs, racks, down to even tables.

The Cassandra metrics are one aspect of the equation but with collectd we can also gather and expose all the OS level metrics, like context switches and disk/network performance.

MCAC also creates a historical log on the nodes of metric and non-metric diagnostic events related to activity on the node. Non-metric events include details on Flushes, Compactions, Exceptions, GC, etc. This DataLog can be used to help analyze performance or other impacting issue on the cluster. If you need help our SRE team is available to help you diagnose problems with this log https://www.datastax.com/keepcalm and if you have any questions we're here to help at https://community.datastax.com/.

Finally, what good are all these metrics without a way to visualize them! To tie it all together, MCAC comes with pre-built grafana dashboards which give operators the best Cassandra monitoring solution out there. These dashboards will change over time to focus on specific aspects of the system to make it easier to drill into the cluster.

\"Grafana\"

\"mcac\"

\"mcac2\"

","id":"344cc497-9d5b-5ea8-b392-f5271d516517","title":"Monitoring Apache Cassandra™ Made Simple","origin_url":"https://www.datastax.com/blog/2020/05/monitoring-apache-cassandratm-made-simple","url":"https://www.datastax.com/blog/2020/05/monitoring-apache-cassandratm-made-simple","wallabag_created_at":"2020-05-20T21:17:41+00:00","published_at":null,"published_by":"['']","reading_time":2,"domain_name":"www.datastax.com","preview_picture":"https://www.ibm.com/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/d8/31/datastax-overview-leadspace.png/_jcr_content/renditions/cq5dam.thumbnail.1280.1280.png","tags":["monitoring","cassandra","grafana"],"description":"To learn more about the DataStax open-source project, Metric Collector for Apache Cassandra and to try a demo, visit us on GitHub.Apache Cassandra is a resilient system for users to build applications..."},"relatedArticles":[{"content":"

DataStax Mission Control currently focuses on the lifecycle management and observability of DSE clusters. Install DataStax Mission Control quickly, and then use it to create and manage DataStax Enterprise 6.8.26+ clusters. DataStax Mission Control is composed of a suite of operators which handle the orchestration of automation across regional cluster boundaries. This simplifies management of globally deployed DSE clusters. Define a DSE Cluster custom resource (CR) to create your first DSE Cluster with DataStax Mission Control.

","id":"8c9be501-0d76-5325-81f8-ae484d83122d","title":"DataStax Mission Control :: DataStax Project Mission Control","origin_url":"https://docs.datastax.com/en/mission-control/docs/index.html","url":"https://docs.datastax.com/en/mission-control/docs/index.html","wallabag_created_at":"2023-04-15T01:46:36+00:00","published_at":null,"published_by":null,"reading_time":null,"domain_name":"docs.datastax.com","preview_picture":"https://docs.datastax.com/en/_/img/datastax-docs-banner.png","tags":["kubernetes","datastax","cassandra","grafana"],"description":"DataStax Mission Control currently focuses on the lifecycle management and observability of DSE clusters. Install DataStax Mission Control quickly, and then use it to create and manage DataStax Enterp..."},{"content":"

Last week at the Cassandra Summit I gave a talk with Blake Eggleston on diagnosing performance problems in production. We spoke to about 300 people for about 25 minutes followed by a healthy Q&A session. I’ve expanded on our presentation to include a few extra tools, screenshots, and more clarity on our talking points.

There’s finally a lot of material available for someone looking to get started with Cassandra. There’s several introductory videos on YouTube by both me and Patrick McFadin as well as videos on time series data modeling. I’ve posted videos for my own project, cqlengine, (intro & advanced), and plenty more on the PlanetCassandra channel. There’s also a boatload of getting started material on PlanetCassandra written by Rebecca Mills.

This is the guide for what to do once you’ve built your application and you’re ready to put Cassandra in production. Whether you’ve been in operations for years or you are first getting started, this post should give you a good sense of what you need in order to address any issues you encounter.

The original slides are available via Slideshare.

Update: the presentation is now available on YouTube!

Before you even put your cluster under load, there’s a few things you can set up that will help you diagnose problems if they pop up.

  1. \n

    Ops center

    \n

    This is the standard management tool for Cassandra clusters. This is recommended for every cluster. While not open source, the community version is free. It gives you a high level overview of your cluster and provides historical metrics for the most important information. It comes with a variety of graphs that handle about 90% of what you need on a day to day basis.

    \n

    \"image\"

    \n
    2. \n
  2. \n

    Metrics plugins

    \n

    Cassandra has since version 1.1 included the metrics library. In every release it tracks more metrics using it. Why is this awesome? In previous persons of Cassandra, the standard way to access what was going on in the internals was over JMX, a very Java centric communications protocol. That meant writing a Java Agent, setting up mx4j, or Jolokia, then digging through JMX, which can be a little hairy. Not everyone wants to do this much work.

    \n

    The metrics library allows you to tell Cassandra to report its internal, table level metrics out to a whole slew of different places. Out to CSV, Ganglia, Graphite, and STDOUT, and it’s pluggable to push metrics to anywhere you want.

    \n

    \"image\"

    \n

    Read more about the metrics library integration.

    \n
    3. \n
  3. \n

    Munin, Nagios, Icinga (or other system metrics monitoring)

    \n

    I’ve found these tools to be incredibly useful at graphing system metrics as well as custom application metrics. There are many options. If you’re already familiar with one tool, you can probably keep using it. There are hosted solutions as well (server density, data dog, etc)

    \n
    \"image\"
    4. \n
  4. \n

    Statsd, Graphite, Grafana

    \n

    Your application should be tracking internal metrics. Timing queries, frequently called functions, etc. These tools let you get a profile of what’s going on with your code in production. Statsd collects raw stats and aggregates them together, then kicks them to graphite. Grafana is an optional (better) front end to Graphite.

    \n

    There was a great post by etsy, Measure Anything, Measure Everything, that introduced statsd and outlined its usage with Graphite.

    \n
    \"image\"
    5. \n
  5. \n

    Logstash

    \n

    We didn’t mention Logstash in our presentation, but we’ve found it to be incredibly useful in correlating application issues with other failures. This is useful for application logging aggregation. If you don’t want to host your own log analysis tool, there are hosted services for this as well.

    \n
    \"image\"
    6. \n

There’s a bunch of system tools that are useful if you’re logged onto a machine and want to see real time information.

  1. \n

    iostat

    \n

    iostat is useful for seeing what’s happening with each disk on your machine. If you’re hitting I/O issues, you’ll see it here. Specifically, you’re looking for high read & write rates and a big avgqu-sz (disk queue), or a high svctm (service time) there’s a good chance you’re bottlenecked on your disk. You either want to use more disks or faster disks. Cassandra loves SSDs.

    \n
    \"image\"
    2. \n
  2. \n

    htop

    \n

    Htop is a better version of top, which is useful for getting a quick glance at your system. It shows load, running processes, memory usage, and a bunch of other information at a quick glance.

    \n
    \"image\"
    3. \n
  3. \n

    iftop & netstat

    \n

    iftop is like top, but shows you active connections and the transfer rates between your server and whoever is at the other end.

    \n
    \"image\"

    Netstat is more of a networking swiss army knife. You can see network connections, routing tables, interface statistics, and a variety of other network information.

    \n
    \"image\"
    4. \n
  4. \n

    dstat

    \n

    I prefer to use dstat over iostat now since it includes all of its functionality and much of the functionality of other tools as well.

    \n
    \"image\"
    5. \n
  5. \n

    strace

    \n

    strace is useful when you want to know what system calls are happening for a given process.

    \n
    \"image\"
    6. \n
  6. \n

    pcstat

    \n

    This tool, written by Al Tobey, allows you to examine a bunch of files and quickly determine how much of each file is in the buffer cache. If you’re trying to figure out why table access is slow, this tool can tell you if your data is in cache already or if you have to go out to disk. Here’s a good read to get familiar with buffer cache. Check out the repo.

    \n
    \"image\"
    7. \n

There’s a few issues that are easy to run into that I’d consider “gotchas”, things that come up often enough that they’re worth mentioning.

A important design decision in Cassandra is that it uses last write wins when there are two inserts, updates, or deletes to a cell. To determine the last update, Cassandra uses the system clock (or the client can specify the time explicitly). If server times are different, the last write may not actually win, it’ll be the one that’s the most skewed into the future.

To address this issue, always make sure your clocks are synced. Ntpd will constantly correct for drift. ntpdate will perform a hard adjustment to your system clock. Ntpdate needs to be used if you clock is significantly off, and ntpd will keep it at the correct time.

\"image\"

Disk space not reclaimed

if you add new nodes to a cluster, each replica is responsible for less data. it’s streamed to the new nodes. however, it is not removed from the old nodes. If you’re adding new nodes because you’re running low on disk space, this is extremely important. You are required to run nodetool cleanup in order to reclaim that disk space. This is a good idea any time you change your database topology.

Issues adding nodes, or running repairs

There are two common problems that come up with repair. The first is that repairs take forever in 2.0. This is solved in 2.1 which uses an incremental repair, and does not repair data which has already been repaired. The second issue relates to trying to repair (or add nodes) to a cluster when the versions do not match. It is, in general, not a good idea (yet) to stream data between servers which are of different versions. It will appear to have started, but will just hang around doing nothing.

Cassandra comes with several tools to help diagnose performance problems. They are available via nodetool, Cassandra’s multipurpose administration tool.

Compaction

Compaction is the process of merging SSTables together. It reduces the number of seeks required to return a result. It’s a necessary part of Cassandra. If not configured correctly, it can be problematic. You can limit the I/O used by compaction by using nodetool setcompactionthroughput.

There’s 2 types of compaction available out of the box. Size Tiered is the default and great for write heavy workloads. Leveled compaction is good for read & update heavy workloads, but since it uses much higher I/O it’s recommended you use this only if you’re on SSD. I recommend reading through the documentation to understand more about which is right for your workload.

\"image\"

Histograms let you quickly understand at both a high level and table level what your performance looks like on a single node in your cluster. The first histogram, proxyhistograms, give you a quick top level view of all your tables on a node. This includes network latency. Histogram output has changed between versions to be more user friendly. The screenshot below is from Cassandra 2.1.

\"image\"

If you’d like to find out if you’ve got a performance problem isolated to a particular table, I suggest first running nodetool cfstats on a keyspace. You’ll be able to scan the list of tables and see if there’s any abnormalities. You’ll be able to quickly tell which tables are queried the most (both reads and writes).

\"image\"

nodetool cfhistograms lets you identify performance problems with a single table on a single node. The statistics are more easily read in Cassandra 2.1.

\"image\"

Query Tracing

If you’ve narrowed down your problem to a particular table, you can start to trace the queries that you execute. If you’re coming from a something like MySQL, you’re used to the command explain, which tells in in advance what the query plan is for a given query. Tracing takes a different approach. Instead of showing a query plan, query tracing keeps track of the events in the system whewn it actually executes. Here’s an example where we’ve created a whole bunch of tombstones on a partition. Even on a SSD you still want to avoid a lot of tombstones - it’s disk, CPU, and memory intensive.

\"image\"

The JVM gets a reputation for being a bit of a beast. It’s a really impressive feat of engineering, but it shouldn’t be regarded as black magic. I strongly recommend reading through Blake Eggleston’s post on the JVM, it’s well written and does a great job of explaining things. (Much better than I would here).

OK - we’ve got all these tools under our belt. Now we can start to narrow down the problem.

This started out as a small recap but has evolved into much more than that. The tools above have helped me a wide variety of problems, not just Cassandra ones. If you follow the above recommendations you should be in a great spot to diagnose most problems that come your way.

You can find me on Twitter for any comments or suggestions.

","id":"eef63404-0bd6-5a8d-96dc-2a4c8b270540","title":"Cassandra Summit Recap: Diagnosing Problems in Production - RustyRazorblade.com","origin_url":"http://rustyrazorblade.com/post/2014/2014-09-18-diagnosing-production/","url":"http://rustyrazorblade.com/post/2014/2014-09-18-diagnosing-production/","wallabag_created_at":"2023-03-01T19:46:02+00:00","published_at":null,"published_by":null,"reading_time":10,"domain_name":"rustyrazorblade.com","preview_picture":"http://rustyrazorblade.com/images/default.png","tags":["monitoring","cassandra"],"description":"Last week at the Cassandra Summit I gave a talk with Blake Eggleston on diagnosing performance problems in production. We spoke to about 300 people for about 25 minutes followed by a healthy Q&A sessi..."},{"content":"

{{ message }}

\n

/ cfstats-csv-parser Public

Repo for a utility to parse cfstats into a csv file for analysis

License

MIT license

1 star 4 forks

Star

Notifications

\n

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

\n
","id":"8e3a1b9c-7cd2-578f-a0b1-877ac6f47cb3","title":"GitHub - jlacefie/cfstats-csv-parser: Repo for a utility to parse cfstats into a csv file for analysis","origin_url":"https://github.com/jlacefie/cfstats-csv-parser","url":"https://github.com/jlacefie/cfstats-csv-parser","wallabag_created_at":"2023-03-01T19:40:38+00:00","published_at":null,"published_by":"['jlacefie']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/a44d5ad918f82ef2250d84b5d7d746b2e183f62eb7bd35a736db6977a6ff0bfb/jlacefie/cfstats-csv-parser","tags":["monitoring","cassandra"],"description":"{{ message }}\n / cfstats-csv-parser PublicNotifications\nFork 4\n\n Star 1 \n\nRepo for a utility to parse cfstats into a csv file for analysisLicense MIT license 1 star 4 forks Star Notifications\n\n © 202..."},{"content":"

In Apache Cassandra Lunch #62, guest speaker Sarma Pydipally presented on the Grafana Dashboard for Cassandra. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Register here now!

Grafana Dashboard for Apache Cassandra

We appreciate Sarma Pydipally for taking the time to create a presentation and sharing his knowledge of the Grafana Dashboard and how it can be used with Apache Cassandra.

Grafana Dashboard

Grafana Dashboards is an open-source data and analytics visualization tool. Some key features of Grafana include visualization, queries, alerts, and metrics exploration. It allows users to take data from their time-series databases (TSDBs) and create graphs and visualizations.

Prometheus

Prometheus is an open-source monitoring system. It records and tracks real-time metrics in a time series database. Prometheus supports flexible queries using PromQL and supports real-time alerting. It collects data through a pull model. The Prometheus server queries various data sources from a list at a set frequency and updates the current values based on these queries.

Cassandra

Apache Cassandra is a highly available and highly scalable, open-source, distributed NoSQL database. Cassandra features proven fault-tolerance on hardware or in the cloud.

Grafana Dashboard with Prometheus and Cassandra

To find out more about how to use Grafana Dashboard with Cassandra, please check out Sarma’s video and presentation embedded below. The GitHub repo to the demo featured in the presentation is linked below.

\"Grafana

Demo

https://github.com/sarma1807/Prometheus-Grafana-Cassandra

\n

Cassandra.Link

Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.

We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!

Posted in Data & Analytics, Events | Comments Off on Apache Cassandra Lunch #62: Grafana Dashboard for Apache Cassandra

","id":"efd6b1f7-3f4d-5a7e-8313-2984d4a821c7","title":"Apache Cassandra Lunch #62: Grafana Dashboard for Apache Cassandra - Business Platform Team","origin_url":"https://blog.anant.us/apache-cassandra-lunch-62-grafana-dashboard-for-apache-cassandra/","url":"https://blog.anant.us/apache-cassandra-lunch-62-grafana-dashboard-for-apache-cassandra/","wallabag_created_at":"2022-06-25T14:34:24+00:00","published_at":"2021-09-09T17:41:33+00:00","published_by":"['']","reading_time":1,"domain_name":"blog.anant.us","preview_picture":"https://blog.anant.us/wp-content/uploads/2021/08/SM_Webinar_Deck_Template-KEEP-12-1.png","tags":["cassandra.lunch","grafana.dashboard","cassandra","grafana"],"description":"In Apache Cassandra Lunch #62, guest speaker Sarma Pydipally presented on the Grafana Dashboard for Cassandra. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a de..."},{"content":"

Cassandra DataSource for Grafana

Apache Cassandra Datasource for Grafana. This datasource is to visualise time-series data stored in Cassandra/DSE, if you are looking for Cassandra metrics, you may need datastax/metric-collector-for-apache-cassandra instead.

\"Release\"CodeQL\"\"GitHub

To see the datasource in action, please follow the Quick Demo steps. Documentation is available here

Supports:

Contacts:

Usage

You can find more detailed instructions in the datasource wiki.

Installation

  1. Download the plugin using latest release, please download cassandra-datasource-VERSION.zip and uncompress a file into the Grafana plugins directory (grafana/plugins).
    2. \n
  2. Add the Cassandra DataSource as a datasource at the datasource configuration page.
    3. \n
  3. Configure the datasource specifying contact point and port like \"10.11.12.13:9042\", username and password. It's recommended to use a dedicated user with read-only permissions only to the table you have to access.
    4. \n
  4. Push the \"Save and Test\" button, if there is an error message, check the credentials and connection.
    5. \n

\"Datasource

Panel Setup

There are two ways to query data from Cassandra/DSE, Query Configurator and Query Editor. Configurator is easier to use but has limited capabilities, Editor is more powerful but requires understanding of CQL.

Query Configurator

\"Query

Query Configurator is the easiest way to query data. At first, enter the keyspace and table name, then pick proper columns. If keyspace and table names are given correctly, the datasource will suggest the column names automatically.

After that, you have to specify the ID Value, the particular ID of the data origin you want to show. You may need to enable \"ALLOW FILTERING\" although we recommend to avoid it.

Example Imagine you want to visualise reports of a temperature sensor installed in your smart home. Given the sensor reports its ID, time, location and temperature every minute, we create a table to store the data and put some values there:

CREATE TABLE IF NOT EXISTS temperature (\n    sensor_id uuid,\n    registered_at timestamp,\n    temperature int,\n    location text,\n    PRIMARY KEY ((sensor_id), registered_at)\n);\ninsert into temperature (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd0, 2020-04-01T11:21:59.001+0000, 18, \"kitchen\");\ninsert into temperature (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd0, 2020-04-01T11:22:59.001+0000, 19, \"kitchen\");\ninsert into temperature (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd0, 2020-04-01T11:23:59.001+0000, 20, \"kitchen\");\n

In this case, we have to fill the configurator fields the following way to get the results:

In case of a few origins (multiple sensors) you will need to add more rows. If your case is as simple as that, query configurator will be a good choice, otherwise please proceed to the query editor.

Query Editor

Query Editor is more powerful way to query data. To enable query editor, press \"toggle text edit mode\" button.

\"102781863-a8bd4b80-4398-11eb-8c28-4d06a1f29279\"

Query Editor unlocks all possibilities of CQL including Used-Defined Functions, aggregations etc.

Example using test_data.cql:

SELECT id, CAST(value as double), created_at FROM test.test WHERE id IN (99051fe9-6a9c-46c2-b949-38ef78858dd1, 99051fe9-6a9c-46c2-b949-38ef78858dd0) AND created_at > $__timeFrom and created_at < $__timeTo\n
  1. Follow the order of the SELECT expressions, it's important!
    2. \n
  1. To filter data by time, use $__timeFrom and $__timeTo placeholders as in the example. The datasource will replace them with time values from the panel. Notice It's important to add the placeholders otherwise query will try to fetch data for the whole period of time. Don't try to specify the timeframe on your own, just put the placeholders. It's grafana's job to specify time limits.
    2. \n

\"103153625-1fd85280-4792-11eb-9c00-085297802117\"

Development

Developer documentation

","id":"de9ff26a-f27a-57fb-b10d-bcbd676a6d9d","title":"Apache Cassandra","origin_url":"https://grafana.com/grafana/plugins/hadesarchitect-cassandra-datasource/","url":"https://grafana.com/grafana/plugins/hadesarchitect-cassandra-datasource/","wallabag_created_at":"2022-04-25T21:45:46+00:00","published_at":null,"published_by":null,"reading_time":4,"domain_name":"grafana.com","preview_picture":"https://grafana.com/media/images/meta/grafana-labs-meta-default_1200x630.png","tags":["cassandra","grafana"],"description":"Cassandra DataSource for GrafanaApache Cassandra Datasource for Grafana. This datasource is to visualise time-series data stored in Cassandra/DSE, if you are looking for Cassandra metrics, you may nee..."},{"content":"
\"image\"

Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system used by the largest tech companies in the world. There are numerous reasons behind its popularity, including no single point of failure, exceptional horizontal scaling with a data layout designed as a perfect fit for time-series data.

However, despite these perks, like any other system, Cassandra is prone to performance issues. This makes monitoring imperative. And it all starts with knowing what to measure. In this article, we will explain the key Cassandra performance metrics you should monitor to make sure everything is up and running at all times.

What Is Cassandra and How Does It Work?

Let’s keep it short – Apache Cassandra is a distributed NoSQL database designed to provide fault-tolerant and highly available architecture with performance in mind.

As a distributed system Cassandra is built out of nodes. A node is a single instance of Apache Cassandra that can operate on its own. Multiple nodes can form a cluster – a distributed system holding common data and responding to query requests. Cassandra works in a master-less architecture where each node communicates in a peer to peer fashion using a protocol known as Gossip. The gossip protocol is designed so that each node is informed about the state of all other nodes and a single node performs gossip communication with up to three other nodes every second.

The cluster can be divided into data centers and racks, just like the real-life data centers are divided. In Cassandra terminology, a data center is designed to hold multiple racks and a single rack holds a complete replica of the data.

\"image\"

Cassandra Cluster Logical Overview

When it comes to the data, Cassandra stores it in tables that are organized the same way as in any other database – in rows and columns. A single table is called a column family. The tables themselves are grouped into keyspaces, where a keyspace usually holds logically similar data – for example, from a business perspective. The keyspace is also used for data replication, and the replication itself is configured on a keyspace level.

Getting back to the tables. Each table defines a primary key that is built of the partition key and the clustering columns. Cassandra uses the partition key to index the data. All data that share a common partition key make a single data partition – a basic unit for data retrieval and storage. The clustering columns are optional.

Needless to say, Apache Cassandra is a complicated, distributed system and it’s not uncommon for users to encounter operation problems and difficulties. Everything breaks eventually, from the low-level bare metal components, up to the high-level software. It is not unusual for users to deal with network issues and CPU utilization problems, especially on very large clusters. Cassandra is written in Java and uses both off-heap and heap memory, which means that as the volume of data grows, you may hit issues with the garbage collector. Finally, because of the amount of data that you will process you may need to deal with the hard disk space and performance of your I/O subsystem. All of these can be avoided by keeping an eye for the relevant metrics with the help of a good Cassandra monitoring tool.

How Is Cassandra Performance Measured?

The most complex, distributed systems provide a set of metrics that you should take care of, monitor, and alert on to ensure that your system is healthy and working well. Apache Cassandra is no different. It provides a plethora of performance metrics which we can divide into three categories:

  • Dedicated Apache Cassandra metrics that describe how the system and its parts perform.
  • Java Virtual Machine metrics that tell you about the execution environment on which Apache Cassandra is running.
  • Operating system metrics describing the metrics related to the bare metal servers, virtual machines, or containers, depending on the environment that you are using.

Dedicated Cassandra Performance Metrics

When monitoring Apache Cassandra clusters, is the metrics that the distributed data store exposes via the JMX interface. There are many Cassandra performance metrics exposed in the JMX and having visibility into most of them is a good idea. You never know what can be useful when troubleshooting.

Nodes

One of the most important Cassandra metrics is the number of nodes that are currently available and connected to form a cluster. The ability to store the data and respond to queries is directly related to the availability of nodes.

Compaction Metrics

Compaction is the operation of merging multiple smaller instances of SSTable into one bigger SSTable that contains all the data from the smaller tables. Because of that, it can be very expensive and resource-consuming. Having visibility into compaction performance is critical for long-term observability – the Cassandra monitoring tool of your choice needs to provide the number of compactions and the number of compacted bytes.

During compaction, until the process ends, the total disk space used may be double that before the compaction. Because of that, you should consider leaving about 50% of space free to account for compactions and, of course, set up appropriate alerts to inform you when the amount of free disk space is close to a level where compaction could fail.

Read and Write Performance Metrics

The next set of metrics is dedicated to clients and the read and write side of the operations. You should measure the number of reads happening in a given period, the request latency, and the number of timeouts and failures. Your Cassandra monitoring tool should provide the top-level view and allow for slicing and dicing through the data showing you the aggregated view, per node view, per keyspace view, and per table view. The same goes for write operations.

You should see the number of write requests happening and write latency. Local writes and reads may also be important when troubleshooting.

Table Metrics

Table metrics are also essential. The ones you should pay close attention to are partition size, tombstone scans, and the number of SSTables per read.

Partition Size

Partition size is crucial for cluster performance. Cassandra uses it as a unit of data storage, replication, and retrieval, thus directly dictating the performance of your Cassandra tables. The ideal partition size varies but is usually below 100MB and not less than 10 – 20MB.

Tombstones

Cassandra produces tombstones when you delete the data. They are markers of the deleted data. Data in Cassandra is immutable by design, and because of that, it can only be physically removed from the SSTable during compactions. Because of that, you should keep an eye on how they affect your disk space.

SSTables Per Read

Similar to tombstones, the number of SSTables per read is related to the immutability of the data in Cassandra. A single table can be built of multiple SSTables, which are written sequentially. A single read operation can result in reading multiple SSTables to retrieve the relevant data. The more SSTables Cassandra needs to read to return the data, the more resources are required to complete the read operation. This is why you should minimize the number whenever possible.

Other Metrics

As we mentioned earlier, other Apache Cassandra performance metrics can be helpful and you should consider monitoring them.

Caches

There are two types of caches in Cassandra – the key cache and the row cache. Cassandra uses the key cache to store the location of row keys in memory so that the rows can be accessed without the need to hit the disk. The row cache stores the rows themselves in memory. By using the caches, Cassandra reduces the need to read the data from the disk and trades the memory usage for performance.

You need to monitor the key cache requests and row cache requests, which tell how many requests to a given cache type were made, and the key cache hit ratio and the row cache hit ratio, which show the percentage of results retrieved from the cache instead of the disk.

Threadpool

Cassandra is designed to handle the high load, withstand backpressure, and perform asynchronous tasks. Monitoring various thread pools is crucial for understanding Cassandra’s performance and bottlenecks. Each thread pool exposes the number of active, pending, and blocked tasks. Accumulated, pending, and blocked tasks usually tell about performance issues and the need for more processing power or different data and query architecture.

Bloom Filter

In the read path, Cassandra merges the data stored on a disk inside the SSTables with the data stored in memory. To minimize the amount of checking for data existence in the SSTables on the disk Cassandra uses a data structure called bloom filter.

The bloom filter is a probabilistic data structure that can tell Cassandra that the data is definitely not in a given file or that the data may be present in a given file. The key metrics to monitor here are the amount of space used by bloom filters, the number of false positives, and the ratio. You can reduce the number of false positives by assigning more memory to the bloom filters.

Java Virtual Machine Metrics

Apache Cassandra is a JVM-based application that comes with all the usual JVM pros and cons. From the developer’s perspective, memory management is easier and requires less hassle – you just use an object and forget about it, letting the JVM do the cleaning up. But that means that something has to clean up all the unused objects in memory. This is where the Java Garbage Collection comes in and the metrics that come with it.

A proper Cassandra monitoring tool should provide metrics that allow you to check and troubleshoot issues with the Java Virtual Machine, such as JVM memory utilization and garbage collection count and time. You can read more about them in our guide about JVM metrics.

Operating System Metrics

You can’t ignore Operating System metrics either. Information such as CPU utilization, memory usage, and disk usage is essential and can play a major role when it comes to Cassandra performance.

CPU Utilization

Your CPU is used for data processing and query handling. The more spare CPU cycles you have on a given node, the data and queries it can process. The user part of the CPU usage will show you your Cassandra process needs, while the wait can point to a bottleneck in I/O or network. As with every Java application, CPU cycles are also needed for garbage collection, so keep that in mind when planning.

Memory Usage

Memory usage is crucial for every JVM-based application. The newest version of Cassandra leverages both off-heap and heap memory. This means that you not only need to set the heap size of your Cassandra nodes correctly but also have enough off-heap memory for keeping your cluster performance at its best.

Disk Usage

Disk and I/O are crucial – Cassandra keeps its data on the disk, and each query may require a substantial number of I/O operations to return the results. You need to be sure that your hardware can handle your data retrieval needs. You also need to be sure that you have enough space to hold your data and handle the compaction process.

Monitor Cassandra Performance with Sematext

\"monitoring

Sematext Cloud and its Apache Cassandra monitoring integration provide all that you need to monitor your distributed database. Everything is within a single view available without distractions:

  • The overview report gives you a perfect start point for your metrics, painting a picture of the whole cluster.
  • A dedicated Cassandra report that provides an in-depth view of all relevant metrics related to the distributed database.
  • The OS report provides necessary operating system metrics such as CPU and memory utilization and visibility into your network traffic.
  • Finally, the JVM metrics give the full view of the Java Virtual Machine, such as metrics related to garbage collection and per-heap space memory utilization.

Using the dedicated split-view, you can correlate all the available metrics with other metrics, logs, and real user monitoring data, making Sematext a perfect visibility tool.

Sematext allows you to set up alerts on any metric or log the event and supports both threshold-based and anomaly-based alerts for full flexibility. You don’t have to watch your metrics over and over. Once you configure your alerts, you can sleep well, and Sematext will let you know if something is wrong.

If you want to see how Sematext stacks against similar solutions, read our article about the best Cassandra monitoring tools available today.

Get Started with Cassandra Monitoring

As the distributed database Apache Cassandra can quickly become an operational challenge without visibility into what is happening from a global perspective as well as on a node level. You need to have full visibility from top to bottom, but that is not enough. You need to be sure that your monitoring system can notify you when an issue happens and also predict issues before your customers notice them.

One of the tools that will give you all of that is Sematext’s Apache Cassandra monitoring integration. String monitoring your Cassandra cluster by creating the Sematext Cloud account and then the Cassandra monitoring App. Don’t forget to create a Logs App as well to ship you Cassandra logs for a full observability experience.

You might also like

Products

Services

About

Contact

\n© Sematext Group. All rights reserved\n
Terms Of Service · Privacy Policy

\"Software\n\"Software\n\"GetApp\n\"Crozdesk\n\"Crozdesk\n\"Crozdesk
","id":"d16c68f8-5520-5e6b-a126-62cd8cda87a2","title":"How Do You Monitor Cassandra Performance: Key Metrics to Measure","origin_url":"https://sematext.com/blog/cassandra-monitoring/","url":"https://sematext.com/blog/cassandra-monitoring/","wallabag_created_at":"2021-11-08T17:30:27+00:00","published_at":"2021-10-04T10:55:25+00:00","published_by":"['Rafal Kuć']","reading_time":11,"domain_name":"sematext.com","preview_picture":"https://sematext.com/wp-content/uploads/2021/10/critical-cassandra-metrics-to-monitor.jpg","tags":["monitoring","cassandra","performance"],"description":"Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system..."},{"content":"

\"CassPromGraf_00_Arch.jpg\"

Environment

Following servers are running with CentOS Linux release 7.8.2003 (Core) :
192.168.1.151      eternal1      eternal1.OracleByExample.com\n192.168.1.152      eternal2      eternal2.OracleByExample.com\n192.168.1.153      eternal3      eternal3.OracleByExample.com\n192.168.1.191      PromGraf      PromGraf.OracleByExample.com\n

Apache Cassandra

3 node Cassandra cluster cluster_name: 'id_cluster' version : Apache Cassandra 4.0-beta4 running on following servers :

192.168.1.151      eternal1      eternal1.OracleByExample.com\n192.168.1.152      eternal2      eternal2.OracleByExample.com\n192.168.1.153      eternal3      eternal3.OracleByExample.com\n

We will configure Cassandra's built in metrics-reporter to extract and publish metrics to Prometheus.
We will also configure Prometheus node_exporter on each of our Cassandra nodes to extract and publish metrics to Prometheus.\n

Prometheus & Grafana

We will configure and run Prometheus & Grafana on following server :

192.168.1.191      PromGraf      PromGraf.OracleByExample.com\n

Prometheus will gather and organize all collected metrics into its internal time-series database.
Grafana will consume the metrics from Prometheus and display them in a nice dashboard.

Prometheus Alertmanager

In future, we should implement Alertmanager\n

","id":"3d846338-4870-59b3-8a50-8ba8bf31966d","title":"sarma1807/Prometheus-Grafana-Cassandra","origin_url":"https://github.com/sarma1807/Prometheus-Grafana-Cassandra","url":"https://github.com/sarma1807/Prometheus-Grafana-Cassandra","wallabag_created_at":"2021-07-10T11:53:50+00:00","published_at":null,"published_by":"['sarma1807']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/b79fbf32a62a4dfb02f40b3c4144d102b6d471b6f61ae0f8af63847ce6ea5fbf/sarma1807/Prometheus-Grafana-Cassandra","tags":["cassandra","grafana","prometheus"],"description":" EnvironmentFollowing servers are running with CentOS Linux release 7.8.2003 (Core) :192.168.1.151 eternal1 eternal1.OracleByExample.com\n192.168.1.152 eternal2 eternal2.OracleByExa..."},{"content":"

This blog post is a writeup of the presentation Bartek Plotka and I gave at \nPromCon 2019.

Cortex is a horizontally scalable, clustered \nPrometheus implementation aimed at giving users a global view of all their Prometheus metrics in one place, and providing long term storage for those metrics. Thanos is newer project aimed at solving the same challenges. In this blog post, we compare these two projects and see how it is possible to have two completely different approaches to the same problems.

The Cortex Project was started in 2016 by Julius Volz and me, and joined the CNCF sandbox in 2018. Thanos was started in 2017 by Fabian Reinartz and Bartłomiej Płotka, and joined the CNCF sandbox in 2019. Both are written in Go, \nhosted on \nGithub, and re-use large swathes of Prometheus codebase. Both projects aim to solve the same problems: a global view of all your metrics, highly available monitoring with gaps, and long term storage.

1. Global View: Queries over Data from Multiple Prometheus Servers

Prometheus is a pull-based monitoring system; as such, Prometheus servers need to be colocated with the jobs and infrastructure they are monitoring. This works extremely well when your application is deployed in one region; when you start deploying in multiple regions, you need Prometheus servers in each region. And when you want to run queries that cover data in both regions, you can’t. This is the first problem that Cortex and Thanos set out to solve.

Thanos reuses your existing Prometheus servers in your existing clusters. A new stateless service, the Thanos Querier, “fans out” queries to these existing Prometheus servers. The existing Prometheus servers need an added sidecar (the Thanos Sidecar) deployed alongside them in order to handle these queries.

\n\t\t\t\"Thanos:\n\t\n

In Cortex, requests flow in the opposite direction – your existing Prometheus servers push data to a central, scalable Cortex cluster using Prometheus’ built-in remote-write capability. The central Cortex cluster stores all the data and handles queries “locally.”

\n\t\t\t\"Cortex:\n\t\n

This presents us with our first tradeoff: query performance and availability. With Thanos, chunk data must be pulled back from the “edge” locations to a central location where your Thanos queriers are running. If there is an interruption in this wide-area network, your query performance and availability will suffer. In the worst case, the latest 2 hours of data may not be available for querying. As Cortex relies on a push-based model for centrally aggregating the data, when an edge location becomes unavailable, all data up to that point is still available centrally. On the other hand, with Cortex you need to manage a separate Cortex cluster and storage on top of your Prometheus deployment – Thanos leverages your existing deployment.

2. HA Prometheus: No Gaps in the Graphs

Prometheus runs as a single process on a single computer; if that computer fails, the operating system needs updating, or Prometheus needs restarting, you end up with gaps in your graphs. To work around this, the best practice is to deploy a pair of Prometheus servers in each region; if one fails, the idea is the other one continues to scrape your jobs and store metrics. But this doesn’t solve the gaps problem – out of the box, Prometheus doesn’t provide a way to merge data from multiple replicas.

However, Thanos does! The Thanos querier will read from both replicas and combine the metrics into a single, no-gaps-included result. As the two replicas may naturally be slightly out-of-sync, Thanos uses \na heuristic to determine which results to show to avoid gaps.

\n\t\t\t\"Thanos:\n\t\n

Cortex takes a different approach. As each replica is pushing samples to the central Cortex cluster, Cortex deduplicates the streams and only stores a single copy. Cortex relies on tracking the last push from each replica and using a simple timeout to “elect” the other replica as the master.

\n\t\t\t\"Cortex:\n\t\n

Both systems result in a very similar experience: gapless graphs in Grafana.

3. Long Term Storage: Store Data for Long-Term Analysis

Prometheus is designed to run with as few dependencies as possible; it uses the local network to gather metrics and stores data on the local disk. Even through the storage format is capable of almost indefinite retention, truly durable long-term storage needs to consider disk failure, replication, and recovery.

Thanos offloads Prometheus TSDB blocks to object stores; \n7 different object stores are supported. Two-hour-long blocks are built locally and then uploaded to the object store by the Thanos sidecar. A separate microservice – the Thanos Store Gateway – is used to handle queries against the blocks in object storage. Responsibility for the durability of the blocks is offloaded to the object store – a wise choice, as replication and repair is difficult to get right.

\n\t\t\t\"Thanos:\n\t\n

Cortex takes a similar approach, offloading the responsibility for durability and replication to managed services. The key differences are that Cortex stores individual Prometheus chunks in an object store and a custom index in a NoSQL store – recall that a TSDB block is a collection of chunks from multiple time series with an included index. At Grafana Labs, we store both the index and the chunks in a Google Bigtable cluster, although both Cassandra and AWS DynamoDB are supported.

\n\t\t\t\"Cortex:\n\t\n

When comparing these approaches it boils down to two things: cost and performance. Both systems use managed services, and both systems can be run with a single dependency. Managed NOSQL stores (such as Google Bigtable or AWS DynamoDB) tend to cost more per GB stored than object stores (such as Google Cloud Storage or AWS S3), although they tend to cost less per IOP. This does tend to lead to a higher TCO for a Cortex-style system than a Thanos-style system. On the other hand, using a NOSQL store for the index in Cortex gives it an edge when it comes to query performance and scalability.

\n\t\t\t\"Thanos\n\t\n

Future Collaboration

As the Cortex and Thanos projects both aim to solve the same problems, both make heavy use of the existing Prometheus codebase, and two of the maintainers both live in London, we have been collaborating more and more over the past year.

The first example of this collaboration is that we have made the Cortex query frontend, a caching and parallelization service that accelerates PromQL queries, work for Thanos. You can read more about it in our previous blog post, \nHow to Get Blazin’ Fast PromQL. We are really excited to be able to bring these optimizations to a wider audience and welcome contributions to improve Thanos support.

The second example of collaboration is making it possible for Cortex to use Thanos’ blocks-in-object-storage approach, reusing the existing Thanos code to achieve this. My thanks go out to \nThor at DigitalOcean for putting in the legwork to make this happen. While it’s not quite ready for production usage yet, we’re really excited about the TCO improvements this will bring to Cortex, and are hoping to work closely with the Thanos team to improve scalability and performance.

I’m always amazed at how two different groups of engineers can come up with completely different (and hopefully equally valid) solutions to the same problems. At Grafana Labs, we use Cortex to power \nGrafana Cloud’s Prometheus service, and have recently starting using Thanos internally to monitor it. I’m excited to see what the future holds for Cortex and Thanos!

","id":"6cfd006a-f3f5-5747-847f-6059ff765462","title":"[PromCon Recap] Two Households, Both Alike in Dignity: Cortex and Thanos","origin_url":"https://grafana.com/blog/2019/11/21/promcon-recap-two-households-both-alike-in-dignity-cortex-and-thanos/","url":"https://grafana.com/blog/2019/11/21/promcon-recap-two-households-both-alike-in-dignity-cortex-and-thanos/","wallabag_created_at":"2021-04-29T12:10:17+00:00","published_at":null,"published_by":"['']","reading_time":6,"domain_name":"grafana.com","preview_picture":"https://s3.amazonaws.com/a-us.storyblok.com/f/1022730/769ef3c07d/thanos_cortex7.jpg","tags":["cortex","kubernetes","cassandra","grafana","aws.s3","prometheus","thanos"],"description":"This blog post is a writeup of the presentation Bartek Plotka and I gave at \nPromCon 2019.Cortex is a horizontally scalable, clustered \nPrometheus implementation aimed at giving users a global view of..."},{"content":"

To learn more about the DataStax open-source project, Metric Collector for Apache Cassandra and to try a demo, visit us on GitHub.

Apache Cassandra is a resilient system for users to build applications on, but many operators see Cassandra as a bit of a black box. It’s not that Cassandra doesn’t have hundreds of metrics to consume, it does (over 300 metric series per table!). The fact is visualizing and getting a unified view of the cluster combined with OS-level metrics and application metrics is not an easy thing for Cassandra users to set up.  

What is the Metrics Collector for Apache Cassandra?

To help solve this problem, DataStax released a new open source project called the Metric Collector for Apache Cassandra (MCAC for short).  This project provides a drop-in solution to solve this monitoring gap for Apache Cassandra. Here’s how it works.

MCAC is built on the widely used collectd agent but with a novel twist. Collectd is a metric collection agent that is well adopted and integrates well with all kinds of external metrics systems like, prometheus, graphite, stackdriver, and others. While collectd can scrape JMX metrics out of the box, JMX scraping can be quite slow and works best with only a subset of metrics. Not to mention many people don’t want to maintain and configure the metric agent on every node.  

We use MCAC to power the health tab in Astra and is bundled with our Kubernetes operator for Apache Cassandra. 

Why MCAC is different

To solve this problem MCAC comes as a single bundle with our java agent and a linux portable collectd build all in one. Just add the agent to the cassandra-env.sh, it brings up collectd and ships every metric in Cassandra to collectd via a unix-socket. It works on all Apache Cassandra versions from 2.2 -> 4.0. 

By shipping the metrics this way efficiently it is able to export hundreds of thousands of series per node with little/no impact on C* performance.

Not only does it send the metrics, but it is specially designed to work well with prometheus out of the box, like histograms are tailored for aggregation by prometheus and labels are automatically converted on ingest. This means you can slice and dice metrics across DCs, racks, down to even tables.

The Cassandra metrics are one aspect of the equation but with collectd we can also gather and expose all the OS level metrics, like context switches and disk/network performance.

MCAC also creates a historical log on the nodes of metric and non-metric diagnostic events related to activity on the node. Non-metric events include details on Flushes, Compactions, Exceptions, GC, etc. This DataLog can be used to help analyze performance or other impacting issue on the cluster. If you need help our SRE team is available to help you diagnose problems with this log https://www.datastax.com/keepcalm and if you have any questions we're here to help at https://community.datastax.com/.

Finally, what good are all these metrics without a way to visualize them! To tie it all together, MCAC comes with pre-built grafana dashboards which give operators the best Cassandra monitoring solution out there. These dashboards will change over time to focus on specific aspects of the system to make it easier to drill into the cluster.

\"Grafana\"

\"mcac\"

\"mcac2\"

","id":"44ef75af-85c2-58c6-ac00-d9e7a9bd9296","title":"Monitoring Apache Cassandra™ Made Simple","origin_url":"https://www.datastax.com/blog/monitoring-apache-cassandratm-made-simple","url":"https://www.datastax.com/blog/monitoring-apache-cassandratm-made-simple","wallabag_created_at":"2021-02-12T16:13:22+00:00","published_at":null,"published_by":"['']","reading_time":2,"domain_name":"www.datastax.com","preview_picture":"https://www.ibm.com/content/dam/worldwide-content/stock-assets/adb-stk/ul/g/30/19/adobestock_1831482701.jpeg/_jcr_content/renditions/cq5dam.web.1280.1280.jpeg","tags":["prometheus","monitoring","cassandra","grafana"],"description":"To learn more about the DataStax open-source project, Metric Collector for Apache Cassandra and to try a demo, visit us on GitHub.Apache Cassandra is a resilient system for users to build applications..."},{"content":"
\"\"

The latest beta release of the Apache Cassandra is designed to hit the ground running as the NoSQL database moves steadily to the cloud to provide managed services in production deployments.

Cassandra 4.0 released on Monday (July 20) is the first major update of the database since 2017, incorporating more than 1,000 bug fixes and extensive “battle” testing to improve performance in production, making it “the most stable release ever,” maintainers asserted. Performance testing included running Cassandra on clusters as large as 1,000 nodes using an array of enterprise use cases.

Cassandra promoters note that hyper-scalers such as Apple (NASDAQ: AAPL) have deployed the database in production with more than 75,000 nodes, illustrating its ability to scale.

Among the new features incorporated into version 4.0 is the ability to stream data between nodes during scaling operations such as adding a new node or datacenter during peak traffic times.

It also includes the new data access controls operating on a “per datacenter basis.”  In one scenario, operators of datacenters located in the Europe and the United States could configure Cassandra to allow access to a single datacenter using a “network authorizer” feature. Data governance features are gaining traction as European authorities crack down on the cross-border movement of personal user data.

Monitoring tools are also emphasized in the Cassandra latest release. Previously, open source tools from key code contributors such as DataStax and Instaclustr were the primary tools for observing Cassandra clusters.

“Constant monitoring of key performance indicators such as latency, disk usage, and throughput is critical to maintaining an optimal deployment,” Justin Cameron, a senior software engineer at Instaclustr, wrote last year in Datanami.

Around-the-clock “monitoring is necessary because both internal and external changes to Cassandra usage patterns are very common,” Cameron added.

The latest version allows users to selectively monitor system metrics and configuration settings via a feature called Virtual Tables. Other tools allow users to record and replay production workloads to analyze performance.

Along with DataStax, the data platform developer behind Cassandra, key code contributors to the 4.0 version include Amazon Web Services (NASDAQ: AMZN) and Instaclustr.

The Cassandra 4.0 better release is here.

Recent items:

Cassandra Now Officially in the Cloud with DataStax Astra

4 Apache Cassandra Pitfalls You Must Avoid

Tags: apache cassandra, cassandra, Cassandra 4.0, cloud database, cluster monitoring, Data Governance, Instaclutr, network authorizer, NoSQL database

","id":"4ac5d9e4-3f90-5f78-b0bc-2796eaee3d8e","title":"Cassandra Gets Monitoring, Performance Upgrades","origin_url":"https://www.datanami.com/2020/07/21/cassandra-gets-monitoring-performance-upgrades/","url":"https://www.datanami.com/2020/07/21/cassandra-gets-monitoring-performance-upgrades/","wallabag_created_at":"2020-11-13T15:53:24+00:00","published_at":"2020-07-21T15:19:34+00:00","published_by":null,"reading_time":2,"domain_name":"www.datanami.com","preview_picture":"https://2s7gjr373w3x22jf92z99mgm5w-wpengine.netdna-ssl.com/wp-content/uploads/2016/11/shutterstock_cloud_storage_cassandra.jpg","tags":["monitoring","cassandra"],"description":"The latest beta release of the Apache Cassandra is designed to hit the ground running as the NoSQL database moves steadily to the cloud to provide managed services in production deployments.Cassandra ..."}],"tagSets":[{"tag":"monitoring","articles":[{"content":"

Last week at the Cassandra Summit I gave a talk with Blake Eggleston on diagnosing performance problems in production. We spoke to about 300 people for about 25 minutes followed by a healthy Q&A session. I’ve expanded on our presentation to include a few extra tools, screenshots, and more clarity on our talking points.

There’s finally a lot of material available for someone looking to get started with Cassandra. There’s several introductory videos on YouTube by both me and Patrick McFadin as well as videos on time series data modeling. I’ve posted videos for my own project, cqlengine, (intro & advanced), and plenty more on the PlanetCassandra channel. There’s also a boatload of getting started material on PlanetCassandra written by Rebecca Mills.

This is the guide for what to do once you’ve built your application and you’re ready to put Cassandra in production. Whether you’ve been in operations for years or you are first getting started, this post should give you a good sense of what you need in order to address any issues you encounter.

The original slides are available via Slideshare.

Update: the presentation is now available on YouTube!

Before you even put your cluster under load, there’s a few things you can set up that will help you diagnose problems if they pop up.

  1. \n

    Ops center

    \n

    This is the standard management tool for Cassandra clusters. This is recommended for every cluster. While not open source, the community version is free. It gives you a high level overview of your cluster and provides historical metrics for the most important information. It comes with a variety of graphs that handle about 90% of what you need on a day to day basis.

    \n

    \"image\"

    \n
    2. \n
  2. \n

    Metrics plugins

    \n

    Cassandra has since version 1.1 included the metrics library. In every release it tracks more metrics using it. Why is this awesome? In previous persons of Cassandra, the standard way to access what was going on in the internals was over JMX, a very Java centric communications protocol. That meant writing a Java Agent, setting up mx4j, or Jolokia, then digging through JMX, which can be a little hairy. Not everyone wants to do this much work.

    \n

    The metrics library allows you to tell Cassandra to report its internal, table level metrics out to a whole slew of different places. Out to CSV, Ganglia, Graphite, and STDOUT, and it’s pluggable to push metrics to anywhere you want.

    \n

    \"image\"

    \n

    Read more about the metrics library integration.

    \n
    3. \n
  3. \n

    Munin, Nagios, Icinga (or other system metrics monitoring)

    \n

    I’ve found these tools to be incredibly useful at graphing system metrics as well as custom application metrics. There are many options. If you’re already familiar with one tool, you can probably keep using it. There are hosted solutions as well (server density, data dog, etc)

    \n
    \"image\"
    4. \n
  4. \n

    Statsd, Graphite, Grafana

    \n

    Your application should be tracking internal metrics. Timing queries, frequently called functions, etc. These tools let you get a profile of what’s going on with your code in production. Statsd collects raw stats and aggregates them together, then kicks them to graphite. Grafana is an optional (better) front end to Graphite.

    \n

    There was a great post by etsy, Measure Anything, Measure Everything, that introduced statsd and outlined its usage with Graphite.

    \n
    \"image\"
    5. \n
  5. \n

    Logstash

    \n

    We didn’t mention Logstash in our presentation, but we’ve found it to be incredibly useful in correlating application issues with other failures. This is useful for application logging aggregation. If you don’t want to host your own log analysis tool, there are hosted services for this as well.

    \n
    \"image\"
    6. \n

There’s a bunch of system tools that are useful if you’re logged onto a machine and want to see real time information.

  1. \n

    iostat

    \n

    iostat is useful for seeing what’s happening with each disk on your machine. If you’re hitting I/O issues, you’ll see it here. Specifically, you’re looking for high read & write rates and a big avgqu-sz (disk queue), or a high svctm (service time) there’s a good chance you’re bottlenecked on your disk. You either want to use more disks or faster disks. Cassandra loves SSDs.

    \n
    \"image\"
    2. \n
  2. \n

    htop

    \n

    Htop is a better version of top, which is useful for getting a quick glance at your system. It shows load, running processes, memory usage, and a bunch of other information at a quick glance.

    \n
    \"image\"
    3. \n
  3. \n

    iftop & netstat

    \n

    iftop is like top, but shows you active connections and the transfer rates between your server and whoever is at the other end.

    \n
    \"image\"

    Netstat is more of a networking swiss army knife. You can see network connections, routing tables, interface statistics, and a variety of other network information.

    \n
    \"image\"
    4. \n
  4. \n

    dstat

    \n

    I prefer to use dstat over iostat now since it includes all of its functionality and much of the functionality of other tools as well.

    \n
    \"image\"
    5. \n
  5. \n

    strace

    \n

    strace is useful when you want to know what system calls are happening for a given process.

    \n
    \"image\"
    6. \n
  6. \n

    pcstat

    \n

    This tool, written by Al Tobey, allows you to examine a bunch of files and quickly determine how much of each file is in the buffer cache. If you’re trying to figure out why table access is slow, this tool can tell you if your data is in cache already or if you have to go out to disk. Here’s a good read to get familiar with buffer cache. Check out the repo.

    \n
    \"image\"
    7. \n

There’s a few issues that are easy to run into that I’d consider “gotchas”, things that come up often enough that they’re worth mentioning.

A important design decision in Cassandra is that it uses last write wins when there are two inserts, updates, or deletes to a cell. To determine the last update, Cassandra uses the system clock (or the client can specify the time explicitly). If server times are different, the last write may not actually win, it’ll be the one that’s the most skewed into the future.

To address this issue, always make sure your clocks are synced. Ntpd will constantly correct for drift. ntpdate will perform a hard adjustment to your system clock. Ntpdate needs to be used if you clock is significantly off, and ntpd will keep it at the correct time.

\"image\"

Disk space not reclaimed

if you add new nodes to a cluster, each replica is responsible for less data. it’s streamed to the new nodes. however, it is not removed from the old nodes. If you’re adding new nodes because you’re running low on disk space, this is extremely important. You are required to run nodetool cleanup in order to reclaim that disk space. This is a good idea any time you change your database topology.

Issues adding nodes, or running repairs

There are two common problems that come up with repair. The first is that repairs take forever in 2.0. This is solved in 2.1 which uses an incremental repair, and does not repair data which has already been repaired. The second issue relates to trying to repair (or add nodes) to a cluster when the versions do not match. It is, in general, not a good idea (yet) to stream data between servers which are of different versions. It will appear to have started, but will just hang around doing nothing.

Cassandra comes with several tools to help diagnose performance problems. They are available via nodetool, Cassandra’s multipurpose administration tool.

Compaction

Compaction is the process of merging SSTables together. It reduces the number of seeks required to return a result. It’s a necessary part of Cassandra. If not configured correctly, it can be problematic. You can limit the I/O used by compaction by using nodetool setcompactionthroughput.

There’s 2 types of compaction available out of the box. Size Tiered is the default and great for write heavy workloads. Leveled compaction is good for read & update heavy workloads, but since it uses much higher I/O it’s recommended you use this only if you’re on SSD. I recommend reading through the documentation to understand more about which is right for your workload.

\"image\"

Histograms let you quickly understand at both a high level and table level what your performance looks like on a single node in your cluster. The first histogram, proxyhistograms, give you a quick top level view of all your tables on a node. This includes network latency. Histogram output has changed between versions to be more user friendly. The screenshot below is from Cassandra 2.1.

\"image\"

If you’d like to find out if you’ve got a performance problem isolated to a particular table, I suggest first running nodetool cfstats on a keyspace. You’ll be able to scan the list of tables and see if there’s any abnormalities. You’ll be able to quickly tell which tables are queried the most (both reads and writes).

\"image\"

nodetool cfhistograms lets you identify performance problems with a single table on a single node. The statistics are more easily read in Cassandra 2.1.

\"image\"

Query Tracing

If you’ve narrowed down your problem to a particular table, you can start to trace the queries that you execute. If you’re coming from a something like MySQL, you’re used to the command explain, which tells in in advance what the query plan is for a given query. Tracing takes a different approach. Instead of showing a query plan, query tracing keeps track of the events in the system whewn it actually executes. Here’s an example where we’ve created a whole bunch of tombstones on a partition. Even on a SSD you still want to avoid a lot of tombstones - it’s disk, CPU, and memory intensive.

\"image\"

The JVM gets a reputation for being a bit of a beast. It’s a really impressive feat of engineering, but it shouldn’t be regarded as black magic. I strongly recommend reading through Blake Eggleston’s post on the JVM, it’s well written and does a great job of explaining things. (Much better than I would here).

OK - we’ve got all these tools under our belt. Now we can start to narrow down the problem.

This started out as a small recap but has evolved into much more than that. The tools above have helped me a wide variety of problems, not just Cassandra ones. If you follow the above recommendations you should be in a great spot to diagnose most problems that come your way.

You can find me on Twitter for any comments or suggestions.

","id":"eef63404-0bd6-5a8d-96dc-2a4c8b270540","title":"Cassandra Summit Recap: Diagnosing Problems in Production - RustyRazorblade.com","origin_url":"http://rustyrazorblade.com/post/2014/2014-09-18-diagnosing-production/","url":"http://rustyrazorblade.com/post/2014/2014-09-18-diagnosing-production/","wallabag_created_at":"2023-03-01T19:46:02+00:00","published_at":null,"published_by":null,"reading_time":10,"domain_name":"rustyrazorblade.com","preview_picture":"http://rustyrazorblade.com/images/default.png","tags":["monitoring","cassandra"],"description":"Last week at the Cassandra Summit I gave a talk with Blake Eggleston on diagnosing performance problems in production. We spoke to about 300 people for about 25 minutes followed by a healthy Q&A sessi..."},{"content":"

{{ message }}

\n

/ cfstats-csv-parser Public

Repo for a utility to parse cfstats into a csv file for analysis

License

MIT license

1 star 4 forks

Star

Notifications

\n

You can’t perform that action at this time.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.

\n
","id":"8e3a1b9c-7cd2-578f-a0b1-877ac6f47cb3","title":"GitHub - jlacefie/cfstats-csv-parser: Repo for a utility to parse cfstats into a csv file for analysis","origin_url":"https://github.com/jlacefie/cfstats-csv-parser","url":"https://github.com/jlacefie/cfstats-csv-parser","wallabag_created_at":"2023-03-01T19:40:38+00:00","published_at":null,"published_by":"['jlacefie']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/a44d5ad918f82ef2250d84b5d7d746b2e183f62eb7bd35a736db6977a6ff0bfb/jlacefie/cfstats-csv-parser","tags":["monitoring","cassandra"],"description":"{{ message }}\n / cfstats-csv-parser PublicNotifications\nFork 4\n\n Star 1 \n\nRepo for a utility to parse cfstats into a csv file for analysisLicense MIT license 1 star 4 forks Star Notifications\n\n © 202..."},{"content":"
\"image\"

Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system used by the largest tech companies in the world. There are numerous reasons behind its popularity, including no single point of failure, exceptional horizontal scaling with a data layout designed as a perfect fit for time-series data.

However, despite these perks, like any other system, Cassandra is prone to performance issues. This makes monitoring imperative. And it all starts with knowing what to measure. In this article, we will explain the key Cassandra performance metrics you should monitor to make sure everything is up and running at all times.

What Is Cassandra and How Does It Work?

Let’s keep it short – Apache Cassandra is a distributed NoSQL database designed to provide fault-tolerant and highly available architecture with performance in mind.

As a distributed system Cassandra is built out of nodes. A node is a single instance of Apache Cassandra that can operate on its own. Multiple nodes can form a cluster – a distributed system holding common data and responding to query requests. Cassandra works in a master-less architecture where each node communicates in a peer to peer fashion using a protocol known as Gossip. The gossip protocol is designed so that each node is informed about the state of all other nodes and a single node performs gossip communication with up to three other nodes every second.

The cluster can be divided into data centers and racks, just like the real-life data centers are divided. In Cassandra terminology, a data center is designed to hold multiple racks and a single rack holds a complete replica of the data.

\"image\"

Cassandra Cluster Logical Overview

When it comes to the data, Cassandra stores it in tables that are organized the same way as in any other database – in rows and columns. A single table is called a column family. The tables themselves are grouped into keyspaces, where a keyspace usually holds logically similar data – for example, from a business perspective. The keyspace is also used for data replication, and the replication itself is configured on a keyspace level.

Getting back to the tables. Each table defines a primary key that is built of the partition key and the clustering columns. Cassandra uses the partition key to index the data. All data that share a common partition key make a single data partition – a basic unit for data retrieval and storage. The clustering columns are optional.

Needless to say, Apache Cassandra is a complicated, distributed system and it’s not uncommon for users to encounter operation problems and difficulties. Everything breaks eventually, from the low-level bare metal components, up to the high-level software. It is not unusual for users to deal with network issues and CPU utilization problems, especially on very large clusters. Cassandra is written in Java and uses both off-heap and heap memory, which means that as the volume of data grows, you may hit issues with the garbage collector. Finally, because of the amount of data that you will process you may need to deal with the hard disk space and performance of your I/O subsystem. All of these can be avoided by keeping an eye for the relevant metrics with the help of a good Cassandra monitoring tool.

How Is Cassandra Performance Measured?

The most complex, distributed systems provide a set of metrics that you should take care of, monitor, and alert on to ensure that your system is healthy and working well. Apache Cassandra is no different. It provides a plethora of performance metrics which we can divide into three categories:

  • Dedicated Apache Cassandra metrics that describe how the system and its parts perform.
  • Java Virtual Machine metrics that tell you about the execution environment on which Apache Cassandra is running.
  • Operating system metrics describing the metrics related to the bare metal servers, virtual machines, or containers, depending on the environment that you are using.

Dedicated Cassandra Performance Metrics

When monitoring Apache Cassandra clusters, is the metrics that the distributed data store exposes via the JMX interface. There are many Cassandra performance metrics exposed in the JMX and having visibility into most of them is a good idea. You never know what can be useful when troubleshooting.

Nodes

One of the most important Cassandra metrics is the number of nodes that are currently available and connected to form a cluster. The ability to store the data and respond to queries is directly related to the availability of nodes.

Compaction Metrics

Compaction is the operation of merging multiple smaller instances of SSTable into one bigger SSTable that contains all the data from the smaller tables. Because of that, it can be very expensive and resource-consuming. Having visibility into compaction performance is critical for long-term observability – the Cassandra monitoring tool of your choice needs to provide the number of compactions and the number of compacted bytes.

During compaction, until the process ends, the total disk space used may be double that before the compaction. Because of that, you should consider leaving about 50% of space free to account for compactions and, of course, set up appropriate alerts to inform you when the amount of free disk space is close to a level where compaction could fail.

Read and Write Performance Metrics

The next set of metrics is dedicated to clients and the read and write side of the operations. You should measure the number of reads happening in a given period, the request latency, and the number of timeouts and failures. Your Cassandra monitoring tool should provide the top-level view and allow for slicing and dicing through the data showing you the aggregated view, per node view, per keyspace view, and per table view. The same goes for write operations.

You should see the number of write requests happening and write latency. Local writes and reads may also be important when troubleshooting.

Table Metrics

Table metrics are also essential. The ones you should pay close attention to are partition size, tombstone scans, and the number of SSTables per read.

Partition Size

Partition size is crucial for cluster performance. Cassandra uses it as a unit of data storage, replication, and retrieval, thus directly dictating the performance of your Cassandra tables. The ideal partition size varies but is usually below 100MB and not less than 10 – 20MB.

Tombstones

Cassandra produces tombstones when you delete the data. They are markers of the deleted data. Data in Cassandra is immutable by design, and because of that, it can only be physically removed from the SSTable during compactions. Because of that, you should keep an eye on how they affect your disk space.

SSTables Per Read

Similar to tombstones, the number of SSTables per read is related to the immutability of the data in Cassandra. A single table can be built of multiple SSTables, which are written sequentially. A single read operation can result in reading multiple SSTables to retrieve the relevant data. The more SSTables Cassandra needs to read to return the data, the more resources are required to complete the read operation. This is why you should minimize the number whenever possible.

Other Metrics

As we mentioned earlier, other Apache Cassandra performance metrics can be helpful and you should consider monitoring them.

Caches

There are two types of caches in Cassandra – the key cache and the row cache. Cassandra uses the key cache to store the location of row keys in memory so that the rows can be accessed without the need to hit the disk. The row cache stores the rows themselves in memory. By using the caches, Cassandra reduces the need to read the data from the disk and trades the memory usage for performance.

You need to monitor the key cache requests and row cache requests, which tell how many requests to a given cache type were made, and the key cache hit ratio and the row cache hit ratio, which show the percentage of results retrieved from the cache instead of the disk.

Threadpool

Cassandra is designed to handle the high load, withstand backpressure, and perform asynchronous tasks. Monitoring various thread pools is crucial for understanding Cassandra’s performance and bottlenecks. Each thread pool exposes the number of active, pending, and blocked tasks. Accumulated, pending, and blocked tasks usually tell about performance issues and the need for more processing power or different data and query architecture.

Bloom Filter

In the read path, Cassandra merges the data stored on a disk inside the SSTables with the data stored in memory. To minimize the amount of checking for data existence in the SSTables on the disk Cassandra uses a data structure called bloom filter.

The bloom filter is a probabilistic data structure that can tell Cassandra that the data is definitely not in a given file or that the data may be present in a given file. The key metrics to monitor here are the amount of space used by bloom filters, the number of false positives, and the ratio. You can reduce the number of false positives by assigning more memory to the bloom filters.

Java Virtual Machine Metrics

Apache Cassandra is a JVM-based application that comes with all the usual JVM pros and cons. From the developer’s perspective, memory management is easier and requires less hassle – you just use an object and forget about it, letting the JVM do the cleaning up. But that means that something has to clean up all the unused objects in memory. This is where the Java Garbage Collection comes in and the metrics that come with it.

A proper Cassandra monitoring tool should provide metrics that allow you to check and troubleshoot issues with the Java Virtual Machine, such as JVM memory utilization and garbage collection count and time. You can read more about them in our guide about JVM metrics.

Operating System Metrics

You can’t ignore Operating System metrics either. Information such as CPU utilization, memory usage, and disk usage is essential and can play a major role when it comes to Cassandra performance.

CPU Utilization

Your CPU is used for data processing and query handling. The more spare CPU cycles you have on a given node, the data and queries it can process. The user part of the CPU usage will show you your Cassandra process needs, while the wait can point to a bottleneck in I/O or network. As with every Java application, CPU cycles are also needed for garbage collection, so keep that in mind when planning.

Memory Usage

Memory usage is crucial for every JVM-based application. The newest version of Cassandra leverages both off-heap and heap memory. This means that you not only need to set the heap size of your Cassandra nodes correctly but also have enough off-heap memory for keeping your cluster performance at its best.

Disk Usage

Disk and I/O are crucial – Cassandra keeps its data on the disk, and each query may require a substantial number of I/O operations to return the results. You need to be sure that your hardware can handle your data retrieval needs. You also need to be sure that you have enough space to hold your data and handle the compaction process.

Monitor Cassandra Performance with Sematext

\"monitoring

Sematext Cloud and its Apache Cassandra monitoring integration provide all that you need to monitor your distributed database. Everything is within a single view available without distractions:

  • The overview report gives you a perfect start point for your metrics, painting a picture of the whole cluster.
  • A dedicated Cassandra report that provides an in-depth view of all relevant metrics related to the distributed database.
  • The OS report provides necessary operating system metrics such as CPU and memory utilization and visibility into your network traffic.
  • Finally, the JVM metrics give the full view of the Java Virtual Machine, such as metrics related to garbage collection and per-heap space memory utilization.

Using the dedicated split-view, you can correlate all the available metrics with other metrics, logs, and real user monitoring data, making Sematext a perfect visibility tool.

Sematext allows you to set up alerts on any metric or log the event and supports both threshold-based and anomaly-based alerts for full flexibility. You don’t have to watch your metrics over and over. Once you configure your alerts, you can sleep well, and Sematext will let you know if something is wrong.

If you want to see how Sematext stacks against similar solutions, read our article about the best Cassandra monitoring tools available today.

Get Started with Cassandra Monitoring

As the distributed database Apache Cassandra can quickly become an operational challenge without visibility into what is happening from a global perspective as well as on a node level. You need to have full visibility from top to bottom, but that is not enough. You need to be sure that your monitoring system can notify you when an issue happens and also predict issues before your customers notice them.

One of the tools that will give you all of that is Sematext’s Apache Cassandra monitoring integration. String monitoring your Cassandra cluster by creating the Sematext Cloud account and then the Cassandra monitoring App. Don’t forget to create a Logs App as well to ship you Cassandra logs for a full observability experience.

You might also like

Products

Services

About

Contact

\n© Sematext Group. All rights reserved\n
Terms Of Service · Privacy Policy

\"Software\n\"Software\n\"GetApp\n\"Crozdesk\n\"Crozdesk\n\"Crozdesk
","id":"d16c68f8-5520-5e6b-a126-62cd8cda87a2","title":"How Do You Monitor Cassandra Performance: Key Metrics to Measure","origin_url":"https://sematext.com/blog/cassandra-monitoring/","url":"https://sematext.com/blog/cassandra-monitoring/","wallabag_created_at":"2021-11-08T17:30:27+00:00","published_at":"2021-10-04T10:55:25+00:00","published_by":"['Rafal Kuć']","reading_time":11,"domain_name":"sematext.com","preview_picture":"https://sematext.com/wp-content/uploads/2021/10/critical-cassandra-metrics-to-monitor.jpg","tags":["monitoring","cassandra","performance"],"description":"Apache Cassandra is a distributed database known for its high availability, fault tolerance, and near-linear scaling. It was initially developed by Facebook, but it is a widely used open-source system..."},{"content":"

To learn more about the DataStax open-source project, Metric Collector for Apache Cassandra and to try a demo, visit us on GitHub.

Apache Cassandra is a resilient system for users to build applications on, but many operators see Cassandra as a bit of a black box. It’s not that Cassandra doesn’t have hundreds of metrics to consume, it does (over 300 metric series per table!). The fact is visualizing and getting a unified view of the cluster combined with OS-level metrics and application metrics is not an easy thing for Cassandra users to set up.  

What is the Metrics Collector for Apache Cassandra?

To help solve this problem, DataStax released a new open source project called the Metric Collector for Apache Cassandra (MCAC for short).  This project provides a drop-in solution to solve this monitoring gap for Apache Cassandra. Here’s how it works.

MCAC is built on the widely used collectd agent but with a novel twist. Collectd is a metric collection agent that is well adopted and integrates well with all kinds of external metrics systems like, prometheus, graphite, stackdriver, and others. While collectd can scrape JMX metrics out of the box, JMX scraping can be quite slow and works best with only a subset of metrics. Not to mention many people don’t want to maintain and configure the metric agent on every node.  

We use MCAC to power the health tab in Astra and is bundled with our Kubernetes operator for Apache Cassandra. 

Why MCAC is different

To solve this problem MCAC comes as a single bundle with our java agent and a linux portable collectd build all in one. Just add the agent to the cassandra-env.sh, it brings up collectd and ships every metric in Cassandra to collectd via a unix-socket. It works on all Apache Cassandra versions from 2.2 -> 4.0. 

By shipping the metrics this way efficiently it is able to export hundreds of thousands of series per node with little/no impact on C* performance.

Not only does it send the metrics, but it is specially designed to work well with prometheus out of the box, like histograms are tailored for aggregation by prometheus and labels are automatically converted on ingest. This means you can slice and dice metrics across DCs, racks, down to even tables.

The Cassandra metrics are one aspect of the equation but with collectd we can also gather and expose all the OS level metrics, like context switches and disk/network performance.

MCAC also creates a historical log on the nodes of metric and non-metric diagnostic events related to activity on the node. Non-metric events include details on Flushes, Compactions, Exceptions, GC, etc. This DataLog can be used to help analyze performance or other impacting issue on the cluster. If you need help our SRE team is available to help you diagnose problems with this log https://www.datastax.com/keepcalm and if you have any questions we're here to help at https://community.datastax.com/.

Finally, what good are all these metrics without a way to visualize them! To tie it all together, MCAC comes with pre-built grafana dashboards which give operators the best Cassandra monitoring solution out there. These dashboards will change over time to focus on specific aspects of the system to make it easier to drill into the cluster.

\"Grafana\"

\"mcac\"

\"mcac2\"

","id":"44ef75af-85c2-58c6-ac00-d9e7a9bd9296","title":"Monitoring Apache Cassandra™ Made Simple","origin_url":"https://www.datastax.com/blog/monitoring-apache-cassandratm-made-simple","url":"https://www.datastax.com/blog/monitoring-apache-cassandratm-made-simple","wallabag_created_at":"2021-02-12T16:13:22+00:00","published_at":null,"published_by":"['']","reading_time":2,"domain_name":"www.datastax.com","preview_picture":"https://www.ibm.com/content/dam/worldwide-content/stock-assets/adb-stk/ul/g/30/19/adobestock_1831482701.jpeg/_jcr_content/renditions/cq5dam.web.1280.1280.jpeg","tags":["prometheus","monitoring","cassandra","grafana"],"description":"To learn more about the DataStax open-source project, Metric Collector for Apache Cassandra and to try a demo, visit us on GitHub.Apache Cassandra is a resilient system for users to build applications..."}]},{"tag":"cassandra","articles":[{"content":"

This is the second post in my series on improving node density and lowering costs with Apache Cassandra. In the previous post, I examined how streaming performance impacts node density and operational costs. In this post, I’ll focus on compaction throughput, and a recent optimization in Cassandra 5.0.4 that significantly improves it, CASSANDRA-15452.

This post assumes some familiarity with Apache Cassandra storage engine fundamentals. The documentation has a nice section covering the storage engine if you’d like to brush up before reading this post.

The Compaction Bottleneck

Compaction in Cassandra is the process of merging multiple SSTables and writing out new ones, discarding tombstones, resolving overwrites, and generally organizing data for efficient reads. It’s an I/O intensive background operation that directly competes with foreground operations for system resources. In a later post I’ll look at how compaction strategies impact node density, but for now, I’ll just focus on throughput.

Why Compaction Throughput Matters for Node Density

As we continue to increase the amount of data we store per node, compaction performance becomes increasingly important. It affects:

Simply put: as your data volume and write throughput increase, compaction throughput must as well. If it doesn’t, you’ll hit a performance wall that effectively caps your maximum practical node density.

Despite the significant improvements to compaction throughput over the years, there are some circumstances where compaction performance is inadequate. Let’s take a look at the reason why, then dive into what can be done about it.

When doing any performance evaluation, it’s important to understand how to measure where your time is spent. A lot of folks make incorrect assumptions, and then waste a lot of time trying to optimize something that doesn’t matter. I’ve written several posts about how useful profiling with the async-profiler can be from an application perspective. For looking at the OS and hardware, the eBPF based toolkit bcc-tools can help you identify process bottlenecks. I’ve used these tools extensively over the years, and in this post I’ll show how they’ve helped identify two major performance bottlenecks in compaction. My easy-cass-lab software includes all these tools, as well as integration with AxonOps for Cassandra dashboards and operational tooling.

Being 10x Smarter With Our Disk Access

When investigating compaction behavior, I discovered an major inefficiency in how Cassandra was accessing disk. The problem was especially severe in cloud environments with disaggregated storage like AWS EBS, where IOPS (Input/Output Operations Per Second) are both limited and expensive when used improperly.

When Cassandra would read in data during compaction, it would read individual compressed chunks off disk, one small read at a time. Using bcc-tools, we can monitor every filesystem operation. Here I’m using xfsslower to record every read operation on the filesystem (original headers back in for clarity):

$ sudo /usr/share/bcc/tools/xfsslower 0 -p 26988 | awk '$4 == \"R\" { print $0 }'\nTracing XFS operations\nTIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME\n22:27:38 CompactionExec 26988  R 4096    0           0.01 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988  R 4096    4           0.00 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988  R 2062    8           0.00 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988  R 14907   0           0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14924   14          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14896   29          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14844   43          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14923   58          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14931   72          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14905   87          0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14891   101         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14919   116         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14965   130         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14918   145         0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988  R 14930   160         0.01 nb-7-big-Data.db\n

The above is showing we’re reading about 14KB at a time. That’s the size of the compressed page. This pattern is terrible for performance on cloud storage systems like EBS, where:

  1. Each read operation, no matter how small, counts against your provisioned IOPS
    2. \n
  2. Small reads waste IOPS quota while delivering minimal data
    3. \n
  3. You pay for IOPS allocation whether you use it efficiently or not
    4. \n

Looking at a wall clock performance profile, we can see compaction is spending a LOT of time waiting on disk, in the really wide column with the pread call at the top:

\"wall-clock-profile-compaction.png\"

Readahead is a disk optimization strategy where the operating system reads a larger block of data than was requested into memory. The objective is reduce latency and improve performance for sequential read operations. Unfortunately, when you don’t need the data it’s reading, it can be the source of major performance problems. In my experience, read ahead is one of the worst culprits in the world of Cassandra performance. It’s especially terrible for lightweight transactions and counters, where we perform a read before write.

My advice to Cassandra operators is to reduce readahead to 4KB to avoid unnecessary read amplification on the read path.

Readahead does have one place, however, where it can benefit performance. You may have already guessed that it’s compaction. Let’s take a step back and look at how the size of our reads impacts our throughput in a simple benchmark. Larger reads, initiated either from read ahead or the user, should deliver improved throughput, especially when we’re dealing with a quota on our IOPS (EBS), our drives have higher latency (SAN), or both.

Benchmarking

I ran benchmark tests with sequential fio workloads using different request sizes on a 3K IOPS GP3 EBS volume. Here’s the configuration used:

[global]\nrw=read\ndirectory=data\ndirect=1\ntime_based=1\nfile_service_type=normal\nstonewall\nsize=100M\nnumjobs=12\ngroup_reporting\n[bs4]\nstonewall\nruntime=60s\nblocksize=4k\n[bs8]\nstonewall\nruntime=60s\nblocksize=8k\n[bs16]\nstonewall\nruntime=60s\nblocksize=16k\n[bs32]\nstonewall\nruntime=60s\nblocksize=32k\n[bs64]\nstonewall\nruntime=60s\nblocksize=64k\n[bs128]\nstonewall\nblocksize=128k\nruntime=60s\n[bs256]\nstonewall\nruntime=60s\nblocksize=256k\n

When reviewing the results, the benefits of using larger request sizes were evident:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Request SizeIOPSThroughput
4K304911.9 MB/s
8K301223 MB/s
16K301347 MB/s
32K301394 MB/s
64K1938121 MB/s
128K957120 MB/s
256K478120 MB/s

The data shows that using 256KB reads instead of 16KB reads would deliver almost 3x the throughput while using only 1/6th of the provisioned IOPS. That’s a massive efficiency improvement. Rather than chewing through all our IOPS to deliver a paltry 47MB/s of throughput, we’re only using about 500 for 120MB/s. That means if we can see these gains in the database, we’ll be able to compact faster, put more data on each node, and lower our total cost.

The Solution: Internally Buffering Sequential Reads

In CASSANDRA-15452, I worked with my fellow Cassandra committer Jordan West to implement a solution: an efficient, internal read-ahead buffer for bulk reading operations. Here’s how it works:

  1. Instead of reading tiny chunks, we use a 256KB off-heap buffer
    2. \n
  2. Each read operation pulls in a full 256KB of data at once
    3. \n
  3. Compressed chunks are extracted from this buffer as needed
    4. \n
  4. The buffer is refilled only when necessary
    5. \n

This approach maximizes IOPS efficiency by using larger reads during compaction (as well as repair and range reads) that deliver more data per operation. For cloud environments, it’s a game-changer that directly aligns with storage provider recommendations. AWS EBS, for instance, considers any I/O operation up to 256KB as a single operation, so by using the largest possible size we should get optimal performance.

Real-World Impact: A Major Improvement in Compaction Throughput

When Jordan and I tested the implementation using easy-cass-lab on EBS, the results were nothing short of spectacular. The 10.0.2.171 node is running our patched version, the other two nodes are running an unpatched release. The graphs clearly show a 2-3x improvement to throughput and a 3x reduction in IOPS.

\"15452-bytes-read.png\"

\"Compaction

You can see the results in the flamegraph as well. The calls to pread take up significantly less time.

\"wall-clock-profile-compaction-after-15452.png\"

We can use xfsslower from bcc-tools again to watch the filesystem access:

$ sudo /usr/share/bcc/tools/xfsslower 0 -p $(cassandra-pid) | awk '$4 == \"R\" { print $0 }'\nTIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME\n14:40:29 CompactionExec 1782   R 262144  256         0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  512         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  768         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1024        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1280        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1536        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 241123  1792        0.07 nb-4-big-Data.db\n

This is a lot better, now we’re fetching 256KB at a time using way fewer requests.

The EBS test configuration used a GP3 volume with 3K IOPS and 256MB throughput. With the existing code, compaction was bottlenecked by IOPS, peaking at exactly 3K IOPS but achieving only about 51MB/s throughput. With our optimization, the same operation used only ~500 IOPS to achieve around 106MB/s—a more than 2x improvement in throughput with 1/3IOPS.

In our most aggressive testing, we actually hit the EBS throughput limit rather than the IOPS limit. That’s a significant transformation in Cassandra’s resource utilization profile.

The patch also has the benefit of applying to anti-compaction, repair, and range reads. We can see a significant reduction in range reads, aka table scans:

\"15452-range-reads.png\"

If you’re running Spark jobs using the Cassandra connector, you should see an improvement in performance, and your repair times should decrease.

What’s next? Can we do more?

Yes, absolutely! There’s several more improvements to IO that will help improve things. I’ll cover them here very quickly, and if there’s interest I’ll write about them in detail in a future post.

Avoid Reading the Statistics

When compacting, we read data out of the Statistics.db file before reading the data itself. This is completely unnecessary, as it’s stats about the data we’re about to read. Skipping this can reduce IO even further. Looking at a compaction’s IO activity, I see about 30% of the filesystem access is reading from Statistics.db:

14:40:29 CompactionExec 1782   R 4096    0           0.00 nb-3-big-Statistics.db\n14:40:29 CompactionExec 1782   R 701     4           0.00 nb-3-big-Statistics.db\n14:40:29 CompactionExec 1782   R 4096    0           0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782   R 4096    4           0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782   R 1962    8           0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782   R 262144  0           0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 2115    0           0.01 nb-3-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  256         0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  512         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  768         0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1024        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1280        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 262144  1536        0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782   R 241123  1792        0.07 nb-4-big-Data.db\n

This has already been fixed in trunk by Branimir Lambov in CASSANDRA-20092 and is being backported to 5.0 by Jordan.

Direct I/O for Compaction

Let’s talk more about page cache. Since we go through the Linux page cache when doing reads, we want to make sure it’s working optimally. Page cache lets us avoid going to disk! Unfortunately we also use it when reading for compaction. This is a problem because we’re pulling data into the page cache that we plan on deleting. To make room for the new data, other data will be evicted. If we compact 10GB of data, we’re pushing out a lot of valuable data from the page cache, meaning it needs to be fetched back into memory later on. Using Direct I/O we can bypass the page cache entirely, which will prevent data from being evicted. This can be a huge help in latency sensitive systems or systems where IOPS are limited like EBS.

I’ve filed CASSANDRA-19987 to look at this.

Non Blocking Compression

Next, compression. When we’re writing to disk, we fill a buffer, sized by the chunk_length_in_kb table setting, compress, and write to disk. The compression here is a blocking call, which means we can spend a lot of time waiting on compression to finish, when we could be reading and merging the next chunk in parallel This can show up as a performance bottleneck, so I’ve filed CASSANDRA-20085 to look into it.

Better Memory Management

When a system is not bottlenecked on disk I/O, such as when using NVMe, the main issue we’ll run into is our heap allocation rate. I’ll go into details in a future post, but for now, it’s enough to know that the more memory we allocate, the worse our performance. Being smart about memory allocations can make a big difference in overall time spent, as allocations aren’t free. It also reduces both the frequency and duration of Garbage Collection. Big wins all around.

I recently profiled an instance where the row size was about 2KB (not out of the ordinary) and found that a single call was accounting for roughly 50% of memory allocated. Fixing this one thing has the potential to deliver a massive performance improvement, especially in workloads where we have either lots of fields, or large fields like serialized blobs.

Reaching again for async-profiler, this time we run it with -e alloc to track allocations and --reverse to reverse the stacks. I do this because the same underlying call comes from the read path and compaction, and I want to see the time in aggregate.

\"allocation-profile-compaction.png\"

Addressing this single allocation won’t just deliver faster compaction, but will reduce pressure on the heap, which in turn reduces GC overhead. As part of this series I’ll also be covering GC, as a lot’s changed since I wrote about it last.

I’ve filed CASSANDRA-20428 and there’s already a fair bit of discussion about different approaches to solving the problem.

Conclusion

Maximizing compaction throughput is critical for achieving higher node density with Apache Cassandra. The improvements in CASSANDRA-15452 have removed one of the primary bottlenecks that previously limited practical node size in a lot of clusters.

By upgrading to Cassandra 5.0.4 (or later) you can:

  1. Dramatically improve compaction throughput
    2. \n
  2. Reduce IOPS consumption significantly
    3. \n
  3. Improve overall system stability during write-heavy workloads
    4. \n
  4. Increase the maximum practical data density per node
    5. \n
  5. Significantly reduce your cloud storage costs
    6. \n

This improvement, combined with the streaming optimizations discussed in the previous post, creates a multiplier effect on your ability to increase node density. Each optimization removes a bottleneck, allowing you to push your hardware further and achieve more with less.

In my next post, I’ll be discussing how and why compaction strategies affect node density. Picking the right strategy can have a significant impact on your cluster’s performance and cost efficiency. Make sure you sign up for my mailing list if you’re interested in getting notified when it’s released!

If you found this post helpful, please consider sharing to your network. I'm also available to help you be successful with your distributed systems! Pleasereach outif you're interested in working with me, and I'll be happy to schedule a free one-hour consultation.","id":"53b52a64-a2c0-5b1e-83ff-89bcfe2da9b9","title":"Cassandra Compaction Throughput Performance Explained","origin_url":"https://rustyrazorblade.com/post/2025/04-compaction-throughput/","url":"https://rustyrazorblade.com/post/2025/04-compaction-throughput/","wallabag_created_at":"2025-04-24T12:03:02+00:00","published_at":"2025-04-16T00:00:00+00:00","published_by":"['']","reading_time":14,"domain_name":"rustyrazorblade.com","preview_picture":"https://rustyrazorblade.com/images/2025/wall-clock-profile-compaction.png","tags":["cassandra","performance"],"description":"This is the second post in my series on improving node density and lowering costs with Apache Cassandra. In the previous post, I examined how streaming performance impacts node density and operational..."},{"content":"

Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building reliable applications.

Accord is in active development and still a feature branch in the Apasche Cassandra® Repo. You will find bug. What we ask is that you help with a contribution of a bug report.

You can use the Github discussions bug report forum for this or use the Planet Cassandra Discord channel for accord listed below. A bug report should have the folowing:

If you have suggestions about syntax or improving the overall developer expirience, we want to hear about that to! Add it as a suggestion or feature request using Github discussions or let us know in the Planet Cassandra Discord.

Now, on to the fun!

docker pull pmcfadin/cassandra-accord\ndocker run -d --name cassandra-accord -p 9042:9042 pmcfadin/cassandra-accord
brew tap rustyrazorblade/rustyrazorblade\nbrew install easy-cass-lab
/\n├── docker/              # Docker configuration and setup\n├── easy-cass-lab/      # Multi-node testing environment\n├── examples/           # Implementation examples\n│   ├── banking/       # Financial transaction examples\n│   ├── inventory/     # Stock management examples\n│   └── user-mgmt/     # User operations examples\n└── docs/              # Guides and documentation\n

Our documentation includes:

  1. Choose your deployment option:\n
    2. \n
  2. Follow the Quick Start Guide
    3. \n
  3. Explore example implementations
    4. \n
  4. Connect with our Discord community
    5. \n
  5. Feedback! Github Discussions
    6. \n
BEGIN TRANSACTION\n    LET fromBalance = (SELECT account_balance \n                      FROM ks.accounts \n                      WHERE account_holder='alice');\n    IF fromBalance.account_balance >= 20 THEN\n        UPDATE ks.accounts \n        SET account_balance -= 20 \n        WHERE account_holder='alice';\n        UPDATE ks.accounts \n        SET account_balance += 20 \n        WHERE account_holder='bob';\n    END IF\nCOMMIT TRANSACTION;

Apache License 2.0

","id":"227c7330-078e-50ac-95e7-f5e8c0264a5c","title":"GitHub - pmcfadin/awesome-accord: Repository of all kinds of things to help you get up and running with ACID transactions on Apache Cassandra®","origin_url":"https://github.com/pmcfadin/awesome-accord","url":"https://github.com/pmcfadin/awesome-accord","wallabag_created_at":"2025-01-16T16:28:31+00:00","published_at":null,"published_by":"['pmcfadin']","reading_time":1,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/3e477fb2dd2b1ded1c5b53477f4848297badc75ece00c5b49bad1476fdb76167/pmcfadin/awesome-accord","tags":["acid","open.source","cassandra","accord"],"description":"Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building..."},{"content":"

Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:

Visual Flow application is divided into the following repositories:

Check the official guide.

Visual flow is an open-source software licensed under the Apache-2.0 license.

","id":"3f0ec87d-a3e1-5490-9ee1-3e96df4176c1","title":"GitHub - ibagroup-eu/Visual-Flow: Visual-Flow main repository","origin_url":"https://github.com/ibagroup-eu/Visual-Flow","url":"https://github.com/ibagroup-eu/Visual-Flow","wallabag_created_at":"2024-12-02T13:34:31+00:00","published_at":null,"published_by":"['ibagroup-eu']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/9187fdecad3a37939c1971bcdec19ffed4090307ee508b009f47c7bcd49a7f8d/ibagroup-eu/Visual-Flow","tags":["mongo","nocode","elasticsearch","open.source","cassandra","data.pipeline","elastic","aws.s3","etl","low.code","postgres"],"description":"Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:Can integrate data from heterogeneous sources:\nA..."},{"content":"

\"GitHub \"Go

\"cql-proxy\"

cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.

The cql-proxy sidecar enables unsupported CQL drivers to work with DataStax Astra. These drivers include both legacy DataStax drivers and community-maintained CQL drivers, such as the gocql driver and the rust-driver.

cql-proxy also enables applications that are currently using Apache Cassandra or DataStax Enterprise (DSE) to use Astra without requiring any code changes. Your application just needs to be configured to use the proxy.

If you're building a new application using DataStax drivers, cql-proxy is not required, as the drivers can communicate directly with Astra. DataStax drivers have excellent support for Astra out-of-the-box, and are well-documented in the driver-guide guide.

Use the -h or --help flag to display a listing all flags and their corresponding descriptions and environment variables (shown below as items starting with $):

$ ./cql-proxy -h\nUsage: cql-proxy\nFlags:\n  -h, --help                                              Show context-sensitive help.\n  -b, --astra-bundle=STRING                               Path to secure connect bundle for an Astra database. Requires '--username' and '--password'. Ignored if using the\n                                                          token or contact points option ($ASTRA_BUNDLE).\n  -t, --astra-token=STRING                                Token used to authenticate to an Astra database. Requires '--astra-database-id'. Ignored if using the bundle path\n                                                          or contact points option ($ASTRA_TOKEN).\n  -i, --astra-database-id=STRING                          Database ID of the Astra database. Requires '--astra-token' ($ASTRA_DATABASE_ID)\n      --astra-api-url=\"https://api.astra.datastax.com\"    URL for the Astra API ($ASTRA_API_URL)\n      --astra-timeout=10s                                 Timeout for contacting Astra when retrieving the bundle and metadata ($ASTRA_TIMEOUT)\n  -c, --contact-points=CONTACT-POINTS,...                 Contact points for cluster. Ignored if using the bundle path or token option ($CONTACT_POINTS).\n  -u, --username=STRING                                   Username to use for authentication ($USERNAME)\n  -p, --password=STRING                                   Password to use for authentication ($PASSWORD)\n  -r, --port=9042                                         Default port to use when connecting to cluster ($PORT)\n  -n, --protocol-version=\"v4\"                             Initial protocol version to use when connecting to the backend cluster (default: v4, options: v3, v4, v5, DSEv1,\n                                                          DSEv2) ($PROTOCOL_VERSION)\n  -m, --max-protocol-version=\"v4\"                         Max protocol version supported by the backend cluster (default: v4, options: v3, v4, v5, DSEv1, DSEv2)\n                                                          ($MAX_PROTOCOL_VERSION)\n  -a, --bind=\":9042\"                                      Address to use to bind server ($BIND)\n  -f, --config=CONFIG                                     YAML configuration file ($CONFIG_FILE)\n      --debug                                             Show debug logging ($DEBUG)\n      --health-check                                      Enable liveness and readiness checks ($HEALTH_CHECK)\n      --http-bind=\":8000\"                                 Address to use to bind HTTP server used for health checks ($HTTP_BIND)\n      --heartbeat-interval=30s                            Interval between performing heartbeats to the cluster ($HEARTBEAT_INTERVAL)\n      --idle-timeout=60s                                  Duration between successful heartbeats before a connection to the cluster is considered unresponsive and closed\n                                                          ($IDLE_TIMEOUT)\n      --readiness-timeout=30s                             Duration the proxy is unable to connect to the backend cluster before it is considered not ready\n                                                          ($READINESS_TIMEOUT)\n      --idempotent-graph                                  If true it will treat all graph queries as idempotent by default and retry them automatically. It may be\n                                                          dangerous to retry some graph queries -- use with caution ($IDEMPOTENT_GRAPH).\n      --num-conns=1                                       Number of connection to create to each node of the backend cluster ($NUM_CONNS)\n      --proxy-cert-file=STRING                            Path to a PEM encoded certificate file with its intermediate certificate chain. This is used to encrypt traffic\n                                                          for proxy clients ($PROXY_CERT_FILE)\n      --proxy-key-file=STRING                             Path to a PEM encoded private key file. This is used to encrypt traffic for proxy clients ($PROXY_KEY_FILE)\n      --rpc-address=STRING                                Address to advertise in the 'system.local' table for 'rpc_address'. It must be set if configuring peer proxies\n                                                          ($RPC_ADDRESS)\n      --data-center=STRING                                Data center to use in system tables ($DATA_CENTER)\n      --tokens=TOKENS,...                                 Tokens to use in the system tables. It's not recommended ($TOKENS)

To pass configuration to cql-proxy, either command-line flags, environment variables, or a configuration file can be used. Using the docker method as an example, the following samples show how the token and database ID are defined with each method.

docker run -p 9042:9042 \\\n  --rm datastax/cql-proxy:v0.1.5 \\\n  --astra-token <astra-token> --astra-database-id <astra-datbase-id>
docker run -p 9042:9042  \\\n  --rm datastax/cql-proxy:v0.1.5 \\\n  -e ASTRA_TOKEN=<astra-token> -e ASTRA_DATABASE_ID=<astra-datbase-id>

Proxy settings can also be passed using a configuration file with the --config /path/to/proxy.yaml flag. This can be mixed and matched with command-line flags and environment variables. Here are some example configuration files:

contact-points:\n  - 127.0.0.1\nusername: cassandra\npassword: cassandra\nport: 9042\nbind: 127.0.0.1:9042\n# ...

or with a Astra token:

astra-token: <astra-token>\nastra-database-id: <astra-database-id>\nbind: 127.0.0.1:9042\n# ...

All configuration keys match their command-line flag counterpart, e.g. --astra-bundle is astra-bundle:, --contact-points is contact-points: etc.

Multi-region failover with DC-aware load balancing policy is the most useful case for a multiple proxy setup.

When configuring peers: it is required to set --rpc-address (or rpc-address: in the yaml) for each proxy and it must match is corresponding peers: entry. Also, peers: is only available in the configuration file and cannot be set using a command-line flag.

Here's an example of configuring multi-region failover with two proxies. A proxy is started for each region of the cluster connecting to it using that region's bundle. They all share a common configuration file that contains the full list of proxies.

Note: Only bundles are supported for multi-region setups.

cql-proxy --astra-bundle astra-region1-bundle.zip --username token --password <astra-token> \\\n  --bind 127.0.0.1:9042 --rpc-address 127.0.0.1 --data-center dc-1 --config proxy.yaml
cql-proxy ---astra-bundle astra-region2-bundle.zip --username token --password <astra-token> \\\n  --bind 127.0.0.2:9042 --rpc-address 127.0.0.2 --data-center dc-2 --config proxy.yaml

The peers settings are configured using a yaml file. It's a good idea to explicitly provide the --data-center flag, otherwise; these values are pulled from the backend cluster and would need to be pulled from the system.local and system.peers table to properly setup the peers data-center: values. Here's an example proxy.yaml:

peers:\n  - rpc-address: 127.0.0.1\n    data-center: dc-1\n  - rpc-address: 127.0.0.2\n    data-center: dc-2

Note: It's okay for the peers: to contain entries for the current proxy itself because they'll just be omitted.

There are three methods for using cql-proxy:

  1. \n

    Build cql-proxy.

    \n
    go build
    \n
    2. \n
  2. \n

    Run with your desired database.

    \n
    3. \n
  1. \n

    Run with your desired database.

    \n
    2. \n

If you wish to have the docker image removed after you are done with it, add --rm before the image name datastax/cql-proxy:v0.1.5.

Using Kubernetes with cql-proxy requires a number of steps:

  1. \n

    Generate a token following the Astra instructions. This step will display your Client ID, Client Secret, and Token; make sure you download the information for the next steps. Store the secure bundle in /tmp/scb.zip to match the example below.

    \n
    2. \n
  2. \n

    Create cql-proxy.yaml. You'll need to add three sets of information: arguments, volume mounts, and volumes. A full example can be found here.

    \n
    3. \n
  1. \n

    Create a configmap. Use the same secure bundle that was specified in the cql-proxy.yaml.

    \n
    kubectl create configmap cql-proxy-configmap --from-file /tmp/scb.zip
    \n
    2. \n
  2. \n

    Check the configmap that was created.

    \n
    kubectl describe configmap cql-proxy-configmap\n  Name:         cql-proxy-configmap\n  Namespace:    default\n  Labels:       <none>\n  Annotations:  <none>\n  Data\n  ====\n  BinaryData\n  ====\n  scb.zip: 12311 bytes
    \n
    3. \n
  3. \n

    Create a Kubernetes deployment with the YAML file you created:

    \n
    kubectl create -f cql-proxy.yaml
    \n
    4. \n
  4. \n

    Check the logs:

    \n
    kubectl logs <deployment-name>
    \n
    5. \n

Drivers that use token-aware load balancing may print a warning or may not work when using cql-proxy. Because cql-proxy abstracts the backend cluster as a single endpoint this doesn't always work well with token-aware drivers that expect there to be at least \"replication factor\" number of nodes in the cluster. Many drivers print a warning (which can be ignored) and fallback to something like round-robin, but other drivers might fail with an error. For the drivers that fail with an error it is required that they disable token-aware or configure the round-robin load balancing policy.

","id":"e7536061-697a-5e22-9f2f-2ff7a472e641","title":"GitHub - datastax/cql-proxy: A client-side CQL proxy/sidecar.","origin_url":"https://github.com/datastax/cql-proxy","url":"https://github.com/datastax/cql-proxy","wallabag_created_at":"2024-11-01T17:26:01+00:00","published_at":null,"published_by":"['datastax']","reading_time":8,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/c2528e3426d98910ed27819e048b4c1081fab2ed2c7adbea6e6a3b1872deb30a/datastax/cql-proxy","tags":["migration","proxy","cassandra","cql"],"description":" cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.The cql-proxy sidecar enables unsu..."}]},{"tag":"grafana","articles":[{"content":"

DataStax Mission Control currently focuses on the lifecycle management and observability of DSE clusters. Install DataStax Mission Control quickly, and then use it to create and manage DataStax Enterprise 6.8.26+ clusters. DataStax Mission Control is composed of a suite of operators which handle the orchestration of automation across regional cluster boundaries. This simplifies management of globally deployed DSE clusters. Define a DSE Cluster custom resource (CR) to create your first DSE Cluster with DataStax Mission Control.

","id":"8c9be501-0d76-5325-81f8-ae484d83122d","title":"DataStax Mission Control :: DataStax Project Mission Control","origin_url":"https://docs.datastax.com/en/mission-control/docs/index.html","url":"https://docs.datastax.com/en/mission-control/docs/index.html","wallabag_created_at":"2023-04-15T01:46:36+00:00","published_at":null,"published_by":null,"reading_time":null,"domain_name":"docs.datastax.com","preview_picture":"https://docs.datastax.com/en/_/img/datastax-docs-banner.png","tags":["kubernetes","datastax","cassandra","grafana"],"description":"DataStax Mission Control currently focuses on the lifecycle management and observability of DSE clusters. Install DataStax Mission Control quickly, and then use it to create and manage DataStax Enterp..."},{"content":"

In Apache Cassandra Lunch #62, guest speaker Sarma Pydipally presented on the Grafana Dashboard for Cassandra. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Register here now!

Grafana Dashboard for Apache Cassandra

We appreciate Sarma Pydipally for taking the time to create a presentation and sharing his knowledge of the Grafana Dashboard and how it can be used with Apache Cassandra.

Grafana Dashboard

Grafana Dashboards is an open-source data and analytics visualization tool. Some key features of Grafana include visualization, queries, alerts, and metrics exploration. It allows users to take data from their time-series databases (TSDBs) and create graphs and visualizations.

Prometheus

Prometheus is an open-source monitoring system. It records and tracks real-time metrics in a time series database. Prometheus supports flexible queries using PromQL and supports real-time alerting. It collects data through a pull model. The Prometheus server queries various data sources from a list at a set frequency and updates the current values based on these queries.

Cassandra

Apache Cassandra is a highly available and highly scalable, open-source, distributed NoSQL database. Cassandra features proven fault-tolerance on hardware or in the cloud.

Grafana Dashboard with Prometheus and Cassandra

To find out more about how to use Grafana Dashboard with Cassandra, please check out Sarma’s video and presentation embedded below. The GitHub repo to the demo featured in the presentation is linked below.

\"Grafana

Demo

https://github.com/sarma1807/Prometheus-Grafana-Cassandra

\n

Cassandra.Link

Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.

We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!

Posted in Data & Analytics, Events | Comments Off on Apache Cassandra Lunch #62: Grafana Dashboard for Apache Cassandra

","id":"efd6b1f7-3f4d-5a7e-8313-2984d4a821c7","title":"Apache Cassandra Lunch #62: Grafana Dashboard for Apache Cassandra - Business Platform Team","origin_url":"https://blog.anant.us/apache-cassandra-lunch-62-grafana-dashboard-for-apache-cassandra/","url":"https://blog.anant.us/apache-cassandra-lunch-62-grafana-dashboard-for-apache-cassandra/","wallabag_created_at":"2022-06-25T14:34:24+00:00","published_at":"2021-09-09T17:41:33+00:00","published_by":"['']","reading_time":1,"domain_name":"blog.anant.us","preview_picture":"https://blog.anant.us/wp-content/uploads/2021/08/SM_Webinar_Deck_Template-KEEP-12-1.png","tags":["cassandra.lunch","grafana.dashboard","cassandra","grafana"],"description":"In Apache Cassandra Lunch #62, guest speaker Sarma Pydipally presented on the Grafana Dashboard for Cassandra. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a de..."},{"content":"

Cassandra DataSource for Grafana

Apache Cassandra Datasource for Grafana. This datasource is to visualise time-series data stored in Cassandra/DSE, if you are looking for Cassandra metrics, you may need datastax/metric-collector-for-apache-cassandra instead.

\"Release\"CodeQL\"\"GitHub

To see the datasource in action, please follow the Quick Demo steps. Documentation is available here

Supports:

Contacts:

Usage

You can find more detailed instructions in the datasource wiki.

Installation

  1. Download the plugin using latest release, please download cassandra-datasource-VERSION.zip and uncompress a file into the Grafana plugins directory (grafana/plugins).
    2. \n
  2. Add the Cassandra DataSource as a datasource at the datasource configuration page.
    3. \n
  3. Configure the datasource specifying contact point and port like \"10.11.12.13:9042\", username and password. It's recommended to use a dedicated user with read-only permissions only to the table you have to access.
    4. \n
  4. Push the \"Save and Test\" button, if there is an error message, check the credentials and connection.
    5. \n

\"Datasource

Panel Setup

There are two ways to query data from Cassandra/DSE, Query Configurator and Query Editor. Configurator is easier to use but has limited capabilities, Editor is more powerful but requires understanding of CQL.

Query Configurator

\"Query

Query Configurator is the easiest way to query data. At first, enter the keyspace and table name, then pick proper columns. If keyspace and table names are given correctly, the datasource will suggest the column names automatically.

After that, you have to specify the ID Value, the particular ID of the data origin you want to show. You may need to enable \"ALLOW FILTERING\" although we recommend to avoid it.

Example Imagine you want to visualise reports of a temperature sensor installed in your smart home. Given the sensor reports its ID, time, location and temperature every minute, we create a table to store the data and put some values there:

CREATE TABLE IF NOT EXISTS temperature (\n    sensor_id uuid,\n    registered_at timestamp,\n    temperature int,\n    location text,\n    PRIMARY KEY ((sensor_id), registered_at)\n);\ninsert into temperature (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd0, 2020-04-01T11:21:59.001+0000, 18, \"kitchen\");\ninsert into temperature (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd0, 2020-04-01T11:22:59.001+0000, 19, \"kitchen\");\ninsert into temperature (sensor_id, registered_at, temperature, location) values (99051fe9-6a9c-46c2-b949-38ef78858dd0, 2020-04-01T11:23:59.001+0000, 20, \"kitchen\");\n

In this case, we have to fill the configurator fields the following way to get the results:

In case of a few origins (multiple sensors) you will need to add more rows. If your case is as simple as that, query configurator will be a good choice, otherwise please proceed to the query editor.

Query Editor

Query Editor is more powerful way to query data. To enable query editor, press \"toggle text edit mode\" button.

\"102781863-a8bd4b80-4398-11eb-8c28-4d06a1f29279\"

Query Editor unlocks all possibilities of CQL including Used-Defined Functions, aggregations etc.

Example using test_data.cql:

SELECT id, CAST(value as double), created_at FROM test.test WHERE id IN (99051fe9-6a9c-46c2-b949-38ef78858dd1, 99051fe9-6a9c-46c2-b949-38ef78858dd0) AND created_at > $__timeFrom and created_at < $__timeTo\n
  1. Follow the order of the SELECT expressions, it's important!
    2. \n
  1. To filter data by time, use $__timeFrom and $__timeTo placeholders as in the example. The datasource will replace them with time values from the panel. Notice It's important to add the placeholders otherwise query will try to fetch data for the whole period of time. Don't try to specify the timeframe on your own, just put the placeholders. It's grafana's job to specify time limits.
    2. \n

\"103153625-1fd85280-4792-11eb-9c00-085297802117\"

Development

Developer documentation

","id":"de9ff26a-f27a-57fb-b10d-bcbd676a6d9d","title":"Apache Cassandra","origin_url":"https://grafana.com/grafana/plugins/hadesarchitect-cassandra-datasource/","url":"https://grafana.com/grafana/plugins/hadesarchitect-cassandra-datasource/","wallabag_created_at":"2022-04-25T21:45:46+00:00","published_at":null,"published_by":null,"reading_time":4,"domain_name":"grafana.com","preview_picture":"https://grafana.com/media/images/meta/grafana-labs-meta-default_1200x630.png","tags":["cassandra","grafana"],"description":"Cassandra DataSource for GrafanaApache Cassandra Datasource for Grafana. This datasource is to visualise time-series data stored in Cassandra/DSE, if you are looking for Cassandra metrics, you may nee..."},{"content":"

\"CassPromGraf_00_Arch.jpg\"

Environment

Following servers are running with CentOS Linux release 7.8.2003 (Core) :
192.168.1.151      eternal1      eternal1.OracleByExample.com\n192.168.1.152      eternal2      eternal2.OracleByExample.com\n192.168.1.153      eternal3      eternal3.OracleByExample.com\n192.168.1.191      PromGraf      PromGraf.OracleByExample.com\n

Apache Cassandra

3 node Cassandra cluster cluster_name: 'id_cluster' version : Apache Cassandra 4.0-beta4 running on following servers :

192.168.1.151      eternal1      eternal1.OracleByExample.com\n192.168.1.152      eternal2      eternal2.OracleByExample.com\n192.168.1.153      eternal3      eternal3.OracleByExample.com\n

We will configure Cassandra's built in metrics-reporter to extract and publish metrics to Prometheus.
We will also configure Prometheus node_exporter on each of our Cassandra nodes to extract and publish metrics to Prometheus.\n

Prometheus & Grafana

We will configure and run Prometheus & Grafana on following server :

192.168.1.191      PromGraf      PromGraf.OracleByExample.com\n

Prometheus will gather and organize all collected metrics into its internal time-series database.
Grafana will consume the metrics from Prometheus and display them in a nice dashboard.

Prometheus Alertmanager

In future, we should implement Alertmanager\n

","id":"3d846338-4870-59b3-8a50-8ba8bf31966d","title":"sarma1807/Prometheus-Grafana-Cassandra","origin_url":"https://github.com/sarma1807/Prometheus-Grafana-Cassandra","url":"https://github.com/sarma1807/Prometheus-Grafana-Cassandra","wallabag_created_at":"2021-07-10T11:53:50+00:00","published_at":null,"published_by":"['sarma1807']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/b79fbf32a62a4dfb02f40b3c4144d102b6d471b6f61ae0f8af63847ce6ea5fbf/sarma1807/Prometheus-Grafana-Cassandra","tags":["cassandra","grafana","prometheus"],"description":" EnvironmentFollowing servers are running with CentOS Linux release 7.8.2003 (Core) :192.168.1.151 eternal1 eternal1.OracleByExample.com\n192.168.1.152 eternal2 eternal2.OracleByExa..."}]}]}},"staticQueryHashes":[]}