Illustration Image

Cassandra.Link

The best knowledge base on Apache Cassandra®

Helping platform leaders, architects, engineers, and operators build scalable real time data platforms.

10/31/2017

Reading time:2 min

Cassandra StatD Agent

by John Doe

Thanks to UnderSiege for getting this project off the ground.REQUIREMENTSMavenJava 7BUILDCheck which metrics library cassandra is using by looking incassandra/lib/metrics-core*, verify that pom.xml points to thesame exact version. For example, if you have metrics-core-2.2.0.jar,make sure pom.xml has 2.2.0.mvn packageAlternatively, grab the binary from bintray:curl -L http://dl.bintray.com/lookout/systems/com/github/lookout/metrics/agent/1.2/agent-1.2.jar -o agent-1.2.jarINSTALLCopy the statsd library from the .m2 folder to cassandra/lib.Add the following to your cassandra startup script:Copy the agent-1.2.jar to a new directory cassandra/pluginsChange cassandra startup to add this agent. This can be done ina stock install by adding the following to /etc/default/cassandra:export JVM_OPTS="-javaagent:/usr/share/cassandra/plugins/agent-1.2.jar=localhost"Note the '=localhost' at the end. This supports the following syntaxes:hostname:port@intervalFor example:your.statsd.host.com:9999@60The default port is 8125 and the default interval is 10 (seconds); thesecan be omitted. IPV6 is also supported with the following syntax:[2001:db8::1]:8888@300which means connect to 2001:db8::1 on port 8888 every 300 seconds.REPORTINGA log message will be added to the system.log at startup toconfirm that everything is running, it looks like this:INFO [metrics-statsd-thread-1] 2014-12-19 19:05:37,120 StatsdReporter.java:65 - Statsd reporting to host localhost port 8125 every 10 secondsWHAT GETS REPORTEDLots of stuff:Gossip statistics:gossip.score., which help decide who is closer/faster for queriesgossip.severity, which indicates how busy this node is self-reporting to othersPer table statistics:cfstats...ReadCountcfstats...WriteCountcfstats...RecentReadLatencyMicroscfstats...RecentWriteLatencyMicroscfstats...TombstonesPerSlicecfstats...estimatedKeysThe last one is great for monitoring general trends, but of course don'trely on that number to be very accurate.PHI reporterAlso supported is the currently-experimental PHI reporter, in PHI.,coming to a Cassandra cluster near you soon.JVM GC metricsAnything else registered with yammer-metricsDEBUGGINGNot working? There's a lot of tracing and debugging available. Change thelog4j-server.properties and add something like this to get extremely detailedtraces of what it's doing in the server.log.log4j.logger.com.github.lookout.metrics.agent.generators=TRACETODOErrors that happen during startup are not reported as well as they shouldbe, mostly because the logging system is not active during startup. The logmessage is only generated when the actual metrics collector has run for thefirst time.

Illustration Image

Thanks to UnderSiege for getting this project off the ground.

REQUIREMENTS

  • Maven
  • Java 7

BUILD

Check which metrics library cassandra is using by looking in cassandra/lib/metrics-core*, verify that pom.xml points to the same exact version. For example, if you have metrics-core-2.2.0.jar, make sure pom.xml has 2.2.0.

mvn package

Alternatively, grab the binary from bintray:

curl -L http://dl.bintray.com/lookout/systems/com/github/lookout/metrics/agent/1.2/agent-1.2.jar -o agent-1.2.jar

INSTALL

Copy the statsd library from the .m2 folder to cassandra/lib. Add the following to your cassandra startup script:

Copy the agent-1.2.jar to a new directory cassandra/plugins

Change cassandra startup to add this agent. This can be done in a stock install by adding the following to /etc/default/cassandra:

export JVM_OPTS="-javaagent:/usr/share/cassandra/plugins/agent-1.2.jar=localhost"

Note the '=localhost' at the end. This supports the following syntaxes: hostname:port@interval For example: your.statsd.host.com:9999@60 The default port is 8125 and the default interval is 10 (seconds); these can be omitted. IPV6 is also supported with the following syntax: [2001:db8::1]:8888@300 which means connect to 2001:db8::1 on port 8888 every 300 seconds.

REPORTING

A log message will be added to the system.log at startup to confirm that everything is running, it looks like this:

INFO [metrics-statsd-thread-1] 2014-12-19 19:05:37,120 StatsdReporter.java:65 - Statsd reporting to host localhost port 8125 every 10 seconds

WHAT GETS REPORTED

Lots of stuff:

  • Gossip statistics: gossip.score., which help decide who is closer/faster for queries gossip.severity, which indicates how busy this node is self-reporting to others
  • Per table statistics: cfstats...ReadCount cfstats...WriteCount cfstats...RecentReadLatencyMicros cfstats...RecentWriteLatencyMicros cfstats...TombstonesPerSlice cfstats...estimatedKeys The last one is great for monitoring general trends, but of course don't rely on that number to be very accurate.
  • PHI reporter Also supported is the currently-experimental PHI reporter, in PHI., coming to a Cassandra cluster near you soon.
  • JVM GC metrics
  • Anything else registered with yammer-metrics

DEBUGGING

Not working? There's a lot of tracing and debugging available. Change the log4j-server.properties and add something like this to get extremely detailed traces of what it's doing in the server.log.

log4j.logger.com.github.lookout.metrics.agent.generators=TRACE

TODO

Errors that happen during startup are not reported as well as they should be, mostly because the logging system is not active during startup. The log message is only generated when the actual metrics collector has run for the first time.

Related Articles

cassandra
tools
admin-monitor

Upgrading a Large Cassandra Cluster with cstar | Official Pythian® Blog

Valerie Parham-Thompson

1/27/2021

Checkout Planet Cassandra

Claim Your Free Planet Cassandra Contributor T-shirt!

Make your contribution and score a FREE Planet Cassandra Contributor T-Shirt! 
We value our incredible Cassandra community, and we want to express our gratitude by sending an exclusive Planet Cassandra Contributor T-Shirt you can wear with pride.

Join Our Newsletter!

Sign up below to receive email updates and see what's going on with our company

Explore Related Topics

AllKafkaSparkScyllaSStableKubernetesApiGithubGraphQl

Explore Further

admin-monitor