In our production Cassandra cluster (At the time of this post was DSE 5.1.18), we were seeing the Total Blocked count for the stat Native-transport-requests increasing regularly on each node. The Native-transport-request metric represents any CQL request.
Here is the view we were seeing on each node in OpsCenter:
The issue was because we were using the default value of 128 for max_queued_native_transport_requests which is set in cassandra-env.sh.
Once the limit is reached for max queued requests any new requests are dropped. Increasing the queue size would allow more space for backed up requests to queue rather than just being dropped.
The solution was to increase the queue size from 128 to 1024 by adding the following line to the end of the cassandra-env.sh file on each node:
JVM_OPTS="$JVM_OPTS -Dcassandra.max_queued_native_transport_requests=1024"
Note: You will need to restart each node after making this change.
This change resolved the issue for us and the Total Blocked count has been significantly reduced on each node:
Note: If you still have issues you can continue to increase the queue size, the DataStax documentation suggests 3074, however I would recommend increasing the value incrementally and evaluating the impact.
I hope that helps!
