Cassandra repairs consist of comparing data from between replica nodes, identifying inconsistencies, and streaming the latest value for mismatched data. We can't compare an entire cassandra database value by value so we create Merkle trees to identify inconsistencies and then we stream them.
Repairs are expensive; CPU is needed to generate the Merkle trees and networking / io is needed to stream missing data. Usually repairs also trigger lots of compactions if they have not been run for a while (especially when there's a lot of inconsistent data and leveled / date tiered compaction strategies are being used).
The OpsCenter repair service splits up the repair job into lots of little slices (256 per table) and runs them around the clock, turning a heavy, manual, weekly operation into an automatic, constant job. Clusters will see higher consistent cpu utilization / load when the repair service is on instead of a big spike once per week.
Steps in a repair
Each repair session will be identified by a UUID (for example #0d4544b0-8fc9-11e5-a498-4b9679ec178d). The following are the logs from a healthy repair. Notice all the messages are INFO messages, there are no WARN or ERROR.
Repair sessions have repair jobs and jobs which occur for each table in the session.
Repeat for every table:
1) RepairJob.java Request merkle trees
2) RepairSession.java Receive merkle trees
3) Differencer.java Check for inconsistencies
4) OptionalStreamingRepairTask.java Stream differences--if any
5) RepairSession.java is fully synced
6) StorageService.java Repair session for range (,] finished
Summarizing repair logs - clean run
To group the tasks use the following bash foo:
$ cat system.log| grep "11ff9870-8fc9-11e5-a498-4b9679ec178d" | sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Source/'|sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Target/' | awk '{ split($3,a,":"); $2=a[0] ; $3=""; $4=""; print }'|uniq -c
In this case there was no streaming and all the jobs complete successfully for the range.
1 INFO RepairSession.java:260 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] new session: will sync /Source, /Target1, /Target2, ...... on range (4393112290973329820,4394202908924102592] for OpsCenter.[rollups86400, events_timeline, rollups7200, events, bestpractice_results, backup_reports, settings, rollups60, rollups300, pdps] 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups86400 (to [/Source, /Target, ...]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups86400 from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups86400 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups86400 is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for events_timeline (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for events_timeline from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for events_timeline 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] events_timeline is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups7200 (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups7200 from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups7200 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups7200 is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for events (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for events from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for events 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] events is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for bestpractice_results (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for bestpractice_results from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for bestpractice_results 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] bestpractice_results is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for backup_reports (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for backup_reports from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for backup_reports 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] backup_reports is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for settings (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for settings from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for settings 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] settings is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups60 (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups60 from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups60 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups60 is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups300 (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups300 from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups300 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups300 is fully synced 1 INFO RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for pdps (to [/Source, /Target, ... ]) 9 INFO RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for pdps from /Source 36 INFO Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for pdps 1 INFO RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] pdps is fully synced 1 INFO RepairSession.java:299 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] session completed successfully 1 INFO StorageService.java:3001 - Repair session 11ff9870-8fc9-11e5-a498-4b9679ec178d for range (4393112290973329820,4394202908924102592] finished
Summarizing repair logs - errors
Now let's look at a repair session with some errors.
$ cat system.log| grep "0fb1b0d0-8fc9-11e5-a498-4b9679ec178d" | sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Source/'|sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Target/' | awk '{ split($3,a,":"); $2=a[0] ; $3=""; $4=""; print }'|uniq -c
1 INFO RepairSession.java:260 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] new session: will sync /Source, /Target, ... on range (4393112290973329820,4394202908924102592] for keyspace.[table1, table2, ...] 1 INFO RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for table1 (to [/Source, /Target, ...]) 9 INFO RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for table1 from /Source 36 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for datasources 1 INFO RepairSession.java:237 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] datasources is fully synced 1 INFO RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for table1 (to [/Source, /Target, ...]) 9 INFO RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for table1 from /Source 36 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex_error 1 INFO RepairSession.java:237 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] tablex_error is fully synced 1 INFO RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for tablex (to [/Source, /Target, ]) 9 INFO RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source 9 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 2 INFO Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex 1 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 3 INFO Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex 1 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 3 INFO Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex 1 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 1 INFO Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex 3 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 3 INFO Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex 1 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 6 INFO Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex 2 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 11 INFO StreamingRepairTask.java:81 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Forwarding streaming repair of 1 ranges to /Source (to be streamed with /Target) 2 INFO StreamingRepairTask.java:68 - [streaming task #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Performing streaming repair of 1 ranges with /Source 1 INFO StreamingRepairTask.java:81 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Forwarding streaming repair of 1 ranges to /Source (to be streamed with /Target) 4 INFO StreamingRepairTask.java:68 - [streaming task #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Performing streaming repair of 1 ranges with /Source 1 INFO RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for tablex (to [/Source, /Target, ...]) 1 INFO RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source 1 INFO StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target 1 INFO RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source 2 INFO StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target 7 INFO RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source 36 INFO Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex 1 INFO RepairSession.java:237 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] tablex is fully synced (1 remaining column family to sync for this session) 1 INFO StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target 1 INFO RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for tablex_processed (to [/Source, /Target, ...]) 2 INFO StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target 2 INFO RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source 1 ERROR RepairSession.java:303 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] session completed with the following error 1 org.apache.cassandra.exceptions.RepairException: keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target 1 java.lang.RuntimeException: on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target 1 Caused #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target 1 ERROR StorageService.java:3008 - Repair session 0fb1b0d0-8fc9-11e5-a498-4b9679ec178d for range (4393112290973329820,4394202908924102592] failed with error org.apache.cassandra.exceptions.RepairException: [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target 1 java.util.concurrent.ExecutionException: #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target 1 Caused [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target 1 Caused #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
Notice the error message
ERROR StorageService.java:3008 - Repair session 0fb1b0d0-8fc9-11e5-a498-4b9679ec178d for range (4393112290973329820,4394202908924102592] failed with error org.apache.cassandra.exceptions.RepairException: [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
Streaming for the range (4393112290973329820, 4394202908924102592] between nodes Source and Target failed and caused the repair session to fail. The OpsCenter repair service will retry this session again (up to a configurable threshold) on failure.
What happened?
The failure may have been due to a networking problem, an sstable corruption, etc. To get more information, we can 1) check our logs at the Target repair node for additional errors and 2) run the slice again and see if it works.
Check the target
Checking the logs at the target, there are no failures associated with this repair session which means there must have been a problem on the stream receiving end.
$ cat system.log| grep "0fb1b0d0-8fc9-11e5-a498-4b9679ec178d" | sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Source/'|sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Target/' | awk '{ split($3,a,":"); $2=a[0] ; $3=""; $4=""; print }'|uniq -c
1 INFO Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
1 INFO Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
1 INFO Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
2 INFO StreamingRepairTask.java:68 - [streaming task #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Performing streaming repair of 1 ranges with /Source
1 INFO Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
1 INFO Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
Run the slice
In this case we can run:
nodetool repair -par -st 4393112290973329820 -et 4394202908924102592
here is the result (notice the networking problem was temporary and now the repair slice succeeds!):
]# nodetool repair -par -st 4393112290973329820 -et 4394202908924102592 [2015-11-23 21:36:44,138] Starting repair command #1, repairing 1 ranges for keyspace dse_perf (parallelism=PARALLEL, full=true) [2015-11-23 21:36:46,086] Repair session 4aa7a290-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished [2015-11-23 21:36:46,086] Repair command #1 finished [2015-11-23 21:36:46,095] Starting repair command #2, repairing 1 ranges for keyspace keyspace (parallelism=PARALLEL, full=true) [2015-11-23 21:36:47,967] Repair session 4bc6f540-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished [2015-11-23 21:36:47,968] Repair command #2 finished [2015-11-23 21:36:47,979] Nothing to repair for keyspace 'system' [2015-11-23 21:36:47,985] Starting repair command #3, repairing 1 ranges for keyspace keyspace (parallelism=PARALLEL, full=true) [2015-11-23 21:36:53,100] Repair session 4ce92e20-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished [2015-11-23 21:36:53,102] Repair command #3 finished [2015-11-23 21:36:53,112] Starting repair command #4, repairing 1 ranges for keyspace dse_system (parallelism=PARALLEL, full=true) [2015-11-23 21:36:53,979] Repair session 4fe50900-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished [2015-11-23 21:36:53,979] Repair command #4 finished [2015-11-23 21:36:53,987] Starting repair command #5, repairing 1 ranges for keyspace keyspace (parallelism=PARALLEL, full=true) [2015-11-23 21:36:58,390] Repair session 507477c0-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished [2015-11-23 21:36:58,390] Repair command #5 finished [2015-11-23 21:36:58,399] Starting repair command #6, repairing 1 ranges for keyspace OpsCenter (parallelism=PARALLEL, full=true) [2015-11-23 21:37:11,448] Repair session 531931f0-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished [2015-11-23 21:37:11,448] Repair command #6 finished [2015-11-23 21:37:11,458] Starting repair command #7, repairing 1 ranges for keyspace system_traces (parallelism=PARALLEL, full=true) [2015-11-23 21:37:11,878] Repair session 5ae2e890-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished [2015-11-23 21:37:11,878] Repair command #7 finished
There are a couple of known streaming issues to keep an eye on.
CASSANDRA-10791 and CASSANDRA-10012 which may cause streaming errors. If you are on an affected version, upgrade. If you encounter a reproducible streaming error and can't find the particular stack trace in an existing jira, open a new one.
A corruption
In a different repair session we see a different repair error, this time it refers to a specific sstable.
WARN [STREAM-IN-/x.x.x.x] 2015-11-20 20:55:45,529 StreamSession.java:625 - [Stream #114cea40-8fc9-11e5-ae1c-4b5d0d7247d3] Retrying for following error
java.lang.RuntimeException: Last written key DecoratedKey(4393675392884570836, 000b313032333432333334303000000343504600) >= current key DecoratedKey(918610503192973903, 00102941d767895c11e5b5dfaadf7c0db7b80000087765627369746573ff) writing into /cassandra/data/mykeyspace/table1-bd0990407e6711e5a4cae7fe9b813e81/mykeyspace-table1-tmp-ka-7890-Data.db
at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:164) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:261) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:168) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:89) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48) [cassandra-all-2.1.11.908.jar:2.1.11.908]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [cassandra-all-2.1.11.908.jar:2.1.11.908]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56) [cassandra-all-2.1.11.908.jar:2.1.11.908]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250) [cassandra-all-2.1.11.908.jar:2.1.11.908]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]
When there is an sstable corruption (due to disk failures or possibly even a bug), the procedure is to run nodetool scrub on the sstable which will correct the corruption.
In this case the issue was due to CASSANDRA-9133 which was fixed in 2.0.15 so an upgrade was also in order!
Summary, stay repaired!
Repairs can fail due to networking issues or sstable corruptions. The former are usually short lived and will go away on retry; the latter are more rare and require admin intervention in the form of running nodetool scrub.
Remember, repairs aren't things you run when your cluster is broken; they are a mandatory anti-entropy administrative task (like an oil change) that keeps your cluster healthy. In many cases, running a repair on an unhealthy cluster will just make things worse.
Hopefully this post will help you understand how repairs work, how to troubleshoot the repair service, and keep your production cluster happy and your database boring. Enjoy!
