thelastpickle/cassandra-reaper

README.md Note: This repo is a fork from the original Reaper project, created by the awesome folks at Spotify. The WebUI has been merged in with support for incremental repairs added.Reaper is a centralized, stateful, and highly configurable tool for running Apache Cassandrarepairs against single or multi-site clusters.The current version supports running Apache Cassandra cluster repairs in a segmented manner,opportunistically running multiple parallel repairs at the same time on different nodeswithin the cluster. Basic repair scheduling functionality is also supported.Reaper comes with a GUI, which if you're running in local mode can be at http://localhost:8080/webui/Please see the Issues section for moreinformation on planned development, and known issues.System OverviewReaper consists of a database containing the full state of the system, a REST-full API,and a CLI tool called spreaper that provides an alternative way to issue commands to a runningReaper instance. Communication with Cassandra nodes in registered clusters is handled through JMX.Reaper system does not use internal caches for state changes regarding running repairs andregistered clusters, which means that any changes done to the storage will reflect to the runningsystem dynamically.You can also run the Reaper with memory storage, which is not persistent, and is meant tobe used only for testing purposes.This project is built on top of Dropwizard:http://dropwizard.io/UsageTo run Cassandra Reaper you need to simply build a project package using Maven, andthen execute the created Java jar file, and give a path to the system configuration fileas the first and only argument. You can also use the provided bin/cassandra-reaper scriptto run the service.When using database based storage, you must setup a PostgreSQL database yourself and configureReaper to use it, or use an embedded H2 database (set the appropriate configuration in the yaml file).The schema will get initialized/upgraded upon startup by FlyWay.When using cassandra based storage, you must setup an Apache Cassandra database yourself and configureReaper to use it. You need to create a keyspace and configure reaper to use it in the yaml file. The schema will be created by Reaper on the first run.Reaper uses the dropwizard-cassandra bundle and full configuration reference is available here : https://github.com/composable-systems/dropwizard-cassandraFind more information on how to use each storage backend in the Configuration section below.For configuring the service, see the available configuration options in later sectionof this readme document.You can call the service directly through the REST API using a tool like curl. You can alsouse the provided CLI tool in bin/spreaper to call the service.Run the tool with -h or --help option to see usage instructions.Debian and RPM PackagesMakeDebian and RPM packages can be built from this project using Make, for example:make allDockerA Docker build environment isalso provided to build the entire project and can be run by usingDocker Compose:docker-compose -f docker-build/docker-compose.yml build \ && docker-compose -f docker-build/docker-compose.yml run buildThe final packages will be located within:./packages/Install RPM or DEB package and run as a serviceInstall the RPM (Fedora based distros like RHEL or Centos) using : sudo rpm -ivh reaper-*.*.*.x86_64.rpmInstall the DEB (Debian based distros like Ubuntu) using : sudo dpkg -i reaper_*.*.*_amd64.debThe yaml file used by the service is located at /etc/cassandra-reaper/cassandra-reaper.yaml and alternate config templates can be found under /etc/cassandra-reaper/configs.It is recommended to create a new file with your specific configuration and symlink it as /etc/cassandra-reaper/cassandra-reaper.yaml to avoid your configuration from being overwritten during upgrades.Adapt the config file to suit your setup and then run sudo service cassandra-reaper start.Log files can be found at /var/log/cassandra-reaper.log and /var/log/cassandra-reaper.err.Stop the service by running : sudo service cassandra-reaper stopConfigurationAn example testing configuration YAML file can be found from within this project repository:src/test/resources/cassandra-reaper.yamlThe configuration file structure is provided by Dropwizard, and help on configuring the server,database connection, or logging, can be found at:http://www.dropwizard.io/1.1.0/docs/manual/configuration.htmlStorage BackendCassandra Reaper can be used with either an ephemeral memory storage or persistent database. Running Reaper with memory storage, which is not persistent, means that allthe registered clusters, column families, and repair runs will be lost upon service restart.The memory based storage is meant to be used for testing purposes only. Enable this type of storage by using the storageType: memory setting in your config file (enabled by default).For persistent relational database storage, you must either setup PostgreSQL or H2. You also need to set storageType: database in the config file.PostgreSQL - The schema will be initialized/upgraded automatically upon startup in the configured database.Make sure to specify the correct credentials in your JDBC settings in cassandra-reaper.yaml to allow objects creation.H2 - the database will automatically created for you under the path configured in cassandra-reaper.yaml. Pleasecomment/uncomment the H2 settings and modify the path as needed or use the cassandra-reaper-h2.yaml as a base.For persistent Apache Cassandra storage, you need to set storageType: cassandra in the config file.You'll also need to fill in the connection details to your Apache Cassandra cluster used to store the Reaper schema (reaper_db by default), in the cassandra: section of the yaml file.A sample yaml file is available in the resource directory for each storage backend :cassandra-reaper-memory.yamlcassandra-reaper-postgres.yamlcassandra-reaper-h2.yamlcassandra-reaper-cassandra.yamlFor configuring other aspects of the service, see the available configuration options in later sectionof this readme document.Reaper SettingsThe Reaper service specific configuration values are:segmentCount:Defines the default amount of repair segments to create for newly registered Cassandrarepair runs (token rings). When running a repair run by the Reaper, each segment isrepaired separately by the Reaper process, until all the segments in a token ringare repaired. The count might be slightly off the defined value, as clusters residingin multiple data centers require additional small token ranges in addition to the expected.You can overwrite this value per repair run, when calling the Reaper.repairParallelism:Defines the default type of parallelism to use for repairs.Repair parallelism value must be one of: "sequential", "parallel", or "datacenter_aware".If you try to use "datacenter_aware" in clusters that don't support it yet (older than 2.0.12),Reaper will fall back into using "sequential" for those clusters.repairIntensity:Repair intensity is a value between 0.0 and 1.0, but not zero. Repair intensity definesthe amount of time to sleep between triggering each repair segment while running a repair run.When intensity is one, it means that Reaper doesn't sleep at all before triggering next segment,and otherwise the sleep time is defined by how much time it took to repair the last segmentdivided by the intensity value. 0.5 means half of the time is spent sleeping, and half running.Intensity 0.75 means that 25% of the total time is used sleeping and 75% running.This value can also be overwritten per repair run when invoking repairs.incrementalRepair:Incremental repair is a boolean value (true | false).Note that this is only supported with the PARALLEL repairParallelism setting.For more details in incremental repair, please refer to the following articlehttp://www.datastax.com/dev/blog/more-efficient-repairsrepairRunThreadCount:The amount of threads to use for handling the Reaper tasks. Have this big enough not to causeblocking in cause some thread is waiting for I/O, like calling a Cassandra cluster through JMX.hangingRepairTimeoutMins:The amount of time in minutes to wait for a single repair to finish. If this timeout is reached,the repair segment in question will be cancelled, if possible, and then scheduled for laterrepair again within the same repair run process.scheduleDaysBetween:Defines the amount of days to wait between scheduling new repairs.The value configured here is the default for new repair schedules, but you can alsodefine it separately for each new schedule. Using value 0 for continuous repairsis also supported.storageType:Whether to use database or memory based storage for storing the system state.Value can be either "memory", "database" or "cassandra".If you are using the recommended (persistent) storage type "database" or "cassandra", you need to definethe database client parameters in a database/cassandra section in the configuration file. See the examplesettings in provided testing configuration in src/test/resources/cassandra-reaper.yaml.jmxPorts:Optional mapping of custom JMX ports to use for individual hosts. The used default JMX portvalue is 7199. CCM users will find IP and port numberto add in ~/.ccm/<cluster>/*/node.conf or by running ccm <node> show.jmxAuth:Optional setting for giving username and password credentials for the used JMX connectionsin case you are using password based JMX authentication with your Cassandra clusters.jmxAuth: username: cassandra password: cassandraenableCrossOrigin:Optional setting which you can set to be "true", if you wish to enable the CORS headersfor running an external GUI application, like this project.autoScheduling:Optional setting to automatically setup repair schedules for all non-system keyspaces in a cluster.If enabled, adding a new cluster will automatically setup a schedule repairfor each keyspace. Cluster keyspaces are monitored based on a configurable frequency,so that adding or removing a keyspace will result in adding / removing the corresponding scheduled repairs.cassandra:Optional setting to provide cassandra store when storageType: cassandraThe sample here shows setup using plain authentication with ssl.cassandra: clusterName: "test" contactPoints: ["127.0.0.1"] keyspace: reaper_db authProvider: type: plainText username: cassandra password: cassandra ssl: type: jdkRefer to resource/cassandra-reaper-cassandra-ssl.yaml for cassandra with AuthNotice that in the server section of the configuration, if you want to bind the serviceto all interfaces, use value "0.0.0.0", or just leave the bindHost line away completely.Using "*" as bind value won't work.Clusters with closed cross DC JMX portsFor security reasons, it is possible that Reaper will be able to access only a single DC nodes through JMX.The allowUnreachableNodes parameter in cassandra-reaper.yaml must then be set to true in order for Reaper to control the repair process through the reachable nodes only.Limitations of this setup are:All keyspaces must be replicated on the reachable DC using NetworkTopologyStrategyReaper won't be able to check the unreachable DC nodes for pending compactions or running repairs, which disables repair overload preventionLeaving allowUnreachableNodes to false will prevent all repair sessions once a single node from the cluster is unreachable.REST APISource code for all the REST resources can be found from package com.spotify.reaper.resources.Ping ResourceGET /pingExpected query parameters: NoneSimple ping resource that can be used to check whether the reaper is running.Cluster ResourceGET /clusterExpected query parameters:seedHost: Limit the returned cluster list based on the given seed host. (Optional)Returns a list of registered cluster names in the service.GET /cluster/{cluster_name}Expected query parameters:limit: Limit the number of repair runs returned. Recent runs are prioritized. (Optional)Returns a cluster object identified by the given "cluster_name" path parameter.POST /clusterExpected query parameters:seedHost: Host name or IP address of the added Cassandraclusters seed host.Adds a new cluster to the service, and returns the newly added cluster object,if the operation was successful.PUT /cluster/{cluster_name}Expected query parameters:seedHost: New host name or IP address used as Cassandra cluster seed.Modifies a cluster's seed host. Comes in handy when the previous seed has left the cluster.DELETE /cluster/{cluster_name}Expected query parameters: NoneDelete a cluster object identified by the given "cluster_name" path parameter.Cluster will get deleted only if there are no schedules or repair runs for the cluster,or the request will fail. Delete repair runs and schedules first before calling this.Repair Run ResourceGET /repair_runOptional query parameters:state: Comma separated list of repair run state names. Only names found incom.spotify.reaper.core.RunState are accepted.Returns a list of repair runs, optionally fetching only the ones with state state.GET /repair_run/{id}Expected query parameters: NoneReturns a repair run object identified by the given "id" path parameter.GET /repair_run/cluster/{cluster_name} (com.spotify.reaper.resources.RepairRunResource)Expected query parameters: NoneReturns a list of all repair run statuses found for the given "cluster_name" path parameter.POST /repair_runExpected query parameters:clusterName: Name of the Cassandra cluster.keyspace: The name of the table keyspace.tables: The name of the targeted tables (column families) as comma separated list.If no tables given, then the whole keyspace is targeted. (Optional)owner: Owner name for the run. This could be any string identifying the owner.cause: Identifies the process, or cause the repair was started. (Optional)segmentCount: Defines the amount of segments to create for repair run. (Optional)repairParallelism: Defines the used repair parallelism for repair run. (Optional)intensity: Defines the repair intensity for repair run. (Optional)incrementalRepair: Defines if incremental repair should be done. [true/false] (Optional)PUT /repair_run/{id}Expected query parameters:state: New value for the state of the repair run.Possible values for given state are: "PAUSED" or "RUNNING".Starts, pauses, or resumes a repair run identified by the "id" path parameter.Can also be used to reattempt a repair run in state "ERROR", picking up where it left off.DELETE /repair_run/{id}Expected query parameters:owner: Owner name for the run. If the given owner does not match the stored owner,the delete request will fail.Delete a repair run object identified by the given "id" path parameter.Repair run and all the related repair segments will be deleted from the database.Repair Schedule ResourceGET /repair_scheduleExpected query parameters:clusterName: Filter the returned schedule list based on the givencluster name. (Optional)keyspaceName: Filter the returned schedule list based on the givenkeyspace name. (Optional)Returns all repair schedules present in the ReaperGET /repair_schedule/{id}Expected query parameters: NoneReturns a repair schedule object identified by the given "id" path parameter.POST /repair_scheduleExpected query parameters:clusterName: Name of the Cassandra cluster.keyspace: The name of the table keyspace.tables: The name of the targeted tables (column families) as comma separated list.If no tables given, then the whole keyspace is targeted. (Optional)owner: Owner name for the schedule. This could be any string identifying the owner.segmentCount: Defines the amount of segments to create for scheduled repair runs. (Optional)repairParallelism: Defines the used repair parallelism for scheduled repair runs. (Optional)intensity: Defines the repair intensity for scheduled repair runs. (Optional)incrementalRepair: Defines if incremental repair should be done. [true/false] (Optional)scheduleDaysBetween: Defines the amount of days to wait between scheduling new repairs.For example, use value 7 for weekly schedule, and 0 for continuous.scheduleTriggerTime: Defines the time for first scheduled trigger for the run.If you don't give this value, it will be next mid-night (UTC).Give date values in ISO format, e.g. "2015-02-11T01:00:00". (Optional)DELETE /repair_schedule/{id}Expected query parameters:owner: Owner name for the schedule. If the given owner does not match the stored owner,the delete request will fail.Delete a repair schedule object identified by the given "id" path parameter.Repair schedule will get deleted only if there are no associated repair runs for the schedule.Delete all the related repair runs before calling this endpoint.Building and Running ReaperTo build Reaper without rebuilding the UI, run the following command :mvn clean packageTo only regenerate the UI (requires npm and bower) :mvn generate-sources -Pbuild-uiTo rebuild both the UI and Reaper :mvn clean package -Pbuild-uiTo build the docker image :mvn clean package docker:buildRunning ReaperAfter modifying the resource/cassandra-reaper.yaml config file, Reaper can be started using the following command line :java -jar target/cassandra-reaper-X.X.X.jar server resource/cassandra-reaper.yamlOnce started, the UI can be accessed through : http://127.0.0.1:8080/webui/Reaper can also be accessed using the REST API exposed on port 8080, or using the command line tool bin/spreaperRunning a Dockerized Reaper ExampleBuild Reaper Docker ImageFirst, build the Docker image and add it to your local image cache using thecassandra-reaper:latest tag:mvn clean package docker:buildStart Docker EnvironmentFirst, start the Cassandra cluster:docker-compose up cassandraYou can use the nodetool Docker Compose service to check on the Cassandranode's status:docker-compose run nodetool statusOnce the Cassandra node is online and accepting CQL connections,create the required reaper_db Cassandra keyspace to allow Reaper to saveits cluster and scheduling data.By default, the reaper_db keyspace is created using a replication factorof 1. To change this replication factor, provide the intended replicationfactor as an optional argument:docker-compose run initialize-reaper_db [$REPLICATION_FACTOR]Wait a few moments for the reaper_db schema change to propagate,then start Reaper:docker-compose up reaperAccess The EnvironmentOnce started, the UI can be accessed through:http://127.0.0.1:8080/webui/When adding the Cassandra node to the Reaper UI, use the IP address found via:docker-compose run nodetool statusThe helper cqlsh Docker Compose service has also been included:docker-compose run cqlshDestroying the Docker EnvironmentWhen terminating the infrastructure, use the following command to stopall related Docker Compose services:docker-compose downTo completely clean up all persistent data, delete the ./data/ directory:rm -rf ./data/