uber/peloton

As compute clusters scale, making efficient use of cluster resourcesbecomes very important. Peloton is a Unified Resource Scheduler toco-schedule mixed types of workloads such as batch, stateless andstateful jobs in a single cluster for better resourceutilization. Peloton is designed for web-scale companies like Uberwith millions of containers and tens of thousands of nodes. Pelotonfeatures advanced resource management capabilities such as elasticresource sharing, hierarchical max-min fairness, resource overcommit,workload preemption, etc. Peloton is also Cloud agnostic and can be runin on-premise datacenters or in the Cloud.For more details, please see the Peloton BlogPost andDocumentation.FeaturesElastic Resource Sharing: Support hierachical resource pools toelastically share resources among different teams.Resource Overcommit and Task Preemption: Improve clusterutilization by scheduling workloads using slack resources andpreempting best effort workloads.Optimized for Big Data and Machine Learning: Support GPU and Gangscheduling for Tensorflow. Also support advanced Spark features suchas dynamic resource allocation.High Scalability: Scale to millions of containers and tens ofthousands of nodes.Protobuf/gRPC based API: Support most of the language bindingssuch as golang, java, python, node.js etc.Co-scheduling Mixed Workloads: Support mixed workloads such asbatch, stateless and stateful jobs in a single cluster.Getting StartedSee the Tutorial for step-by-stepinstructions to start a local minicluster and submit a HelloWorld jobto Peloton.ArchitectureTo achieve high-availability and scalability, Peloton uses anactive-active architecture with four separate daemon types: jobmanager, resource manager, placement engine, and host manager. Theinteractions among those daemons are designed so that the dependenciesare minimized and only occur in one direction. All four daemons dependon Zookeeper for service discovery and leader election.Figure , below, shows the high-level architecture of Peloton built ontop of Mesos, Zookeeper, and Cassandra:Components:Peloton consists of the following components:Peloton UI: is web-based UI for managing jobs, tasks, volumes, andresource pools in Peloton.Peloton CLI: is command-line interface for Peloton with similarfunctionality to the web-based interface.Peloton API: uses Protocol Buffers as the interface definitionlanguage and YARPC as its RPC runtime. Peloton UI, Peloton CLI, andother Peloton extensions are all built on top of the same PelotonAPI.Host Manager: abstracts away Mesos details from other Pelotoncomponents. It registers with Mesos via Mesos HTTP API.Resource Manager: maintains the resource pool hierarchy andperiodically calculates the resource entitlement of each resourcepool, which is then used to schedule or preempt taskscorrespondingly.Placement Engine: finds the placement (i.e., task to host mapping)by taking into consideration the job and task constraints as well ashost attributes. Placement engines could be pluggable for differentjob types such as stateful services and batch jobs.Job Manager: handles the lifecycle of jobs, tasks, and volumes. Italso supports rolling upgrades of tasks in a job for long-runningservices.Storage Gateway: provides an abstraction layer on top of differentstorage backends so that we can migrate from one storage backend toanother without significant change in Peloton itself. We have adefault backend for Cassandra built-in, but can extend it to otherbackends.Group Membership: manages the set of Peloton master instances andelects a leader to both register to Mesos as a framework andinstantiate a resource manager.ReferencesUser GuideSee the User Guide for more detailedinformation on how to use Peloton.Peloton CLIPeloton CLI is a command line interface for interacting with Pelotonclusters, such as creating jobs, check job status etc. For detailedPeloton CLI commands and arguments, see CLI Reference.Peloton APIPeloton defines the APIs using Protobuf as the IDL and the clients canaccess Peloton API via gRPC. Peloton supports three client bindings bydefault including Python, Golang and Java. Any other language bindingssupported by gRPC should work as well.See the API Guide for examples of how to usePeloton clients to access the APIs. For detailed Peloton APIdefinition, see the API Reference.ContributingSee the Developer Guide on how to build Pelotonfrom source code.ResourcesDocumentationPeloton DocumentationBlogsPeloton blog from Uber EngineeringTech TalksKubeCon 2018, Seattle (Slides, Video)MesosCon 2018, San FranciscoMesosCon 2017, Los Angeles (Slides, Video)ContactTo contact us, please join our Slack channel.LicensePeloton is under the Apache 2.0 license. See the LICENSE file for details.