Alluxio Mesos Meetup - SMACK to SMAACK

Alluxio Mesos Meetup - SMACK to SMAACK SlideShare Explore You Successfully reported this slideshow.Alluxio Mesos Meetup - SMACK to SMAACKUpcoming SlideShareLoading in …5× 0 Comments 1 Like Statistics Notes Streaming Analytics , Technology Manager at Trivadis at Trivadis No DownloadsNo notes for slide 1. © 2016 Mesosphere, Inc. All Rights Reserved.From SMACK toSMAACKAlluxio meets DC/OSJörg Schad, MesosphereAdit Madan, Alluxio#smack @Alluxio @dcos @joerg_schad @madanadit 2. © 2017 Mesosphere, Inc. All Rights Reserved.20% OFFMCDCOS20September 13th - 15th● Dedicated Tracks● MesosCon University● Town Halls● HackathonAccelerating Spark workloads in a Mesosenvironment with Alluxio, 09/15, 11AM 3. © 2017 Mesosphere, Inc. All Rights Reserved. 3Fast DataBatch Event ProcessingMicro-BatchDays Hours Minutes Seconds MicrosecondsSolves problems using predictive and prescriptive analyticsReports what has happened using descriptive analyticsPredictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations 4. © 2017 Mesosphere, Inc. All Rights Reserved. 4The SMACK StackEVENTSUbiquitous data streamsfrom connected devicesINGESTApache KafkaSTOREApache SparkANALYZEApache CassandraACTAkkaIngest millions of eventsper secondDistributed & highlyscalable databaseReal-time and batchprocess dataVisualize data and builddata driven applicationsMesos/ DC/OSSensorsDevicesClients 5. © 2017 Mesosphere, Inc. All Rights Reserved. 5Datacenter 6. © 2017 Mesosphere, Inc. All Rights Reserved. 6NAIVE APPROACHTypical Datacentersiloed, over-provisioned servers,low utilizationIndustry Average12-15% utilizationmySQLmicroserviceCassandraSpark/HadoopKafka 7. © 2017 Mesosphere, Inc. All Rights Reserved. 7 8. © 2017 Mesosphere, Inc. All Rights Reserved. 8MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTSTypical Datacentersiloed, over-provisioned servers,low utilizationMesos/ DC/OSautomated schedulers, workload multiplexing onto thesame machinesmySQLmicroserviceCassandraSpark/HadoopKafka 9. Datacenter Operating System (DC/OS)Distributed Systems Kernel (Mesos)DC/OS ENABLES MODERN DISTRIBUTED APPSBig Data + Analytics EnginesMicroservices (in containers)StreamingBatchMachine LearningAnalyticsFunctions &LogicSearchTime SeriesSQL / NoSQLDatabasesModern App ComponentsAny Infrastructure (Physical, Virtual, Cloud)9 10. © 2017 Mesosphere, Inc. All Rights Reserved. 10The SMACK StackEVENTSUbiquitous data streamsfrom connected devicesINGESTApache KafkaSTOREApache SparkANALYZEApache CassandraACTAkkaIngest millions of eventsper secondDistributed & highlyscalable databaseReal-time and batchprocess dataVisualize data and builddata driven applicationsMesos/ DC/OSSensorsDevicesClients 11. © 2017 Mesosphere, Inc. All Rights Reserved. 11The SMACK StackEVENTSUbiquitous data streamsfrom connected devicesINGESTApache KafkaSTOREApache SparkANALYZEApache CassandraACTAkkaIngest millions of eventsper secondDistributed & highlyscalable databaseReal-time and batchprocess dataVisualize data and builddata driven applicationsMesos/ DC/OSSensorsDevicesClients 12. © 2016 Mesosphere, Inc. All Rights Reserved.BIG DATA ECOSYSTEM YESTERDAY© 2017 Alluxio 12 13. © 2016 Mesosphere, Inc. All Rights Reserved.BIG DATA ECOSYSTEM TODAY© 2017 Alluxio……13 14. © 2016 Mesosphere, Inc. All Rights Reserved.BIG DATA ECOSYSTEM ISSUES© 2017 Alluxio……14 15. © 2017 Mesosphere, Inc. All Rights Reserved. 15The SMAACK StackEVENTSUbiquitous data streamsfrom connected devicesINGESTApache KafkaSTOREApache SparkANALYZEApache CassandraACTAkkaIngest millions of eventsper secondDistributed & highlyscalable databaseReal-time and batchprocess dataVisualize data and builddata driven applicationsMesos/ DC/OSSensorsDevicesClientsAlluxio 16. © 2017 Mesosphere, Inc. All Rights Reserved. 16© 2017 Alluxio 17. © 2016 Mesosphere, Inc. All Rights Reserved.BIG DATA ECOSYSTEM WITH ALLUXIO……FUSE Compatible FileSystem InterfaceHadoop Compatible FileSystem InterfaceNative Key-ValueInterfaceNative File SystemInterfaceHDFS Interface Amazon S3 Interface Swift Interface GlusterFS Interface© 2017 Alluxio 17 18. © 2016 Mesosphere, Inc. All Rights Reserved.BIG DATA ECOSYSTEM WITH ALLUXIO……FUSE Compatible FileSystem InterfaceHadoop Compatible FileSystem InterfaceNative Key-ValueInterfaceNative File SystemInterfaceHDFS Interface Amazon S3 Interface Swift Interface GlusterFS InterfaceEnabling Application to Access Data from anyStorage System at Memory-speed© 2017 Alluxio 18 19. © 2016 Mesosphere, Inc. All Rights Reserved.WHY ALLUXIO© 2017 AlluxioCo-located compute and data with memory-speed access to dataVirtualized across different storage systems under a unified namespaceScale-out architectureFile system API, software only19 20. © 2016 Mesosphere, Inc. All Rights Reserved.ALLUXIO BENEFITS© 2017 AlluxioUnificationNew workflows acrossany data in any storagesystemOrders of magnitudeimprovement in runtimeChoice in compute andstorage – grow eachindependently, buyonly what is neededPerformance Flexibility20 21. © 2017 Mesosphere, Inc. All Rights Reserved. 21© 2017 Alluxio 22. © 2016 Mesosphere, Inc. All Rights Reserved. 22WHY DATA SERVICES ON DC/OS?On-demand provisioning123Simplified operationsElastic data infrastructure● Single command install of services● Runtime software upgrade● Runtime application settings update● Monitoring & metrics● Managed persistent storage volumes● Data services and containerized apps share resources● Deploy instances with different versions on the sameinfrastructure● Resize instances● Add more instances© 2017 Alluxio 23. © 2016 Mesosphere, Inc. All Rights Reserved. 23ALLUXIO ON MESOSPHERE DC/OSFast, On-demand Unified Data at Memory Speed for AnalyticsAlluxioMesosphere DC/OSAny InfrastructureBuild apps once in DC/OS, andrun anywhereRuns distributed apps anywhere assimply as running apps on your laptopUnify Data at Memory Speed Unify Data at Memory Speed© 2017 Alluxio 24. © 2016 Mesosphere, Inc. All Rights Reserved. 24ALLUXIO ON MESOSPHERE DC/OSFast, On-demand Unified Data at Memory Speed for Analytics© 2017 Alluxio 25. © 2016 Mesosphere, Inc. All Rights Reserved.WHY ALLUXIO ON MESOSPHERE DC/OS?● Without Mesosphere DC/OS, provisioning of infrastructure is tedious○ Mesosphere DC/OS automates app & cluster provisioning, management & elastic scaling● Alluxio brings○ A unified view of data across disparate storage systems○ High performance & predictable SLA for analytics workloads● Benefits include:○ Process data in your existing cluster faster with Spark and other analytics frameworks○ Process data from hybrid cloud storage systems (HDFS, S3, On-prem Object Stores etc)© 2017 Alluxio 25 26. © 2016 Mesosphere, Inc. All Rights Reserved. 26BIG DATA STACK WITH ALLUXIO ON MESOSPHERE DC/OSFast, On-demand Unified Data at Memory Speed for AnalyticsMesosContainer Orchestration Management & Monitoring Tools Apps UniverseSecurity Advanced Operations Multitenancy Adv. Network & StorageUnifying Data at Memory Speed© 2017 Alluxio 27. © 2017 Mesosphere, Inc. All Rights Reserved. 27© 2017 AlluxioDEMO 28. © 2016 Mesosphere, Inc. All Rights Reserved.WHAT HAPPENED?● Alluxio scheduler (developed using the DC/OS SDK) launched as a Marathon application○ Marathon manages and restarts the scheduler in case of failures○ Scheduler consists of YAML + scripting● Alluxio scheduler launched master and worker processes○ Scheduler manages the configured number of instances even w/ failures● Configuration changes take effect on the fly○ Scaled up the worker instances© 2017 Alluxio 28 29. © 2016 Mesosphere, Inc. All Rights Reserved.GET STARTED TODAYRead:● Mesosphere Blog: http://ow.ly/ou0530ax9aM● Alluxio Blog: http://ow.ly/ILOZ30ax8YETry it out:● Install Alluxio from DC/OS UniverseQuestions?© 2017 Alluxio 29 Recommended Teacher Tech Tips WeeklyOnline Course - LinkedIn Learning Insights from a Content MarketerOnline Course - LinkedIn Learning Educational Technology for Student SuccessOnline Course - LinkedIn Learning The Architecture of Decoupling Compute and Storage with AlluxioAlluxio, Inc. Best Practices for Using Alluxio with SparkAlluxio, Inc. Spark Pipelines in the Cloud with AlluxioAlluxio, Inc. Accelerating Spark Workloads in a Mesos Environment with AlluxioAlluxio, Inc. Accelerating Spark Workloads in an Apache Mesos Environment with AlluxioAlluxio, Inc. Best Practices for Using Alluxio with SparkAlluxio, Inc. Best Practices for Using Alluxio with SparkAlluxio, Inc. About Blog Terms Privacy Copyright LinkedIn Corporation © 2018 Public clipboards featuring this slideNo public clipboards found for this slideSelect another clipboard ×Looks like you’ve clipped this slide to already.Create a clipboardYou just clipped your first slide! Clipping is a handy way to collect important slides you want to go back to later. Now customize the name of a clipboard to store your clips. Description Visibility Others can see my Clipboard