Upgrade to Pro — share decks privately, control downloads, hide ads and more …

mesos.devoxx.2014

Sam Bessalah
November 12, 2014

 mesos.devoxx.2014

Sam Bessalah

November 12, 2014
Tweet

More Decks by Sam Bessalah

Other Decks in Technology

Transcript

  1. #Devoxx #DV14 #mesos #docker @samklr Issues with static partitionning -

    Increased Complexity : Machines/vm sprawl Software failures gets harder to handle as machines number grow - Huge overheard, on development, - Planning for failure and maintenance gets harder - Not really developer friendly. Long cycles to deployments.
  2. #Devoxx #DV14 #mesos #docker @samklr Enters Google stage left ...

    Google “Borg” .Unofficial name. Moved to a new system called Omega Bundles all the cluster’s machines into a single pooled view of resources. Evolved to Omega.
  3. #Devoxx #DV14 #mesos #docker @samklr Mesos : Level of indirection

    1. Run multiple types of distributed systems on the same cluster of machines, and dynamically share resources more eficiently. 2. Provide failure detection, task distribution, task tracking, task monitoring, low level resources management, and fine grained resources sharing. And scale to the thousands of nodes.
  4. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is

    an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : gets information from the framework like environmental variables,configuration information and run tasks inside the mesos slaves.
  5. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is

    an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
  6. #Devoxx #DV14 #mesos #docker @samklr Resources allocation process - Mesos

    allocate resources via “resources offers” - Resources are a snapshot of available resources that a scheduler will use to run tasks on mesos slaves.
  7. #Devoxx #DV14 #mesos #docker @samklr Mesos uses Linux Containers for

    resource isolation. Mesos containers are built around cgroups , to provide isolation of CPU, memory, disk … and namespaces for network isolation. Started leveraging containerirsation since 2010
  8. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA -

    Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus
  9. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA -

    Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under.
  10. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA -

    Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. Slaves checkpoint information to their local disks (task info, status updates, etc ), they recover checkpointed data and reconnect with old tasks when they restart.
  11. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is

    an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
  12. #Devoxx #DV14 #mesos #docker @samklr - A generic mesos framework

    to run long running services (web apps, etc …) - A distributed Init.d for the cluster. - Runs any linux binary without modification (Tomcat, Play, …) - Cluster wide process supervisor -Private PaaS - Service discovery - Provide a self service Rest API for deployment - Authentification & SSL - Placement constrainst (nodes, racks, etc. (Like with Spark) ) - Service discovery and load balancing via HAProxy What is Marathon ?
  13. #Devoxx #DV14 #mesos #docker @samklr POST /v2/apps { “id” :

    “IPython Notebook” “container “ : { “image “ : “docker: //ipython/notebook” “options” : [ . . . ] } } Marathon Rest APi
  14. #Devoxx #DV14 #mesos #docker @samklr Wants to scale by addind

    more instances .. PATCH /v2/apps { “instances” : 4 } Marathon Rest APi