The Road to Akka Cluster, and Beyond…

Today, the skills of writing distributed applications is both more important and at the same time more challenging than ever. With the advent of mobile devices, NoSQL databases, cloud services etc. you most likely already have a distributed system at your hands—whether you like it or not. Distributed computing is the new norm.

In this talk we will take you on a journey across the distributed computing landscape. We will start with walking through some of the early work in computer architecture—setting the stage for what we are doing today. Then continue through distributed computing—discussing things like important Impossibility Theorems (FLP, CAP), Consensus Protocols (Raft, HAT, Epidemic Gossip etc.), Failure Detection (Accrual, Byzantine etc.), up to today’s very exciting research in the field, like ACID 2.0, Disorderly Programming (CRDTs, CALM etc). 

Along the way we will discuss the decisions and trade-offs that were made when creating Akka Cluster, its theoretical foundation, why it is designed the way it is and what the future holds. 

1. The Road Jonas Bonér CTO Typesafe @jboner to Akka Cluster and Beyond…
2. What is a Distributed System?
3. What is a and Why would You Need one? Distributed System?
4. Distributed Computing is the New normal
5. Distributed Computing is the New normal you already have a distributed system, WHETHER you want it or not
6. Distributed Computing is the New normal you already have a distributed system, WHETHER you want it or not Mobile NOSQL Databases Cloud & REST Services SQL Replication
7. essence of distributed computing? What is the
8. essence of distributed computing? overcome 1. Information travels at the speed of light 2. Independent things fail independently What is the It’s to try to
9. Why do we need it?
10. Why do we need it? Elasticity When you outgrow the resources of a single node
11. Why do we need it? Elasticity When you outgrow the resources of a single node Availability Providing resilience if one node fails
12. Why do we need it? Elasticity When you outgrow the resources of a single node Availability Providing resilience if one node fails Rich stateful clients
13. So, what’s the problem?
14. It is still Very Hard So, what’s the problem?
15. The network is Inherently Unreliable
16. You can’t tell the DIFFERENCE Between a Slow NODE and a Dead NODE
17. Fallacies Peter Deutsch’s 8 Fallacies of Distributed Computing
18. Fallacies 1. The network is reliable 2. Latency is zero 3. Bandwidth is infinite 4. The network is secure 5. Topology doesn't change 6. There is one administrator 7. Transport cost is zero 8. The network is homogeneous Peter Deutsch’s 8 Fallacies of Distributed Computing
19. So, oh yes…
20. It is still Very Hard So, oh yes…
21. 1. Guaranteed Delivery 2. Synchronous RPC 3. Distributed Objects 4. Distributed Shared Mutable State 5. Serializable Distributed Transactions Graveyard of distributed systems
22. Partition for scale Replicate for resilience General strategies Divide & Conquer
23. WHICH Requires SHARE NOTHING
 Designs General strategies Asynchronous Message-Passing
24. WHICH Requires SHARE NOTHING
 Designs General strategies Location Transparency Asynchronous Message-Passing ISolation & Containment
25. theoretical Models
26. A model for distributed Computation Should Allow explicit reasoning abouT 1. Concurrency 2. Distribution 3. Mobility Carlos Varela 2013
28. Lambda Calculus Alonzo Church 1930
29. Lambda Calculus state Immutable state Managed through functional application Referential transparent Alonzo Church 1930
30. order β-reduction—can be performed in any order Normal order Applicative order Call-by-name order Call-by-value order Call-by-need order Lambda Calculus state Immutable state Managed through functional application Referential transparent Alonzo Church 1930
31. Even in parallel order β-reduction—can be performed in any order Normal order Applicative order Call-by-name order Call-by-value order Call-by-need order Lambda Calculus state Immutable state Managed through functional application Referential transparent Alonzo Church 1930
32. Even in parallel order β-reduction—can be performed in any order Normal order Applicative order Call-by-name order Call-by-value order Call-by-need order Lambda Calculus state Immutable state Managed through functional application Referential transparent Alonzo Church 1930 Supports Concurrency
33. Even in parallel order β-reduction—can be performed in any order Normal order Applicative order Call-by-name order Call-by-value order Call-by-need order Lambda Calculus state Immutable state Managed through functional application Referential transparent Alonzo Church 1930 Supports Concurrency No model for Distribution
34. Even in parallel order β-reduction—can be performed in any order Normal order Applicative order Call-by-name order Call-by-value order Call-by-need order Lambda Calculus state Immutable state Managed through functional application Referential transparent Alonzo Church 1930 Supports Concurrency No model for Distribution No model for Mobility
36. Memory Control Unit Arithmetic Logic Unit Input Output Accumulator Von neumann machine John von Neumann 1945
37. Von neumann machine John von Neumann 1945
38. Von neumann machine state Mutable state In-place updates John von Neumann 1945
39. order Total order List of instructions Array of memory Von neumann machine state Mutable state In-place updates John von Neumann 1945
40. order Total order List of instructions Array of memory Von neumann machine state Mutable state In-place updates John von Neumann 1945 No model for Concurrency
41. order Total order List of instructions Array of memory Von neumann machine state Mutable state In-place updates John von Neumann 1945 No model for Concurrency No model for Distribution
42. order Total order List of instructions Array of memory Von neumann machine state Mutable state In-place updates John von Neumann 1945 No model for Concurrency No model for Distribution No model for Mobility
44. transactions Jim Gray 1981
45. transactions state Isolation of updates Atomicity Jim Gray 1981
46. order Serializability Disorder across transactions Illusion of order within transactions transactions state Isolation of updates Atomicity Jim Gray 1981
47. order Serializability Disorder across transactions Illusion of order within transactions transactions state Isolation of updates Atomicity Jim Gray 1981 Concurrency Works Work Well
48. order Serializability Disorder across transactions Illusion of order within transactions transactions state Isolation of updates Atomicity Jim Gray 1981 Concurrency Works Work Well Distribution Does Not Work Well
50. actors Carl HEWITT 1973
51. actors state Share nothing Atomicity within the actor Carl HEWITT 1973
52. order Async message passing Non-determinism in message delivery actors state Share nothing Atomicity within the actor Carl HEWITT 1973
53. order Async message passing Non-determinism in message delivery actors state Share nothing Atomicity within the actor Carl HEWITT 1973 Great model for Concurrency
54. order Async message passing Non-determinism in message delivery actors state Share nothing Atomicity within the actor Carl HEWITT 1973 Great model for Concurrency Great model for Distribution
55. order Async message passing Non-determinism in message delivery actors state Share nothing Atomicity within the actor Carl HEWITT 1973 Great model for Concurrency Great model for Distribution Great model for Mobility
56. other interesting models That are suitable for distributed systems 1. Pi Calculus 2. Ambient Calculus 3. Join Calculus
57. state of the The Art
58. Impossibility Theorems
59. Impossibility of Distributed Consensus with One Faulty Process
60. Impossibility of Distributed Consensus with One Faulty Process FLP Fischer Lynch Paterson 1985
61. Impossibility of Distributed Consensus with One Faulty Process FLP Fischer Lynch Paterson 1985 Consensus is impossible
62. Impossibility of Distributed Consensus with One Faulty Process FLP “The FLP result shows that in an asynchronous setting, where only one processor might crash, there is no distributed algorithm that solves the consensus problem” - The Paper Trail Fischer Lynch Paterson 1985 Consensus is impossible
63. Impossibility of Distributed Consensus with One Faulty Process FLP Fischer Lynch Paterson 1985
64. Impossibility of Distributed Consensus with One Faulty Process FLP “These results do not show that such problems cannot be “solved” in practice; rather, they point up the need for more refined models of distributed computing” - FLP paper Fischer Lynch Paterson 1985
66. CAP Theorem
67. Linearizability is impossible CAP Theorem
68. Conjecture by Eric Brewer 2000 Proof by Lynch & Gilbert 2002 Linearizability is impossible CAP Theorem
69. Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services Conjecture by Eric Brewer 2000 Proof by Lynch & Gilbert 2002 Linearizability is impossible CAP Theorem
70. linearizability
71. linearizability “Under linearizable consistency, all operations appear to have executed atomically in an order that is consistent with the global real-time ordering of operations.” Herlihy & Wing 1991
72. linearizability “Under linearizable consistency, all operations appear to have executed atomically in an order that is consistent with the global real-time ordering of operations.” Herlihy & Wing 1991 Less formally: A read will return the last completed write (made on any replica)
73. dissecting CAP
74. dissecting CAP 1. Very influential—but very NARROW scope
75. dissecting CAP 1. Very influential—but very NARROW scope 2. “[CAP] has lead to confusion and misunderstandings regarding replica consistency, transactional isolation and high availability” - Bailis et.al in HAT paper
76. dissecting CAP 1. Very influential—but very NARROW scope 2. “[CAP] has lead to confusion and misunderstandings regarding replica consistency, transactional isolation and high availability” - Bailis et.al in HAT paper 3. Linearizability is very often NOT required
77. dissecting CAP 1. Very influential—but very NARROW scope 2. “[CAP] has lead to confusion and misunderstandings regarding replica consistency, transactional isolation and high availability” - Bailis et.al in HAT paper 3. Linearizability is very often NOT required 4. Ignores LATENCY—but in practice latency & partitions are deeply related
78. dissecting CAP 1. Very influential—but very NARROW scope 2. “[CAP] has lead to confusion and misunderstandings regarding replica consistency, transactional isolation and high availability” - Bailis et.al in HAT paper 3. Linearizability is very often NOT required 4. Ignores LATENCY—but in practice latency & partitions are deeply related 5. Partitions are RARE—so why sacrifice C or A ALL the time?
79. dissecting CAP 1. Very influential—but very NARROW scope 2. “[CAP] has lead to confusion and misunderstandings regarding replica consistency, transactional isolation and high availability” - Bailis et.al in HAT paper 3. Linearizability is very often NOT required 4. Ignores LATENCY—but in practice latency & partitions are deeply related 5. Partitions are RARE—so why sacrifice C or A ALL the time? 6. NOT black and white—can be fine-grained and dynamic
80. dissecting CAP 1. Very influential—but very NARROW scope 2. “[CAP] has lead to confusion and misunderstandings regarding replica consistency, transactional isolation and high availability” - Bailis et.al in HAT paper 3. Linearizability is very often NOT required 4. Ignores LATENCY—but in practice latency & partitions are deeply related 5. Partitions are RARE—so why sacrifice C or A ALL the time? 6. NOT black and white—can be fine-grained and dynamic 7. Read ‘CAP Twelve Years Later’ - Eric Brewer
81. consensus
82. consensus “The problem of reaching agreement among remote processes is one of the most fundamental problems in distributed computing and is at the core of many algorithms for distributed data processing, distributed file management, and fault- tolerant distributed applications.” Fischer, Lynch & Paterson 1985
83. Consistency models
84. Consistency models Strong
85. Consistency models Strong Weak
86. Consistency models Strong Weak Eventual
87. Time & Order
88. Last write wins global clock timestamp
89. Last write wins global clock timestamp
90. lamport clocks logical clock causal consistency Leslie lamport 1978
91. lamport clocks logical clock causal consistency Leslie lamport 1978 1. When a process does work, increment the counter
92. lamport clocks logical clock causal consistency Leslie lamport 1978 1. When a process does work, increment the counter 2. When a process sends a message, include the counter
93. lamport clocks logical clock causal consistency Leslie lamport 1978 1. When a process does work, increment the counter 2. When a process sends a message, include the counter 3. When a message is received, merge the counter (set the counter to max(local, received) + 1)
94. vector clocks Extends lamport clocks colin fidge 1988
95. vector clocks Extends lamport clocks colin fidge 1988 1. Each node owns and increments its own Lamport Clock
96. vector clocks Extends lamport clocks colin fidge 1988 1. Each node owns and increments its own Lamport Clock [node -> lamport clock]
97. vector clocks Extends lamport clocks colin fidge 1988 1. Each node owns and increments its own Lamport Clock [node -> lamport clock]
98. vector clocks Extends lamport clocks colin fidge 1988 1. Each node owns and increments its own Lamport Clock [node -> lamport clock] 2. Alway keep the full history of all increments
99. vector clocks Extends lamport clocks colin fidge 1988 1. Each node owns and increments its own Lamport Clock [node -> lamport clock] 2. Alway keep the full history of all increments 3. Merges by calculating the max—monotonic merge
100. Quorum
101. Quorum Strict majority vote
102. Quorum Strict majority vote Sloppy partial vote
103. Quorum Strict majority vote Sloppy partial vote • Most use R + W > N 㱺 R & W overlap
104. Quorum Strict majority vote Sloppy partial vote • Most use R + W > N 㱺 R & W overlap • If N / 2 + 1 is still alive 㱺 all good
105. Quorum Strict majority vote Sloppy partial vote • Most use R + W > N 㱺 R & W overlap • If N / 2 + 1 is still alive 㱺 all good • Most use N ⩵ 3
106. failure Detection
107. Failure detection Formal model
108. Strong completeness Failure detection Formal model
109. Strong completeness Every crashed process is eventually suspected by every correct process Failure detection Formal model
110. Strong completeness Every crashed process is eventually suspected by every correct process Failure detection Formal model Everyone knows
111. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Failure detection Formal model Everyone knows
112. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Failure detection Formal model Everyone knows
113. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Failure detection Formal model Everyone knows Someone knows
114. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Strong accuracy Failure detection Formal model Everyone knows Someone knows
115. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Strong accuracy No correct process is suspected ever Failure detection Formal model Everyone knows Someone knows
116. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Strong accuracy No correct process is suspected ever Failure detection No false positives Formal model Everyone knows Someone knows
117. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Strong accuracy No correct process is suspected ever Weak accuracy Failure detection No false positives Formal model Everyone knows Someone knows
118. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Strong accuracy No correct process is suspected ever Weak accuracy Some correct process is never suspected Failure detection No false positives Formal model Everyone knows Someone knows
119. Strong completeness Every crashed process is eventually suspected by every correct process Weak completeness Every crashed process is eventually suspected by some correct process Strong accuracy No correct process is suspected ever Weak accuracy Some correct process is never suspected Failure detection No false positives Some false positives Formal model Everyone knows Someone knows
120. Accrual Failure detector Hayashibara et. al. 2004
121. Keeps history of heartbeat statistics Accrual Failure detector Hayashibara et. al. 2004
122. Keeps history of heartbeat statistics Decouples monitoring from interpretation Accrual Failure detector Hayashibara et. al. 2004
123. Keeps history of heartbeat statistics Decouples monitoring from interpretation Calculates a likelihood (phi value) that the process is down Accrual Failure detector Hayashibara et. al. 2004
124. Not YES or NO Keeps history of heartbeat statistics Decouples monitoring from interpretation Calculates a likelihood (phi value) that the process is down Accrual Failure detector Hayashibara et. al. 2004
125. Not YES or NO Keeps history of heartbeat statistics Decouples monitoring from interpretation Calculates a likelihood (phi value) that the process is down Accrual Failure detector Hayashibara et. al. 2004 Takes network hiccups into account
126. Not YES or NO Keeps history of heartbeat statistics Decouples monitoring from interpretation Calculates a likelihood (phi value) that the process is down Accrual Failure detector Hayashibara et. al. 2004 Takes network hiccups into account phi = -log10(1 - F(timeSinceLastHeartbeat)) F is the cumulative distribution function of a normal distribution with mean and standard deviation estimated from historical heartbeat inter-arrival times
127. SWIM Failure detector das et. al. 2002
128. SWIM Failure detector das et. al. 2002 Separates heartbeats from cluster dissemination
129. SWIM Failure detector das et. al. 2002 Separates heartbeats from cluster dissemination Quarantine: suspected 㱺 time window 㱺 faulty
130. SWIM Failure detector das et. al. 2002 Separates heartbeats from cluster dissemination Quarantine: suspected 㱺 time window 㱺 faulty Delegated heartbeat to bridge network splits
131. byzantine Failure detector liskov et. al. 1999
132. Supports misbehaving processes byzantine Failure detector liskov et. al. 1999
133. Supports misbehaving processes byzantine Failure detector liskov et. al. 1999 Omission failures
134. Supports misbehaving processes byzantine Failure detector liskov et. al. 1999 Omission failures Crash failures, failing to receive a request, or failing to send a response
135. Supports misbehaving processes byzantine Failure detector liskov et. al. 1999 Omission failures Crash failures, failing to receive a request, or failing to send a response Commission failures
136. Supports misbehaving processes byzantine Failure detector liskov et. al. 1999 Omission failures Crash failures, failing to receive a request, or failing to send a response Commission failures Processing a request incorrectly, corrupting local state, and/or sending an incorrect or inconsistent response to a request
137. Supports misbehaving processes byzantine Failure detector liskov et. al. 1999 Omission failures Crash failures, failing to receive a request, or failing to send a response Commission failures Processing a request incorrectly, corrupting local state, and/or sending an incorrect or inconsistent response to a request Very expensive, not practical
138. replication
139. Active (Push) ! Asynchronous Types of replication Passive (Pull) ! Synchronous VS VS
140. master/slave Replication
141. Tree replication
142. master/master Replication
143. buddy Replication
144. buddy Replication
145. analysis of replication consensus strategies Ryan Barrett 2009
146. Strong Consistency
147. Distributed transactions Strikes Back
148. Highly Available Transactions Peter Bailis et. al. 2013 CAP HAT NOT
149. Executive Summary Highly Available Transactions Peter Bailis et. al. 2013 CAP HAT NOT
150. Executive Summary • Most SQL DBs do not provide Serializability, but weaker guarantees— for performance reasons Highly Available Transactions Peter Bailis et. al. 2013 CAP HAT NOT
151. Executive Summary • Most SQL DBs do not provide Serializability, but weaker guarantees— for performance reasons • Some weaker transaction guarantees are possible to implement in a HA manner Highly Available Transactions Peter Bailis et. al. 2013 CAP HAT NOT
152. Executive Summary • Most SQL DBs do not provide Serializability, but weaker guarantees— for performance reasons • Some weaker transaction guarantees are possible to implement in a HA manner • What transaction semantics can be provided with HA? Highly Available Transactions Peter Bailis et. al. 2013 CAP HAT NOT
153. HAT
154. UnAvailable • Serializable • Snapshot Isolation • Repeatable Read • Cursor Stability • etc. Highly Available • Read Committed • Read Uncommited • Read Your Writes • Monotonic Atomic View • Monotonic Read/Write • etc. HAT
155. Other scalable or Highly Available Transactional Research
156. Other scalable or Highly Available Transactional Research Bolt-On Consistency Bailis et. al. 2013
157. Other scalable or Highly Available Transactional Research Bolt-On Consistency Bailis et. al. 2013 Calvin Thompson et. al. 2012
158. Other scalable or Highly Available Transactional Research Bolt-On Consistency Bailis et. al. 2013 Calvin Thompson et. al. 2012 Spanner (Google) Corbett et. al. 2012
159. consensus Protocols
160. Specification
161. Specification Properties
162. Events 1. Request(v) 2. Decide(v) Specification Properties
163. Events 1. Request(v) 2. Decide(v) Specification Properties 1. Termination: every process eventually decides on a value v
164. Events 1. Request(v) 2. Decide(v) Specification Properties 1. Termination: every process eventually decides on a value v 2. Validity: if a process decides v, then v was proposed by some process
165. Events 1. Request(v) 2. Decide(v) Specification Properties 1. Termination: every process eventually decides on a value v 2. Validity: if a process decides v, then v was proposed by some process 3. Integrity: no process decides twice
166. Events 1. Request(v) 2. Decide(v) Specification Properties 1. Termination: every process eventually decides on a value v 2. Validity: if a process decides v, then v was proposed by some process 3. Integrity: no process decides twice 4. Agreement: no two correct processes decide differently
167. Consensus Algorithms CAP
168. Consensus Algorithms CAP
169. Consensus Algorithms VR Oki & liskov 1988 CAP
170. Consensus Algorithms VR Oki & liskov 1988 Paxos Lamport 1989 CAP
171. Consensus Algorithms VR Oki & liskov 1988 Paxos Lamport 1989 ZAB reed & junquiera 2008 CAP
172. Consensus Algorithms VR Oki & liskov 1988 Paxos Lamport 1989 ZAB reed & junquiera 2008 Raft ongaro & ousterhout 2013 CAP
173. Event Log
174. “Immutability Changes Everything” - Pat Helland Immutable Data Immutability Share Nothing Architecture
175. “Immutability Changes Everything” - Pat Helland Immutable Data Immutability Share Nothing Architecture TRUE Scalability Is the path towards
176. "The database is a cache of a subset of the log” - Pat Helland Think In Facts
177. "The database is a cache of a subset of the log” - Pat Helland Think In Facts Never delete data Knowledge only grows Append-Only Event Log Use Event Sourcing and/or CQRS
178. Aggregate Roots Can wrap multiple Entities Aggregate Root is the Transactional Boundary
179. Aggregate Roots Can wrap multiple Entities Strong Consistency Within Aggregate Eventual Consistency Between Aggregates Aggregate Root is the Transactional Boundary
180. Aggregate Roots Can wrap multiple Entities Strong Consistency Within Aggregate Eventual Consistency Between Aggregates Aggregate Root is the Transactional Boundary No limit to scalability
181. eventual Consistency
182. Dynamo VerY influential CAP Vogels et. al. 2007
183. Dynamo Popularized • Eventual consistency • Epidemic gossip • Consistent hashing ! • Hinted handoff • Read repair • Anti-Entropy W/ Merkle trees VerY influential CAP Vogels et. al. 2007
184. Consistent Hashing Karger et. al. 1997
185. Consistent Hashing Support elasticity— easier to scale up and down Avoids hotspots Enables partitioning and replication Karger et. al. 1997
186. Consistent Hashing Support elasticity— easier to scale up and down Avoids hotspots Enables partitioning and replication Karger et. al. 1997 Only K/N nodes needs to be remapped when adding or removing a node (K=#keys, N=#nodes)
187. How eventual is
188. How eventual is Eventual consistency?
189. How eventual is How consistent is Eventual consistency?
190. How eventual is How consistent is Eventual consistency? Probabilistically Bounded Staleness Peter Bailis et. al 2012 PBS
191. How eventual is How consistent is Eventual consistency? Probabilistically Bounded Staleness Peter Bailis et. al 2012 PBS
192. epidemic Gossip
193. Node ring & Epidemic Gossip CHORD Stoica et al 2001
194. Node ring & Epidemic Gossip Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node CHORD Stoica et al 2001
195. Node ring & Epidemic Gossip Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node CHORD Stoica et al 2001
196. Node ring & Epidemic Gossip Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node CHORD Stoica et al 2001
197. Node ring & Epidemic Gossip Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node Member Node CHORD Stoica et al 2001 CAP
198. Decentralized P2P No SPOF or SPOB Very Scalable Fully Elastic Benefits of Epidemic Gossip ! Requires minimal administration Often used with VECTOR CLOCKS
199. 1. Separation of failure detection heartbeat and dissemination of data - DAS et. al. 2002 (SWIM) 2. Push/Pull gossip - Khambatti et. al 2003 1. Hash and compare data 2. Use single hash or Merkle Trees Some Standard Optimizations to Epidemic Gossip
200. disorderly Programming
201. ACID 2.0
202. ACID 2.0 Associative Batch-insensitive (grouping doesn't matter) a+(b+c)=(a+b)+c
203. ACID 2.0 Associative Batch-insensitive (grouping doesn't matter) a+(b+c)=(a+b)+c Commutative Order-insensitive (order doesn't matter) a+b=b+a
204. ACID 2.0 Associative Batch-insensitive (grouping doesn't matter) a+(b+c)=(a+b)+c Commutative Order-insensitive (order doesn't matter) a+b=b+a Idempotent Retransmission-insensitive (duplication does not matter) a+a=a
205. ACID 2.0 Associative Batch-insensitive (grouping doesn't matter) a+(b+c)=(a+b)+c Commutative Order-insensitive (order doesn't matter) a+b=b+a Idempotent Retransmission-insensitive (duplication does not matter) a+a=a Eventually Consistent
206. Convergent & Commutative Replicated Data Types Shapiro et. al. 2011
207. Convergent & Commutative Replicated Data Types CRDTShapiro et. al. 2011
208. Convergent & Commutative Replicated Data Types CRDTShapiro et. al. 2011 Join Semilattice Monotonic merge function
209. Convergent & Commutative Replicated Data Types Data types Counters Registers Sets Maps Graphs CRDTShapiro et. al. 2011 Join Semilattice Monotonic merge function
210. Convergent & Commutative Replicated Data Types Data types Counters Registers Sets Maps Graphs CRDT CAP Shapiro et. al. 2011 Join Semilattice Monotonic merge function
211. 2 TYPES of CRDTs CvRDT Convergent State-based CmRDT Commutative Ops-based
212. 2 TYPES of CRDTs CvRDT Convergent State-based CmRDT Commutative Ops-based Self contained, holds all history
213. 2 TYPES of CRDTs CvRDT Convergent State-based CmRDT Commutative Ops-based Self contained, holds all history Needs a reliable broadcast channel
214. CALM theorem Consistency As Logical Monotonicity Hellerstein et. al. 2011
215. CALM theorem Consistency As Logical Monotonicity Hellerstein et. al. 2011 Bloom Language Compiler help to detect & encapsulate non- monotonicity
216. CALM theorem Consistency As Logical Monotonicity Distributed Logic Datalog/Dedalus Monotonic functions Just add facts to the system Model state as Lattices Similar to CRDTs (without the scope problem) Hellerstein et. al. 2011 Bloom Language Compiler help to detect & encapsulate non- monotonicity
217. The Akka Way
218. Akka Actors
219. Akka Actors Akka IO
220. Akka Actors Akka IO Akka REMOTE
221. Akka Actors Akka IO Akka REMOTE Akka CLUSTER
222. Akka Actors Akka IO Akka REMOTE Akka CLUSTER Akka CLUSTER EXTENSIONS
223. What is Akka CLUSTER all about? • Cluster Membership • Leader & Singleton • Cluster Sharding • Clustered Routers (adaptive, consistent hashing, …) • Clustered Supervision and Deathwatch • Clustered Pub/Sub • and more
224. cluster membership in Akka
225. cluster membership in Akka • Dynamo-style master-less decentralized P2P
226. cluster membership in Akka • Dynamo-style master-less decentralized P2P • Epidemic Gossip—Node Ring
227. cluster membership in Akka • Dynamo-style master-less decentralized P2P • Epidemic Gossip—Node Ring • Vector Clocks for causal consistency
228. cluster membership in Akka • Dynamo-style master-less decentralized P2P • Epidemic Gossip—Node Ring • Vector Clocks for causal consistency • Fully elastic with no SPOF or SPOB
229. cluster membership in Akka • Dynamo-style master-less decentralized P2P • Epidemic Gossip—Node Ring • Vector Clocks for causal consistency • Fully elastic with no SPOF or SPOB • Very scalable—2400 nodes (on GCE)
230. cluster membership in Akka • Dynamo-style master-less decentralized P2P • Epidemic Gossip—Node Ring • Vector Clocks for causal consistency • Fully elastic with no SPOF or SPOB • Very scalable—2400 nodes (on GCE) • High throughput—1000 nodes in 4 min (on GCE)
231. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock)
232. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) Is a CRDT
233. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) Is a CRDT Ordered node ring
234. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) Is a CRDT Ordered node ring Seen set for convergence
235. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) Is a CRDT Ordered node ring Seen set for convergence Unreachable set
236. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) Is a CRDT Ordered node ring Seen set for convergence Unreachable set Version
237. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) 1. Picks random node with older/newer version Is a CRDT Ordered node ring Seen set for convergence Unreachable set Version
238. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) 1. Picks random node with older/newer version 2. Gossips in a request/reply fashion Is a CRDT Ordered node ring Seen set for convergence Unreachable set Version
239. State Gossip GOSSIPING case class Gossip( members: SortedSet[Member], seen: Set[Member], unreachable: Set[Member], version: VectorClock) 1. Picks random node with older/newer version 2. Gossips in a request/reply fashion 3. Updates internal state and adds himself to ‘seen’ set Is a CRDT Ordered node ring Seen set for convergence Unreachable set Version
240. Cluster Convergence
241. Cluster Convergence Reached when: 1. All nodes are represented in the seen set 2. No members are unreachable, or 3. All unreachable members have status down or exiting
242. GOSSIP BIASED
243. GOSSIP BIASED 80% bias to nodes not in seen table Up to 400 nodes, then reduced
244. PUSH/PULL GOSSIP
245. PUSH/PULL GOSSIP Variation
246. PUSH/PULL GOSSIP Variation case class Status(version: VectorClock)
247. ROLE LEADER
248. ROLE LEADER Any node can be the leader
249. ROLE 1. No election, but deterministic LEADER Any node can be the leader
250. ROLE 1. No election, but deterministic 2. Can change after cluster convergence LEADER Any node can be the leader
251. ROLE 1. No election, but deterministic 2. Can change after cluster convergence 3. Leader has special duties LEADER Any node can be the leader
252. Node Lifecycle in Akka
253. Failure Detection
254. Failure Detection Hashes the node ring Picks 5 nodes Request/Reply heartbeat
255. Failure Detection Hashes the node ring Picks 5 nodes Request/Reply heartbeat To increase likelihood of bridging racks and data centers
256. Failure Detection Cluster Membership Remote Death Watch Remote Supervision Hashes the node ring Picks 5 nodes Request/Reply heartbeat To increase likelihood of bridging racks and data centers Used by
257. Failure Detection Is an Accrual Failure Detector
258. Failure Detection Is an Accrual Failure Detector Does not help much in practice
259. Failure Detection Is an Accrual Failure Detector Does not help much in practice Need to add delay to deal with Garbage Collection
260. Failure Detection Is an Accrual Failure Detector Does not help much in practice Instead of this Need to add delay to deal with Garbage Collection
261. Failure Detection Is an Accrual Failure Detector Does not help much in practice Instead of this It often looks like this Need to add delay to deal with Garbage Collection
262. Network Partitions
263. Network Partitions • Failure Detector can mark an unavailable member Unreachable
264. Network Partitions • Failure Detector can mark an unavailable member Unreachable • If one node is Unreachable then no cluster Convergence
265. Network Partitions • Failure Detector can mark an unavailable member Unreachable • If one node is Unreachable then no cluster Convergence • This means that the Leader can no longer perform it’s duties
266. Network Partitions • Failure Detector can mark an unavailable member Unreachable • If one node is Unreachable then no cluster Convergence • This means that the Leader can no longer perform it’s duties Split Brain
267. Network Partitions • Failure Detector can mark an unavailable member Unreachable • If one node is Unreachable then no cluster Convergence • This means that the Leader can no longer perform it’s duties • Member can come back from Unreachable—Else: Split Brain
268. Network Partitions • Failure Detector can mark an unavailable member Unreachable • If one node is Unreachable then no cluster Convergence • This means that the Leader can no longer perform it’s duties • Member can come back from Unreachable—Else: • The node needs to be marked as Down—either through: Split Brain
269. Network Partitions • Failure Detector can mark an unavailable member Unreachable • If one node is Unreachable then no cluster Convergence • This means that the Leader can no longer perform it’s duties • Member can come back from Unreachable—Else: • The node needs to be marked as Down—either through: 1. auto-down 2. Manual down Split Brain
270. Potential FUTURE Optimizations
271. Potential FUTURE Optimizations • Vector Clock HISTORY pruning
272. Potential FUTURE Optimizations • Vector Clock HISTORY pruning • Delegated heartbeat
273. Potential FUTURE Optimizations • Vector Clock HISTORY pruning • Delegated heartbeat • “Real” push/pull gossip
274. Potential FUTURE Optimizations • Vector Clock HISTORY pruning • Delegated heartbeat • “Real” push/pull gossip • More out-of-the-box auto-down patterns
275. Akka Modules For Distribution
276. Akka Modules For Distribution Akka Cluster Akka Remote Akka HTTP Akka IO
277. Akka Modules For Distribution Akka Cluster Akka Remote Akka HTTP Akka IO Clustered Singleton Clustered Routers Clustered Pub/Sub Cluster Client Consistent Hashing
278. Beyond …and
279. Akka & The Road Ahead Akka HTTP Akka Streams Akka CRDT Akka Raft
280. Akka & The Road Ahead Akka HTTP Akka Streams Akka CRDT Akka Raft Akka 2.4
281. Akka & The Road Ahead Akka HTTP Akka Streams Akka CRDT Akka Raft Akka 2.4 Akka 2.4
282. Akka & The Road Ahead Akka HTTP Akka Streams Akka CRDT Akka Raft Akka 2.4 Akka 2.4 ?
283. Akka & The Road Ahead Akka HTTP Akka Streams Akka CRDT Akka Raft Akka 2.4 Akka 2.4 ? ?
284. Eager for more?
285. Try AKKA out akka.io
286. Join us at React Conf San Francisco Nov 18-21 reactconf.com
287. Join us at React Conf San Francisco Nov 18-21 reactconf.com Early Registration ends tomorrow
288. References • General Distributed Systems • Summary of network reliability post-mortems—more terrifying than the most horrifying Stephen King novel: http://aphyr.com/posts/288-the-network-is- reliable • A Note on Distributed Computing: http://citeseerx.ist.psu.edu/viewdoc/ summary?doi=10.1.1.41.7628 • On the problems with RPC: http://steve.vinoski.net/pdf/IEEE- Convenience_Over_Correctness.pdf • 8 Fallacies of Distributed Computing: https://blogs.oracle.com/jag/resource/ Fallacies.html • 6 Misconceptions of Distributed Computing: www.dsg.cs.tcd.ie/~vjcahill/ sigops98/papers/vogels.ps • Distributed Computing Systems—A Foundational Approach: http:// www.amazon.com/Programming-Distributed-Computing-Systems- Foundational/dp/0262018985 • Introduction to Reliable and Secure Distributed Programming: http:// www.distributedprogramming.net/ • Nice short overview on Distributed Systems: http://book.mixu.net/distsys/ • Meta list of distributed systems readings: https://gist.github.com/macintux/ 6227368
289. References ! • Actor Model • Great discussion between Erik Meijer & Carl Hewitt or the essence of the Actor Model: http:// channel9.msdn.com/Shows/Going+Deep/Hewitt- Meijer-and-Szyperski-The-Actor-Model- everything-you-wanted-to-know-but-were-afraid- to-ask • Carl Hewitt’s 1973 paper defining the Actor Model: http://worrydream.com/refs/Hewitt- ActorModel.pdf • Gul Agha’s Doctoral Dissertation: https:// dspace.mit.edu/handle/1721.1/6952
290. References • FLP • Impossibility of Distributed Consensus with One Faulty Process: http:// cs-www.cs.yale.edu/homes/arvind/cs425/doc/fischer.pdf • A Brief Tour of FLP: http://the-paper-trail.org/blog/a-brief-tour-of-flp- impossibility/ • CAP • Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services: http://lpd.epfl.ch/sgilbert/pubs/ BrewersConjecture-SigAct.pdf • You Can’t Sacrifice Partition Tolerance: http://codahale.com/you-cant- sacrifice-partition-tolerance/ • Linearizability: A Correctness Condition for Concurrent Objects: http:// courses.cs.vt.edu/~cs5204/fall07-kafura/Papers/TransactionalMemory/ Linearizability.pdf • CAP Twelve Years Later: How the "Rules" Have Changed: http:// www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have- changed • Consistency vs. Availability: http://www.infoq.com/news/2008/01/ consistency-vs-availability
291. References • Time & Order • Post on the problems with Last Write Wins in Riak: http:// aphyr.com/posts/285-call-me-maybe-riak • Time, Clocks, and the Ordering of Events in a Distributed System: http://research.microsoft.com/en-us/um/people/lamport/pubs/ time-clocks.pdf • Vector Clocks: http://zoo.cs.yale.edu/classes/cs426/2012/lab/ bib/fidge88timestamps.pdf • Failure Detection • Unreliable Failure Detectors for Reliable Distributed Systems: http://www.cs.utexas.edu/~lorenzo/corsi/cs380d/papers/p225- chandra.pdf • The ϕ Accrual Failure Detector: http://ddg.jaist.ac.jp/pub/HDY +04.pdf • SWIM Failure Detector: http://www.cs.cornell.edu/~asdas/ research/dsn02-swim.pdf • Practical Byzantine Fault Tolerance: http://www.pmg.lcs.mit.edu/ papers/osdi99.pdf
292. References • Transactions • Jim Gray’s classic book: http://www.amazon.com/Transaction- Processing-Concepts-Techniques-Management/dp/1558601902 • Highly Available Transactions: Virtues and Limitations: http:// www.bailis.org/papers/hat-vldb2014.pdf • Bolt on Consistency: http://db.cs.berkeley.edu/papers/sigmod13- bolton.pdf • Calvin: Fast Distributed Transactions for Partitioned Database Systems: http://cs.yale.edu/homes/thomson/publications/calvin-sigmod12.pdf • Spanner: Google's Globally-Distributed Database: http:// research.google.com/archive/spanner.html • Life beyond Distributed Transactions: an Apostate’s Opinion https:// cs.brown.edu/courses/cs227/archives/2012/papers/weaker/ cidr07p15.pdf • Immutability Changes Everything—Pat Hellands talk at Ricon: http:// vimeo.com/52831373 • Unschackle Your Domain (Event Sourcing): http://www.infoq.com/ presentations/greg-young-unshackle-qcon08 • CQRS: http://martinfowler.com/bliki/CQRS.html
293. References • Consensus • Paxos Made Simple: http://research.microsoft.com/en- us/um/people/lamport/pubs/paxos-simple.pdf • Paxos Made Moderately Complex: http:// www.cs.cornell.edu/courses/cs7412/2011sp/paxos.pdf • A simple totally ordered broadcast protocol (ZAB): labs.yahoo.com/files/ladis08.pdf • In Search of an Understandable Consensus Algorithm (Raft): https://ramcloud.stanford.edu/wiki/download/ attachments/11370504/raft.pdf • Replication strategy comparison diagram: http:// snarfed.org/transactions_across_datacenters_io.html • Distributed Snapshots: Determining Global States of Distributed Systems: http://www.cs.swarthmore.edu/ ~newhall/readings/snapshots.pdf
294. References • Eventual Consistency • Dynamo: Amazon’s Highly Available Key-value Store: http://www.read.seas.harvard.edu/ ~kohler/class/cs239-w08/ decandia07dynamo.pdf • Consistency vs. Availability: http:// www.infoq.com/news/2008/01/consistency- vs-availability • Consistent Hashing and Random Trees: http:// thor.cs.ucsb.edu/~ravenben/papers/coreos/kll +97.pdf • PBS: Probabilistically Bounded Staleness: http://pbs.cs.berkeley.edu/
295. References • Epidemic Gossip • Chord: A Scalable Peer-to-peer Lookup Service for Internet • Applications: http://pdos.csail.mit.edu/papers/chord:sigcomm01/ chord_sigcomm.pdf • Gossip-style Failure Detector: http://www.cs.cornell.edu/home/rvr/ papers/GossipFD.pdf • GEMS: http://www.hcs.ufl.edu/pubs/GEMS2005.pdf • Efficient Reconciliation and Flow Control for Anti-Entropy Protocols: http://www.cs.cornell.edu/home/rvr/papers/flowgossip.pdf • 2400 Akka nodes on GCE: http://typesafe.com/blog/running-a-2400- akka-nodes-cluster-on-google-compute-engine • Starting 1000 Akka nodes in 4 min: http://typesafe.com/blog/starting- up-a-1000-node-akka-cluster-in-4-minutes-on-google-compute- engine • Push Pull Gossiping: http://khambatti.com/mujtaba/ ArticlesAndPapers/pdpta03.pdf • SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol: http://www.cs.cornell.edu/~asdas/research/ dsn02-swim.pdf
296. References • Conflict-Free Replicated Data Types (CRDTs) • A comprehensive study of Convergent and Commutative Replicated Data Types: http://hal.upmc.fr/docs/ 00/55/55/88/PDF/techreport.pdf • Mark Shapiro talks about CRDTs at Microsoft: http:// research.microsoft.com/apps/video/dl.aspx?id=153540 • Akka CRDT project: https://github.com/jboner/akka-crdt • CALM • Dedalus: Datalog in Time and Space: http:// db.cs.berkeley.edu/papers/datalog2011-dedalus.pdf • CALM: http://www.cs.berkeley.edu/~palvaro/cidr11.pdf • Logic and Lattices for Distributed Programming: http:// db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf • Bloom Language website: http://bloom-lang.net • Joe Hellerstein talks about CALM: http://vimeo.com/ 53904989
297. References • Akka Cluster • My Akka Cluster Implementation Notes: https:// gist.github.com/jboner/7692270 • Akka Cluster Specification: http://doc.akka.io/docs/ akka/snapshot/common/cluster.html • Akka Cluster Docs: http://doc.akka.io/docs/akka/ snapshot/scala/cluster-usage.html • Akka Failure Detector Docs: http://doc.akka.io/docs/ akka/snapshot/scala/remoting.html#Failure_Detector • Akka Roadmap: https://docs.google.com/a/ typesafe.com/document/d/18W9- fKs55wiFNjXL9q50PYOnR7-nnsImzJqHOPPbM4E/ mobilebasic?pli=1&hl=en_US • Where Akka Came From: http://letitcrash.com/post/ 40599293211/where-akka-came-from
298. any Questions?
299. The Road Jonas Bonér CTO Typesafe @jboner to Akka Cluster and Beyond…