Coordination-Free Designs for Mobile Gaming

Code Mesh 2015, September 3rd, 2015

1. Coordination-Free Designs for Mobile Gaming Christopher Meiklejohn // @cmeik Code Mesh 2015, September 3rd, 2015 1
2. RA RB
3. RA RB 1 set(1)
4. RA RB 1 set(1) 3 2 set(2) set(3)
5. RA RB 1 set(1) 3 2 set(2) set(3) ? ?
6. Synchronization • To enforce an order
 Makes programming easier 6
7. Synchronization • To enforce an order
 Makes programming easier • Eliminate accidental nondeterminism
 Prevent race conditions 6
8. Synchronization • To enforce an order
 Makes programming easier • Eliminate accidental nondeterminism
 Prevent race conditions • Techniques
 Locks, mutexes, semaphores, monitors, etc. 6
9. Difficult Cases • “Internet of Things”
 Low power, limited memory and connectivity 7
10. Difficult Cases • “Internet of Things”
 Low power, limited memory and connectivity • Mobile Gaming
 Offline operation with replicated, shared state 7
11. Zero Synchronization • We want to have our cake and eat it too. 8
12. Zero Synchronization • We want to have our cake and eat it too. • “Zero synchronization” as a starting point
 Add synchronization only where it is necessary 8
13. Zero Synchronization • We want to have our cake and eat it too. • “Zero synchronization” as a starting point
 Add synchronization only where it is necessary • Explore the limits
 Can we design a model of computation where it’s easy to write coordination-free computations that are free from concurrency anomalies? 8
14. The Fundamentals of Distributed Computation 9
15. Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure 10
16. Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion 10
17. Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract 10
18. Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract • Enforced through synchronization 10
19. Distributed Computation • Distributed to concurrent programming
 Consistency and now partial failure • Enforcing the “single system” illusion • Consistency model as a contract • Enforced through synchronization • Consistency model
 We can think of a consistency model as analogous to a programming paradigm 10
20. Why Is Synchronization Undesirable? 11
21. The Avatars of Time • The problem of time
 Handling physical time in applications is difficult 12
22. The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing 12
23. The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems 12
24. The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems 12
25. The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ̣ Network latency in distributed systems 12
26. The Avatars of Time • The problem of time
 Handling physical time in applications is difficult • Time has three major avatars in computing • Mutable state in sequential systems • Nondeterminism in concurrent systems ̣ Network latency in distributed systems • Unavoidable
 Time is how users interact with programs 12
27. Parable of the Car • “Car driving on a highway” 13
28. Parable of the Car • “Car driving on a highway” • Friction needed for the car to grip the road 13
29. Parable of the Car • “Car driving on a highway” • Friction needed for the car to grip the road • Motors want as little friction as possible 13
30. Parable of the Car • “Car driving on a highway” • Friction needed for the car to grip the road • Motors want as little friction as possible • Time is like friction
 We can not completely eliminate friction, but aim to reduce it as much as possible 13
31. Physical Time (real world) Parable of the Car 14 • Time only at the interface
 The interface is a small part of the system, and this is where we introduction friction
32. Physical Time (real world) Parable of the Car 14 • Time only at the interface
 The interface is a small part of the system, and this is where we introduction friction • Physical time-free execution
 Internally, avoid synchronization as much as possible and virtualize notion of time
33. No Time In The Box? 15
34. No Time In The Box? 15 You program the box.
35. Weak Synchronization • Can we achieve anything without synchronization?
 Not really. 16
36. Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” 16
37. Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” • Primary requirement
 Eventual replica-to-replica communication 16
38. Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” • Primary requirement
 Eventual replica-to-replica communication • Order insensitive! (Commutativity) 16
39. Weak Synchronization • Can we achieve anything without synchronization?
 Not really. • Strong Eventual Consistency (SEC)
 “Replicas that deliver the same updates have equivalent state” • Primary requirement
 Eventual replica-to-replica communication • Order insensitive! (Commutativity) • Duplicate insensitive! (Idempotent) 16
40. RA RB
41. RA RB 1 set(1)
42. RA RB 1 set(1) 3 2 set(2) set(3)
43. RA RB 1 3 2 3 3 set(1) set(2) set(3) max(2,3) max(2,3)
44. How can we succeed with Strong Eventual Consistency? 21
45. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 22
46. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 22
47. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 22
48. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 23
49. Convergent Objects
 Conflict-Free 
 Replicated Data Types 24 SSS 2011
50. Conflict-Free 
 Replicated Data Types • Many types exist with different properties
 Sets, counters, registers, flags, maps, graphs 25
51. Conflict-Free 
 Replicated Data Types • Many types exist with different properties
 Sets, counters, registers, flags, maps, graphs • Strong Eventual Consistency
 Instances satisfy SEC property per-object 25
52. RA RB RC
53. RA RB RC {1} (1, {a}, {}) add(1)
54. RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {c}, {}) add(1)
55. RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {c}, {}) add(1) {} (1, {c}, {c}) remove(1)
56. RA RB RC {1} (1, {a}, {}) add(1) {1} (1, {c}, {}) add(1) {} (1, {c}, {c}) remove(1) {1} {1} {1} (1, {a, c}, {c}) (1, {a, c}, {c}) (1, {a, c}, {c})
57. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 31
58. Convergent Programs Lattice Processing 32 PPDP 2015
59. Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations 33
60. Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures
 Primary data abstraction is the CRDT 33
61. Lattice Processing (Lasp) • Distributed, deterministic dataflow
 Distributed, deterministic dataflow programming model for “eventually consistent” computations • Convergent data structures
 Primary data abstraction is the CRDT • Enables composition
 Provides functional composition of CRDTs that preserves the SEC property 33
62. 34 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
63. 35 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
64. 36 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
65. 37 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
66. 38 %% Create initial set. S1 = declare(set), %% Add elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
67. Lattice Processing (Lasp) • Functional and set-theoretic operations on sets
 Product, intersection, union, filter, map, fold 39
68. Lattice Processing (Lasp) • Functional and set-theoretic operations on sets
 Product, intersection, union, filter, map, fold • Metadata computation
 Performs transformation on the internal metadata of CRDTs allowing creation of “composed” CRDTs 39
69. Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states 40
70. Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream 40
71. Lasp Processes • Replicas as monotonic streams
 Each replica of a CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream • Inflationary reads
 Read operation ensures that we only read inflationary updates to replicas 40
72. Lattice Processing Monotonic Streams 41
73. RA {} C1 {} C2 {} 42
74. RA {} (1, {a}, {}) C1 (1, {a}, {}) {} C2 {} 43
75. RA {} (1, {a}, {}) (1, {a, c}, {}) C1 (1, {a}, {}) {} C2 (1, {c}, {}) {} 44
76. RA {} (1, {a}, {}) (1, {a, c}, {}) (1, {a, c}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {c}, {}) {} 45
77. RA {} (1, {a}, {}) (1, {a, c}, {}) (1, {a, c}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {c}, {}) {} (1, {a, c}, {a}) (1, {a, c}, {a}) (1, {a, c}, {a}) 46
78. RA {} (1, {a}, {}) (1, {a, c}, {}) (1, {a, c}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {c}, {}) {} (1, {a, c}, {a}) (1, {a, c}, {a}) (1, {a, c}, {a}) 47
79. Lattice Processing Monotonic Processes 48
80. RA {} P1 F(RA) {} 49
81. RA {} P1 F(RA) {} strict_read({}) 50
82. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) 51
83. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) 52
84. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) 53
85. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) 54
86. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1, {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 55
87. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 56
88. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 57
89. Distributed Runtime Selective Hearing 58 W-PSDS 2015
90. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution 59
91. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) 59
92. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient 59
93. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures 59
94. Selective Hearing • Epidemic broadcast based runtime system
 Provide a runtime system that can scale to large numbers of nodes, that is resilient to failures and provides efficient execution • Well-matched to Lattice Processing (Lasp) • Epidemic broadcast mechanisms provide weak ordering but are resilient and efficient • Lasp’s programming model is tolerant to message re-ordering, disconnections, and node failures • “Selective Receive”
 Nodes selectively receive and process messages based on interest 59
95. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, referential transparency over composition) 3. Distributed, and fault-tolerant runtime
 (ex. replication) 60
96. What can we build? Leaderboard 61
97. Leaderboard • Mobile game platform
 Local leaderboard tracking top-k highest scored games 62
98. Leaderboard • Mobile game platform
 Local leaderboard tracking top-k highest scored games • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline 62
99. Client 1 Leaderboard Client 3 Leaderboard Client 2 Leaderboard Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT 63
100. Leaderboard • Peer-to-peer dissemination
 Nodes periodically “merge” their state with a random peer 64
101. Leaderboard • Peer-to-peer dissemination
 Nodes periodically “merge” their state with a random peer • Complexity in the data type
 Each node tracks a top-k set of its own games in a bounded set 64
102. 65 %% Create a leaderboard datatype. L = declare({top_k, [2]}). %% Update leaderboard. update({set, Name, Score}, L).
103. 66 %% Create a leaderboard datatype. L = declare({top_k, [2]}). %% Update leaderboard. update({set, Name, Score}, L).
104. 67 %% Create a leaderboard datatype. L = declare({top_k, [2]}). %% Update leaderboard. update({set, Name, Score}, L).
105. What if we want to enhance the behavior? 68
106. What if we want to enhance the behavior? 68 Without the creation of a new datatype
107. What can we build? Per-User Leaderboard 69 EdgeCom 2016
108. Per-User Leaderboard • Enhance existing design
 Only the top score for each user at each device 70
109. Per-User Leaderboard • Enhance existing design
 Only the top score for each user at each device • Minimize transmitted state
 Prevent transmission of state that is not necessary to perform the computation 70
110. Per-User Leaderboard • Enhance existing design
 Only the top score for each user at each device • Minimize transmitted state
 Prevent transmission of state that is not necessary to perform the computation • Compose data types
 Build a per-user leaderboard through the composition of existing types 70
111. Client1 Scores Local Top-K Fold Global Top-K Lasp Operation Input User-Maintained CRDT Output Lasp-Maintained CRDT Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold 71
112. Client1 Scores Local Top-K Fold Global Top-K Fold 72 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
113. Client1 Scores Local Top-K Fold Global Top-K Fold 73 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
114. Client1 Scores Local Top-K Fold Global Top-K Fold 74 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
115. Client1 Scores Local Top-K Fold Global Top-K Fold 75 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
116. Client1 Scores Local Top-K Fold Global Top-K Fold 76 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
117. Client1 Scores Local Top-K Fold Global Top-K Fold 77 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
118. Client1 Scores Local Top-K Fold Global Top-K Lasp Operation Input User-Maintained CRDT Output Lasp-Maintained CRDT Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold 78 Client1 Scores Local Top-K Fold Global Top-K Fold Client3 Scores Local Top-K Fold Global Top-K Fold Client2 Scores Local Top-K Fold Global Top-K Fold
119. Per-User Leaderboard • Dynamically scoped variables
 Variable which take different values depending on where it is executing 79
120. Per-User Leaderboard • Dynamically scoped variables
 Variable which take different values depending on where it is executing • Dynamically scoped fold operation
 Perform a distributed “reduce” operation that combines the state of a dynamically scoped variables across 79
121. 80 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
122. 81 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
123. 82 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
124. 83 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
125. 84 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
126. 85 %% Create a global leaderboard. G = declare({top_k, [10]}). %% Create a local leaderboard. L = declare_dynamic({top_k, [10]}). %% Create a set of scores. S = declare_dynamic(set). %% Compute local top-k list. fold(S, fun max_by_name/2, L). %% Compute global top-k list. fold_dynamic(L, fun max_by_name/2, G).
127. Let’s look at a larger application 86
128. Let’s look at a larger application 86 With visible 
 non-monotonicity
129. What can we build? Advertisement Counter 87
130. Advertisement Counter • Mobile game platform selling advertisement space
 Advertisements are paid according to a minimum number of impressions 88
131. Advertisement Counter • Mobile game platform selling advertisement space
 Advertisements are paid according to a minimum number of impressions • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline 88
132. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 89
133. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Product Read 50,000 Remove Increment Union 90 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
134. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 1 Client 91 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
135. Ads ovio Ad ounter 1 ovio Ad ounter 2 iot Ad ounter 1 iot Ad ounter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Rovio Ad Counter 1 Ro C Client Side, Sing 92 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
136. Ads Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product move Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 93 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
137. Ads Contracts Ads Contracts Ads With Contracts Filter Product Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 94 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
138. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 95 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
139. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Riot Ads Rovio Ads Fil Product Read 50,000 Remove Increment Union 96 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
140. Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client 97 Ads Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 2 Contracts Ads Contracts Ads With Contracts Riot Ads Rovio Ads Filter Product Read 50,000 Remove Increment Read Union Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Rovio Ad Counter 1 Rovio Ad Counter 2 Riot Ad Counter 1 Client Side, Single Copy at Client
141. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth 98
142. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed 98
143. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed • Divergence
 Divergence is a factor of synchronization period 98
144. 99 %% Generate a series of contracts. Contracts = declare(set), %% Generate advertisements. RiotAds = declare(set), RovioAds = declare(set), create_advertisements_and_contracts(RiotAds, Contracts), create_advertisements_and_contracts(RovioAds, Contracts), %% Union ads. Ads = declare(set), union(RovioAds, RiotAds, Ads), %% Compute the product of both ads and contracts. AdsContracts = declare(set), product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. AdsWithContracts = declare(set), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, filter(AdsContracts, FilterFun, AdsWithContracts).
145. 100 %% Generate a series of contracts. Contracts = declare(set), %% Generate advertisements. RiotAds = declare(set), RovioAds = declare(set), create_advertisements_and_contracts(RiotAds, Contracts), create_advertisements_and_contracts(RovioAds, Contracts), %% Union ads. Ads = declare(set), union(RovioAds, RiotAds, Ads), %% Compute the product of both ads and contracts. AdsContracts = declare(set), product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. AdsWithContracts = declare(set), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, filter(AdsContracts, FilterFun, AdsWithContracts).
146. 101 %% Generate a series of contracts. Contracts = declare(set), %% Generate advertisements. RiotAds = declare(set), RovioAds = declare(set), create_advertisements_and_contracts(RiotAds, Contracts), create_advertisements_and_contracts(RovioAds, Contracts), %% Union ads. Ads = declare(set), union(RovioAds, RiotAds, Ads), %% Compute the product of both ads and contracts. AdsContracts = declare(set), product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. AdsWithContracts = declare(set), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, filter(AdsContracts, FilterFun, AdsWithContracts).
147. 102 %% Generate a series of contracts. Contracts = declare(set), %% Generate advertisements. RiotAds = declare(set), RovioAds = declare(set), create_advertisements_and_contracts(RiotAds, Contracts), create_advertisements_and_contracts(RovioAds, Contracts), %% Union ads. Ads = declare(set), union(RovioAds, RiotAds, Ads), %% Compute the product of both ads and contracts. AdsContracts = declare(set), product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. AdsWithContracts = declare(set), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, filter(AdsContracts, FilterFun, AdsWithContracts).
148. 103 %% Generate a series of contracts. Contracts = declare(set), %% Generate advertisements. RiotAds = declare(set), RovioAds = declare(set), create_advertisements_and_contracts(RiotAds, Contracts), create_advertisements_and_contracts(RovioAds, Contracts), %% Union ads. Ads = declare(set), union(RovioAds, RiotAds, Ads), %% Compute the product of both ads and contracts. AdsContracts = declare(set), product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. AdsWithContracts = declare(set), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, filter(AdsContracts, FilterFun, AdsWithContracts).
149. 104 %% Generate a series of contracts. Contracts = declare(set), %% Generate advertisements. RiotAds = declare(set), RovioAds = declare(set), create_advertisements_and_contracts(RiotAds, Contracts), create_advertisements_and_contracts(RovioAds, Contracts), %% Union ads. Ads = declare(set), union(RovioAds, RiotAds, Ads), %% Compute the product of both ads and contracts. AdsContracts = declare(set), product(Ads, Contracts, AdsContracts), %% Filter items by join on item it. AdsWithContracts = declare(set), FilterFun = fun({#ad{id=Id1}, #contract{id=Id2}}) -> Id1 =:= Id2 end, filter(AdsContracts, FilterFun, AdsWithContracts).
150. Client3 Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Riot Ad Counter 1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 105
151. Client3 Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Riot Ad Counter 1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 106
152. Client3 Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Riot Ad Counter 1 Client1 Client2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Rovio Ad Counter 2 Rovio Ad Counter 1 Riot Ad Counter 2 Riot Ad Counter 1 Riot Ad Counter 1 Ads With Contracts Ads With Contracts Ads With Contracts Server Ads With Contracts Server Computation! 107
153. Advertisement Counter • “Servers” as peers to “clients”
 Servers are peers to clients that perform additional computation 108
154. Advertisement Counter • “Servers” as peers to “clients”
 Servers are peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information 108
155. Advertisement Counter • “Servers” as peers to “clients”
 Servers are peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information • “Servers” as trusted nodes
 Serve as a location for performing “exactly once” side- effects 108
156. Advertisement Counter • “Servers” as peers to “clients”
 Servers are peers to clients that perform additional computation • Any node can disable an advertisement under this model given enough information • “Servers” as trusted nodes
 Serve as a location for performing “exactly once” side- effects • Billing customers must be done at a central point by a trusted node in the system 108
157. We’ve build up from zero synchronization 109
158. We’ve build up from zero synchronization 109 Instead of working to remove synchronization
159. What have we learned? 110
160. Key Points • Synchronization is expensive
 Locking and other synchronization mechanisms limit scalability to the critical section 111
161. Key Points • Synchronization is expensive
 Locking and other synchronization mechanisms limit scalability to the critical section • Synchronization is sometimes not possible
 Mobile and “Internet of Things” applications make synchronization for replicated state impractical 111
162. Key Points • Synchronization is expensive
 Locking and other synchronization mechanisms limit scalability to the critical section • Synchronization is sometimes not possible
 Mobile and “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required
 Global invariants, atomic visibility, etc. 111
163. How do I learn more? 112
164. Publications • “Lasp: A Language for Distributed, Coordination-Free Programming” 
 ACM SIGPLAN PPDP 2015 • “Selective Hearing: An Approach to Distributed, Eventually Consistent Edge Computation”
 IEEE W-PSDS 2015 • “The Implementation and Use of a Generic Dataflow Behaviour in Erlang”
 ACM SIGPLAN Erlang Workshop ’15 • “Lasp: A Language for Distributed, Eventually Consistent Computations with CRDTs"
 PaPoC 2015 • “Declarative, Sliding Window Aggregations for Computations at the Edge"
 IEEE EdgeCom 2016 113
165. SyncFree is a European research project taking place for 3 years, staring October 2013, and is funded by the European Union, grant agreement n° 609551. 114
166. Thanks! 115 Christopher Meiklejohn @cmeik