Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Declarative, Convergent, Edge Computation

Declarative, Convergent, Edge Computation

Craft Conference, 2017

Christopher Meiklejohn

April 28, 2017
Tweet

More Decks by Christopher Meiklejohn

Other Decks in Research

Transcript

  1. Declarative, Convergent, Edge Computation Christopher S. Meiklejohn Université catholique de

    Louvain, Belgium Instituto Superior Técnico, Portugal LIGHT ONE
  2. RA RB 1 set(1) 3 2 set(2) set(3) ? ?

    Nondeterministic outcome.
  3. Synchronization • To enforce an order
 Makes programming easier •

    Eliminate accidental nondeterminism
 Prevent race conditions 6
  4. Synchronization • To enforce an order
 Makes programming easier •

    Eliminate accidental nondeterminism
 Prevent race conditions • Techniques
 Locks, mutexes, semaphores, monitors, Paxos, Raft, etc. 6
  5. Difficult Cases • “Internet of Things”, 
 Low power, limited

    memory and connectivity • Mobile Gaming
 Offline operation with replicated, shared state 7
  6. 8

  7. Distributed Computation • Concurrent to distributed programming
 Consistency and now

    partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” 10
  8. Distributed Computation • Concurrent to distributed programming
 Consistency and now

    partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models
 Contract between the system and the programmer 10
  9. Distributed Computation • Concurrent to distributed programming
 Consistency and now

    partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models
 Contract between the system and the programmer • Correctness criteria
 For both distributed and concurrent programs 10
  10. Distributed Computation • Concurrent to distributed programming
 Consistency and now

    partial failure • Enforcing the “single system” illusion
 Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models
 Contract between the system and the programmer • Correctness criteria
 For both distributed and concurrent programs • Synchronization as “knob”
 Models live along a spectrum requiring more or less synchronization 10
  11. R2 C1 C2 Value 2 Value 1 Value 2 R1

    R3 Paxos 22 Coordination is extremely expensive.
  12. R1 R2 R3 C1 C2 Write C1 30 Do I

    store both values or do I choose a value arbitrarily?
  13. R1 R2 R3 C1 C2 Read 31 If I store

    both, the programmer must expect multiple values…
  14. R1 R2 R3 C1 C2 Write 32 …and be able

    to semantically resolve them in application code.
  15. Strong Eventual Consistency (SEC) • Convergent
 Same updates realize the

    same state • Primary requirement
 “Eventual” replica-to-replica communication 33
  16. Strong Eventual Consistency (SEC) • Convergent
 Same updates realize the

    same state • Primary requirement
 “Eventual” replica-to-replica communication • Order insensitive!
 Operations are commutative 33
  17. Strong Eventual Consistency (SEC) • Convergent
 Same updates realize the

    same state • Primary requirement
 “Eventual” replica-to-replica communication • Order insensitive!
 Operations are commutative • Duplicate insensitive!
 Operations are idempotent 33
  18. RA RB 1 3 2 3 3 set(1) set(2) set(3)

    max(2,3) max(2,3) Automatic resolution of conflicting updates.
  19. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic

    operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 39
  20. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic

    operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 39
  21. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic

    operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 40
  22. Conflict-Free 
 Replicated Data Types • Many types exist with

    different properties
 Sets, counters, registers, flags, maps 42
  23. Conflict-Free 
 Replicated Data Types • Many types exist with

    different properties
 Sets, counters, registers, flags, maps • Strong Eventual Consistency
 Instances satisfy SEC property per-object 42
  24. Conflict-Free 
 Replicated Data Types • Many types exist with

    different properties
 Sets, counters, registers, flags, maps • Strong Eventual Consistency
 Instances satisfy SEC property per-object • Bounded join-semilattices
 Formalized using bounded join-semilattices where the merge operation is the join 42
  25. RA RB RC {1} (1, {a}, {}) add(1) {1} (1,

    {b}, {}) add(1) {} (1, {b}, {b}) remove(1)
  26. RA RB RC {1} (1, {a}, {}) add(1) {1} (1,

    {b}, {}) add(1) {} (1, {b}, {b}) remove(1) {1} {1} {1} (1, {a, b}, {b}) (1, {a, b}, {b}) (1, {a, b}, {b}) Convergence reached.
  27. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Replicated set of naturals across two nodes.
  28. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} Map \x.2x over a set of naturals.
  29. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} One node…
  30. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} …another node.
  31. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2}
  32. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} Nondeterministic outcome.
  33. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} Correct output that’s seen all updates.
  34. RA {1} (1, {a}, {}) {1} (1, {a, b}, {a})

    RB {1} (1, {b}, {}) {1} (1, {a, b}, {a}) {} (1, {a}, {a}) add(1) remove(1) add(1) Map Process \x.2x F(RA) {2} {2} {} F(RB) {2} ? Map Process \x.2x Map Process \x.2x Map Process \x.2x Map Process \x.2x {2} “Earlier” value that’s been delayed.
  35. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic

    operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 58
  36. Lattice Processing • Asynchronous dataflow with streams
 Combine and transform

    streams of inputs into streams of outputs • Convergent data structures
 Data abstraction (inputs/outputs) is the CRDT 60
  37. Lattice Processing • Asynchronous dataflow with streams
 Combine and transform

    streams of inputs into streams of outputs • Convergent data structures
 Data abstraction (inputs/outputs) is the CRDT • Confluence
 Provides composition that preserves the SEC property 60
  38. A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x

    A B Map Process \x.2x 63 Replication per node.
  39. A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x

    A B Map Process \x.2x 64 Replication per node.
  40. A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x

    A B Map Process \x.2x 65 Replication per node.
  41. A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x

    A B Map Process \x.2x 66 One possible schedule….
  42. A’ A’’ B’ B’’ Map Process \x.2x Map Process \x.2x

    A B Map Process \x.2x 67 …another possible schedule.
  43. 72 %% Create initial set. S1 = declare(set), %% Add

    elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
  44. 73 %% Create initial set. S1 = declare(set), %% Add

    elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
  45. 74 %% Create initial set. S1 = declare(set), %% Add

    elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
  46. 75 %% Create initial set. S1 = declare(set), %% Add

    elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
  47. 76 %% Create initial set. S1 = declare(set), %% Add

    elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
  48. Processes • Replicas as monotonic streams
 Each replica of a

    CRDT produces a monotonic stream of states 77
  49. Processes • Replicas as monotonic streams
 Each replica of a

    CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream 77
  50. Processes • Replicas as monotonic streams
 Each replica of a

    CRDT produces a monotonic stream of states • Monotonic processes
 Read from one or more input replica streams and produce a single output replica stream • Inflationary reads
 Read operation ensures that we only read inflationary updates to replicas 77
  51. RA {} (1, {a}, {}) (1, {a, b}, {}) C1

    (1, {a}, {}) {} C2 (1, {b}, {}) {} 81
  52. RA {} (1, {a}, {}) (1, {a, b}, {}) (1,

    {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} 82
  53. RA {} (1, {a}, {}) (1, {a, b}, {}) (1,

    {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 83
  54. RA {} (1, {a}, {}) (1, {a, b}, {}) (1,

    {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 84 Clients can operate with partial state…
  55. RA {} (1, {a}, {}) (1, {a, b}, {}) (1,

    {a, b}, {a}) C1 (1, {a}, {}) {} (1, {a}, {a}) C2 (1, {b}, {}) {} (1, {a, b}, {a}) (1, {a, b}, {a}) (1, {a, b}, {a}) 85 … and synchronize with their local replica.
  56. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1,

    {a}, {})) (2, {a}, {}) 91 Every time replica changes…
  57. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1,

    {a}, {})) (2, {a}, {}) 92 ….the process will compute a new result.
  58. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1,

    {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) 93
  59. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) F((1,

    {a}, {})) (2, {a}, {}) strict_read((1, {a}, {}) (1, {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 94
  60. RA {} P1 F(RA) {} strict_read({}) (1, {a}, {}) (1,

    {a}, {}) (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a}) strict_read((1, {a}, {a}) (1, {a}, {a}) 95 Omitted interleaving does not sacrifice correctness.
  61. RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1,

    {b}, {}) {1} 98 Transform element, map metadata.
  62. RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1,

    {b}, {}) {1} {} (1, {b}, {b}) {} Map Process \x.2x (2, {b}, {b}) 99
  63. RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1,

    {b}, {}) {1} {} (1, {b}, {b}) {} Map Process \x.2x (2, {b}, {b}) {1} (1, {a, b}, {b}) {2} Map Process \x.2x (2, {a, b}, {b}) 100
  64. RC F(RC) {2} Map Process \x.2x (2, {b}, {}) (1,

    {b}, {}) {1} {} (1, {b}, {b}) {} Map Process \x.2x (2, {b}, {b}) {1} (1, {a, b}, {b}) {2} Map Process \x.2x (2, {a, b}, {b}) {1} {2} 101
  65. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic

    operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 102
  66. Advertisement Counter • Lower-bound invariant
 Advertisements are paid according to

    a minimum number of impressions • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline 104
  67. Advertisement Counter • Lower-bound invariant
 Advertisements are paid according to

    a minimum number of impressions • Clients will go offline
 Clients have limited connectivity and the system still needs to make progress while clients are offline • No lost updates
 All displayed advertisements should be accounted for, with no lost updates 104
  68. RA RB 1 set(1) 2 2 set(2) set(2) 2 2

    max(2,2) max(2,2) 109
  69. RA RB 1 set(1) 2 2 set(2) set(2) 2 2

    max(2,2) max(2,2) 110 Incorrect value is computed because of incompatible lattice.
  70. Ads Contracts Product Filter Map Ads X Contracts Ads With

    Contracts Ads To Display Asynchronous Dataflow Counters Counters Counters Counter Ads Update (Insert) Application Initialization Counter Read > 50,000 Ads Update (Remove) Epsilon-Invariant Ads Map Configure Invariant Enforce Invariant 121
  71. Ads Contracts Product Filter Map Ads X Contracts Ads With

    Contracts Ads To Display Asynchronous Dataflow 122
  72. Counter Read > 50,000 Ads Update (Remove) Epsilon-Invariant Ads Map

    Configure Invariant Enforce Invariant 125 Configure invariants for all of the advertisements.
  73. Counter Read > 50,000 Ads Update (Remove) Epsilon-Invariant Ads Map

    Configure Invariant Enforce Invariant 126 Remove the advertisement from the list.
  74. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are

    all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed 127
  75. Advertisement Counter • Completely monotonic
 Disabling advertisements and contracts are

    all modeled through monotonic state growth • Arbitrary distribution
 Use of convergent data structures allows computational graph to be arbitrarily distributed • Divergence
 Divergence is a factor of synchronization period, concurrency, and throughput rate 127
  76. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic

    operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 128
  77. Anabranch • Layered approach
 Cluster membership and state dissemination for

    large clusters • Delta-state synchronization 
 Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] 130
  78. Anabranch • Layered approach
 Cluster membership and state dissemination for

    large clusters • Delta-state synchronization 
 Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] • Epsilon-invariants
 Lower-bound invariants, configurable at runtime 130
  79. Anabranch • Layered approach
 Cluster membership and state dissemination for

    large clusters • Delta-state synchronization 
 Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] • Epsilon-invariants
 Lower-bound invariants, configurable at runtime • Scalable
 Demonstrated high scalability in production Cloud environments 130
  80. Layered Approach • Backend
 Configurable persistence layer depending on application.

    • Membership
 Configurable membership protocol which can operate in a client-server or peer-to-peer mode [Leitao et al. 2007] 132
  81. Layered Approach • Backend
 Configurable persistence layer depending on application.

    • Membership
 Configurable membership protocol which can operate in a client-server or peer-to-peer mode [Leitao et al. 2007] • Broadcast (via Gossip, Tree, etc.)
 Efficient dissemination of both program state and application state via gossip, broadcast tree, or hybrid mode [Leitao et al. 2007] 132
  82. Mobile Phone Distributed Hash Table Application Language Execution KV Store

    KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer
  83. Mobile Phone Distributed Hash Table Application Language Execution KV Store

    KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer Language and applications.
  84. Mobile Phone Distributed Hash Table Application Language Execution KV Store

    KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer Storage for CRDT state.
  85. Mobile Phone Distributed Hash Table Application Language Execution KV Store

    KV Backend Membership Broadcast Layer Client/Server Peer-to-Peer State dissemination.
  86. Delta-based Dissemination • Delta-state based CRDTs
 Reduces state transmission for

    clients • Operate locally
 Objects are mutated locally; delta’s buffered locally and periodically gossiped 138
  87. Delta-based Dissemination • Delta-state based CRDTs
 Reduces state transmission for

    clients • Operate locally
 Objects are mutated locally; delta’s buffered locally and periodically gossiped • Only fixed number of clients
 Clients resort to full state synchronization when they’ve been partitioned too long 138
  88. RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {})

    (1, {b}, {}) Buffer minimal change representation…
  89. RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {})

    (1, {b}, {}) {} (1, {b}, {b}) (1, {b}, {b}) (1, {b}, {b})
  90. RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {})

    (1, {b}, {}) {} (1, {b}, {b}) (1, {b}, {b}) (1, {b}, {b}) {2} (1, {b}, {b}) (2, {c}, {}) (2, {c}, {}) (2, {c}, {}) …then, disseminate state in causal order to neighbors.
  91. RC BufferA BufferB (1, {b}, {}) {1} (1, {b}, {})

    (1, {b}, {}) {} (1, {b}, {b}) (1, {b}, {b}) (1, {b}, {b}) {2} (1, {b}, {b}) (2, {c}, {}) (2, {c}, {}) (2, {c}, {}) {1, 2} (1, {a, b}, {b}) (2, {c}, {}) {1, 2} (1, {a}, {}) (1, {a}, {}) Only ship inflation from incoming state.
  92. Scalability • 1024+ nodes
 Demonstrated scalability to 1024 nodes in

    Amazon cloud computing environment • Modular approach
 Many of the components built and can be operated outside of Lasp to improve scalability of Erlang 145
  93. Scalability • 1024+ nodes
 Demonstrated scalability to 1024 nodes in

    Amazon cloud computing environment • Modular approach
 Many of the components built and can be operated outside of Lasp to improve scalability of Erlang • Automated and repeatable
 Fully automated deployment, scenario execution, log aggregation and archival of experimental results 145
  94. Programming SEC 1. Eliminate accidental nondeterminism
 (ex. deterministic, modeling non-monotonic

    operations monotonically) 2. Retain the properties of functional programming
 (ex. confluence, “correct-by-construction” applications) 3. Distributed, and fault-tolerant runtime
 (ex. replication, membership, dissemination) 146
  95. Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated

    Data Types [Shapiro et al. 2011] 2. Retain the properties of functional programming
 Convergent Programs: Lattice Processing
 [Meiklejohn et al. 2015] 149
  96. Programming SEC 1. Eliminate accidental nondeterminism
 Convergent Objects: Conflict-Free Replicated

    Data Types [Shapiro et al. 2011] 2. Retain the properties of functional programming
 Convergent Programs: Lattice Processing
 [Meiklejohn et al. 2015] 3. Distributed, and fault-tolerant runtime
 Distributed Runtime: Anabranch
 [Meiklejohn et al. work-in-progress] 149
  97. Key Points • Synchronization is sometimes not possible
 Mobile and

    “Internet of Things” applications make synchronization for replicated state impractical 150
  98. Key Points • Synchronization is sometimes not possible
 Mobile and

    “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required
 Global invariants, atomic visibility, etc. 150
  99. Key Points • Synchronization is sometimes not possible
 Mobile and

    “Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required
 Global invariants, atomic visibility, etc. • Holistic approach
 Taking a holistic approach to the design of distributed systems is important for building higher-availability applications 150
  100. Artifacts • Lasp
 CRDT based programming model: testbed for Loquat


    http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan 151
  101. Artifacts • Lasp
 CRDT based programming model: testbed for Loquat


    http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan • Plumtree
 Epidemic broadcast trees for use with Partisan
 https://github.com/lasp-lang/plumtree 151
  102. Artifacts • Lasp
 CRDT based programming model: testbed for Loquat


    http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan • Plumtree
 Epidemic broadcast trees for use with Partisan
 https://github.com/lasp-lang/plumtree • Sprinter
 Service discovery and deployment for Mesos and Kubernetes
 https://github.com/lasp-lang/sprinter 151
  103. Artifacts • Lasp
 CRDT based programming model: testbed for Loquat


    http://github.com/lasp-lang/lasp • Partisan
 TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols
 https://github.com/lasp-lang/partisan • Plumtree
 Epidemic broadcast trees for use with Partisan
 https://github.com/lasp-lang/plumtree • Sprinter
 Service discovery and deployment for Mesos and Kubernetes
 https://github.com/lasp-lang/sprinter • Types
 CRDT implementations for Erlang
 https://github.com/lasp-lang/types 151