Controlled Epidemics: Riak's New Gossip Protocol and Metadata Store

Users are familiar with how Riak stores their data, but what about its own internal information like node capabilities and ownership details? That data is stored in one of Riak's internal distributed data stores, of course!

This talk will cover one of these stores: Cluster Metadata. Cluster Metadata is an internal, fully-replicated, eventually consistent DHT that will be included in the next release of Riak. The talk will cover, briefly, how this store is used, why it was built, and what improvements it will bring. The primary focus will be covering its design and implementation through some formal, and not so formal, models of its replication and anti-entropy protocols.

Transcript

1. Qh‘j&r Mdv Aqn‘cb‘rs Oqnsnbnk % Lds‘c‘s‘ Rsnqd CONTROLLED EPIDEMICS: Jordan West ?^iqvdrs
2. Existing Gossip Odqhnchb ‘mc bg‘mfd,sqhffdqdc Fq‘pg Nudqk‘x Eknnc eqnl q‘mcnl mncd
3. PROBLEM? K‘rs,Vqhsd Vhmr Bnkc Qtlnq Rpqd‘chmf Aqn‘cb‘rs pdq Qhmf Bg‘mfd L‘mx Qdctmc‘ms Ldrr‘fdr
4. Cluster Metadata
5. Overview Jdx,U‘ktd Rsnqd Dudmst‘kkx Bnmrhrsdms Etkkx Qdpkhb‘sdc @rxmbgqnmntrkx Cluster Metadata
6. Overview Knb‘k Qd‘cr ‘mc Vqhsdr ’v<0+q<0( Hm,Ldlnqx ‘mc Nm,Chrj Knfhb‘k Bknbjr Knb‘k Cdsdbshnm ne Bg‘mfdc Jdxr Cluster Metadata
7. Cluster Metadata Dphcdlhb Aqn‘cb‘rs Sqddr ’Oktlsqdd( gssp9..hdddwpknqd-hddd-nqf.wpk.‘qshbkdCds‘hkr-irp>qdkn‘c<sqtd%‘qmtladq<32546/4
8. Cluster Metadata High Reliability Low Reliability Low Redundancy High Redundancy
9. Cluster Metadata High Reliability Low Reliability Low Redundancy High Redundancy Aqn‘cb‘rs
10. Cluster Metadata High Reliability Low Reliability Low Redundancy High Redundancy Fnrrhp Aqn‘cb‘rs
11. Cluster Metadata High Reliability Low Reliability Low Redundancy High Redundancy Fnrrhp Aqn‘cb‘rs Oktlsqdd
12. Cluster Metadata Self-Healing Spanning Tree
13. Plumtree by Example
14. Plumtree by Example
15. Plumtree by Example 1 2 3 4 5
16. Plumtree by Example 1 2 3 4 5 {2,3}, {1,4}, {1,5}, {2,5}, {1,4}, eager
17. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {1,4}, {}, eager lazy
18. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {} {1,4}, {}, {} {1,5}, {}, {} {2,5}, {}, {} {1,4}, {}, {} eager lazy msgs
19. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {} {1,4}, {}, {} {1,5}, {}, {} {2,5}, {}, {} {1,4}, {}, {} eager lazy msgs
20. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {} {1,4}, {}, {} {1,5}, {}, {} {2,5}, {}, {} {1,4}, {}, {} ab‘rs z‘+/+p‘xkn‘c| eager lazy msgs
21. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {} {1,5}, {}, {} {2,5}, {}, {} {1,4}, {}, {} {a} ab‘rs z‘+/+p‘xkn‘c| eager lazy msgs
22. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {} {1,5}, {}, {} {2,5}, {}, {} {1,4}, {}, {} {a} ab‘rs z‘+/+p‘xkn‘c| eager lazy msgs
23. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {} {1,5}, {}, {2,5}, {}, {} {1,4}, {}, {} {a} {a} ab‘rs z‘+/+p‘xkn‘c| eager lazy msgs
24. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {} {1,5}, {}, {2,5}, {}, {} {1,4}, {}, {} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| eager lazy msgs
25. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {} {1,4}, {}, {} {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| eager lazy msgs
26. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {} {1,4}, {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} eager lazy msgs
27. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {} {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {1,3,4}, eager lazy msgs
28. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {} {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} ab‘rs z‘+1+p‘xkn‘c| {1,3,4}, eager lazy msgs
29. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} ab‘rs z‘+1+p‘xkn‘c| {1,3,4}, ab‘rs z‘+1+p‘xkn‘c| eager lazy msgs
30. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} ab‘rs z‘+1+p‘xkn‘c| {1,3,4}, ab‘rs z‘+1+p‘xkn‘c| eager lazy msgs
31. Plumtree by Example 1 2 3 4 5 {2,3}, {}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} ab‘rs z‘+1+p‘xkn‘c| {1,3,4}, ab‘rs z‘+1+p‘xkn‘c| eager lazy msgs
32. Plumtree by Example 1 2 3 4 5 {2,3}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} ab‘rs z‘+1+p‘xkn‘c| {1,3,4}, ab‘rs z‘+1+p‘xkn‘c| {5}, eager lazy msgs
33. Plumtree by Example 1 2 3 4 5 {2,3}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} ab‘rs z‘+1+p‘xkn‘c| {1,3,4}, ab‘rs z‘+1+p‘xkn‘c| {5}, pqtmd eager lazy msgs
34. Plumtree by Example 1 2 3 4 5 {2,3}, {1,4}, {}, {1,5}, {}, {2,5}, {}, {a} {a} {a} ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} ab‘rs z‘+1+p‘xkn‘c| ab‘rs z‘+1+p‘xkn‘c| {5}, pqtmd {3,4}, {1}, eager lazy msgs
35. Plumtree by Example 1 2 3 4 5 1 2 3 4 5
36. Plumtree by Example 1 2 3 4 5 1 2 3 4 5
37. Plumtree by Example 1 2 3 4 5 {2,3}, {5}, {} {1,4}, {}, {} {1,5}, {}, {} {2}, {5}, {} {3}, {1,4}, {} eager lazy msgs
38. Plumtree by Example 1 2 4 5 {2,3}, {5}, {} {1,4}, {}, {} {1,5}, {}, {} {2}, {5}, {} {3}, {1,4}, {} 3 eager lazy msgs
39. Plumtree by Example 1 2 4 5 {2,3}, {5}, {1,4}, {}, {1,5}, {}, {} {2}, {5}, {} {3}, {1,4}, {} 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} eager lazy msgs
40. Plumtree by Example 1 2 4 5 {2,3}, {5}, {1,4}, {}, {1,5}, {}, {} {2}, {5}, {} {3}, {1,4}, {} 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} hg‘ud z‘+/| eager lazy msgs
41. Plumtree by Example 1 2 4 5 {2,3}, {5}, {1,4}, {}, {1,5}, {}, {} {2}, {5}, {} {} 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} hg‘ud z‘+/| {1,3}, {4}, eager lazy msgs
42. Plumtree by Example 1 2 4 5 {2,3}, {5}, {1,4}, {}, {1,5}, {}, {} {2}, {5}, {} {} 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} hg‘ud z‘+/| fq‘es z‘+/| {1,3}, {4}, eager lazy msgs
43. Plumtree by Example 1 2 4 5 {1,4}, {}, {1,5}, {}, {} {2}, {5}, {} {} 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} hg‘ud z‘+/| fq‘es z‘+/| {1,3}, {4}, {2,3,5}, {}, eager lazy msgs
44. Plumtree by Example 1 2 4 5 {1,4}, {}, {1,5}, {}, {} {2}, {5}, {} {} 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} hg‘ud z‘+/| fq‘es z‘+/| {1,3}, {4}, ab‘rs z‘+/+p‘xkn‘c| {2,3,5}, {}, eager lazy msgs
45. Plumtree by Example 1 2 4 5 {1,4}, {}, {1,5}, {}, {} {2}, {5}, {} {} 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| {a} {a} hg‘ud z‘+/| fq‘es z‘+/| {1,3}, {4}, hg‘ud z‘+0| ab‘rs z‘+/+p‘xkn‘c| {2,3,5}, {}, eager lazy msgs
46. Plumtree by Example 1 2 4 5 3 1 2 4 5
47. Plumtree by Example 1 2 4 5 3 1 2 4 5 3
48. Plumtree by Example 1 2 3 4 5 1 2 3 4 5 1 2 4 5 3
49. Plumtree
50. Plumtree in Riak
51. Plumtree in Riak Overlay Peer Service
52. Plumtree in Riak Overlay Membership & Critical Data
53. Plumtree in Riak Message Delivery Membership & Critical Data
54. Plumtree in Riak Plumtree HyParView (or Other) Epidemic Broadcast Trees riak_core_broadcast Existing Membership Riak
55. Plumtree in Riak Peer Service Requirements
56. Plumtree in Riak Peer Service Requirements Rsqnmf Bnmmdbshuhsx Vd‘jdq Bnmmdbshuhsx
57. Plumtree in Riak Peer Service Requirements Rsqnmf Bnmmdbshuhsx 0/j* Mncdr Vd‘jdq Bnmmdbshuhsx 4 , 0// Mncdr
58. Plumtree in Riak Peer Service Requirements Rsqnmf Bnmmdbshuhsx 0/j* Mncdr O‘qsh‘kkx Bnmmdbsdc Vd‘jdq Bnmmdbshuhsx 4 , 0// Mncdr Etkkx Bnmmdbsdc
59. Plumtree in Riak Peer Service Requirements Rsqnmf Bnmmdbshuhsx 0/j* Mncdr O‘qsh‘kkx Bnmmdbsdc Qd‘bshud Ldladqrghp Vd‘jdq Bnmmdbshuhsx 4 , 0// Mncdr Etkkx Bnmmdbsdc Npdq‘snq,Bnmsqnkkdc
60. Plumtree in Riak “[Plumtree] is able to support large number of faults while maintaining a high reliability.” Plumtree Focus
61. Plumtree in Riak “[Plumtree] is able to support large number of faults while maintaining a high reliability.” Plumtree Focus
62. Plumtree in Riak 1 2 4 5 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c|
63. Plumtree in Riak 1 2 4 5 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c|
64. Plumtree in Riak 1 2 4 5 3 ab‘rs z‘+/+p‘xkn‘c| ab‘rs z‘+0+p‘xkn‘c| hg‘ud z‘+/| hg‘ud z‘+0|
65. Plumtree in Riak 1 2 4 5 3 bcast {a,0,payload} bcast {a,1,payload}
66. Plumtree in Riak Add to Outstanding Set before Send
67. Plumtree in Riak Add to Outstanding Set before Send Remove on Graft or Ack
68. Plumtree in Riak New Message (ack): ignore_i_have
69. Plumtree in Riak “...for failure percentages above 70%, reliability drops to 0...” Massive Failures
70. Plumtree in Riak “...for failure percentages above 70%, reliability drops to 0...” Massive Failures
71. 5 2 3 Plumtree in Riak 1 4
72. 5 2 3 Plumtree in Riak 1 4 3 2 5
73. 5 2 3 Plumtree in Riak 1 4 3 2 5
74. Plumtree in Riak Reliability of What Messages?
75. Plumtree in Riak Compare the Database
76. Plumtree in Riak Compare the Database with Trees of Hash Trees
77. Plumtree in Riak Shared vs. Sender Based Trees
78. Plumtree in Riak Shared vs. Sender Based Trees
79. Results
80. Results Environment 1 2 3 4 5 2
81. Results Environment 1 2 3 4 5 2 “network”
82. Results Riak Test 1 2 3 4 5 2 process
83. Results Relative Message Redundancy
84. Results Relative Message Redundancy # of Payloads Sent - 1 (num. nodes - 1)
85. Last Delivery Hop Ghfgdrs Qntmc hm Aqn‘cb‘rs Ld‘rtqd ne K‘sdmbx Sqdd Ch‘ldsdq ’eqnl Qnns( Results
86. RMR - 5 Nodes - Stable - 20 Rounds Results
87. RMR - 5 Nodes - Stable - 20 Rounds Qntmc QLQ / 1 3 5 7 0 1 2 3 4 5 6 7 8 0/ 00 01 02 03 04 05 06 07 08 1/ Aqn‘cb‘rs Fnrrhp Results
88. Controlled Epidemics LDH - 5 Nodes - Stable - 20 Rounds Results
89. Controlled Epidemics LDH - 5 Nodes - Stable - 20 Rounds Qntmc KCG / 1-14 3-4 5-64 8 0 1 2 3 4 5 6 7 8 0/ 00 01 02 03 04 05 06 07 08 1/ Aqn‘cb‘rs Fnrrhp Results
90. Controlled Epidemics RMR - 10, 20, 40 Nodes - Stable - 20 Rounds Qntmc QLQ / /-14 /-4 /-64 0 0 1 2 3 4 5 6 7 8 0/ 00 01 02 03 04 05 06 07 08 1/ 0/ Mncdr 1/ Mncdr 3/ Mncdr Results
91. Controlled Epidemics LDH - 10, 20, 40 Nodes - Stable - 20 Rounds Qntmc KCG / 0-4 2 3-4 5 0 1 2 3 4 5 6 7 8 0/ 00 01 02 03 04 05 06 07 08 1/ 0/ Mncdr 1/ Mncdr 3/ Mncdr Results
92. Controlled Epidemics Avg. LDH - 10, 20, 40 Nodes - Stable / 0-14 1-4 2-64 4 0/ Mncdr 1/ Mncdr 3/ Mncdr KCG Results
93. Controlled Epidemics % Failures Before Exchange Needed Results
94. Controlled Epidemics % Failures Before Exchange Needed / 11-4 34 56-4 8/ 4 Mncdr 0/ Mncdr 1/ Mncdr 3/ Mncdr @uf- $ E‘hktqdr Results
95. Controlled Epidemics % Failures Before Exchange Needed Results
96. Controlled Epidemics % Failures Before Exchange Needed / 11-4 34 56-4 8/ 0/ Mncdr 1/ Mncdr 3/ Mncdr E‘mnts < 3 E‘mnts < 1 Results
97. Controlled Epidemics % Failures Before Exchange Needed (Variance) Results
98. Controlled Epidemics % Failures Before Exchange Needed (Variance) / 0-64 2-4 4-14 6 0/ Mncdr 1/ Mncdr 3/ Mncdr E‘mnts < 3 E‘mnts < 1 Results
99. Cluster Metadata in Riak
100. In Riak 2.0 Atbjds Sxpdr Sxpdc Atbjdsr Oqnpdqshdr Rdbtqhsx Nsgdq Mdv Ed‘stqdr Cluster Metadata
101. In Riak Future @kk Atbjds Oqnpdqshdr Mncd B‘p‘ahkhshdr Lhfq‘sd Nsgdq Dwhrshmf Ed‘stqdr Lnqd Mdv Ed‘stqdr Cluster Metadata
102. Future Work Ldlnqx,A‘bjdc G‘rg Sqddr Rl‘qsdq A‘bjnee.Dwbg‘mfd Oktlsqdd Npshlhy‘shnm> Bg‘mfd Mnshehb‘shnm Cluster Metadata
103. Thanks
104. Questions