Yokozuna: Scaling Solr With Riak

Transcript

1. Yokozuna, Scaling Solr With Riak Ryan Zezeski Berlin Buzzwords - June 4th 2013 1
2. WHO AM I? • DEVELOPER @ BASHO TECHNOLOGIES • PREVIOUS @ AOL FOR ADVERTISING.COM • MOST EXPERIENCE IN JAVA & ERLANG • 2+ YEARS WORKING ON SEARCH • @RZEZESKI ON TWITTER 2
3. NOT TALKING ABOUT SEARCH 3
4. AGENDA • OVERVIEW OF RIAK & YOKOZUNA • DATA PARTITIONING & OWNERSHIP • HIGH AVAILABILITY & CONSISTENCY • SELF HEALING (ANTI ENTROPY) • DEMOS 4
5. WHAT IS RIAK? • KEY-VALUE STORE (+ SOME EXTRAS) • DISTRIBUTED • HIGHLY AVAILABLE • MASTERLESS • EVENTUALLY CONSISTENT • SCALE UP/DOWN 5
6. DATABASE • KEY/VALUE MODEL • BASIC SECONDARY INDEX SUPPORT • MAP/REDUCE (NOT LIKE HADOOP) • SEARCH (YOKOZUNA/SOLR) 6
7. DISTRIBUTED • MANY NODES IN LAN • RECOMMEND STARTING WITH 5 • ENTERPRISE REPLICATION CAN SPAN WAN 7
8. HIGH AVAILABILITY • ALWAYS TAKE WRITES • ALWAYS SERVICE READS • FAVORS YIELD OVER HARVEST • IMPLIES EVENTUAL CONSISTENCY 8
9. MASTERLESS • NO NOTION OF MASTER OR SLAVE • ANY NODE MAY SERVICE READ/WRITE/ QUERY 9
10. EVENTUALLY CONSISTENT • READS CAN BE STALE • CONCURRENT WRITES CAN CAUSE SIBLINGS • EVENTUALLY VALUES CONVERGES 10
11. YOKOZUNA • INTEGRATION OF RIAK AND SOLR • INDEX RIAK DATA WITH SOLR • DISTRIBUTE SOLR WITH RIAK • TOGETHER DO WHAT EACH ALONE CANNOT 11
12. YOKOZUNA • EACH NODE RUN A LOCAL SOLR INSTANCE • CREATE AN INDEX SAME AS BUCKET NAME • DOCUMENT IS “EXTRACTED” FROM VALUE • SUPPORTS PLAIN TEXT, XML, AND JSON • SOLR CELL SUPPORT COMING SOON 12
13. YOKOZUNA • SUPPORTS “TAGGING” • USE SOLR QUERY SYNTAX • PARAMETERS PASSED VERBATIM • IF DISTRIBUTED SEARCH SUPPORTS IT - YOKOZUNA SUPPORTS IT • NO SOLR CLOUD INVOLVED 13
14. PARTITIONING & OWNERSHIP Aufteilen & Eigentum 14
15. NAIVE HASHING NODE # = HASH(KEY) % NUM_NODES NH(Ka) = 0 NH(Kb) = 1 NH(Kc) = 2 NH(Kd) = 0 ... 15
16. NAIVE HASHING NODE 0 NODE 1 NODE 2 Ka Kb Kc Kd Ke Kf Kg Kh Ki Kj Kk Km Kl Kp Kn Ko Kq Kr 16
17. NAIVE HASHING NODE 0 NODE 1 NODE 2 Ka Kb Kc Kd Kg Ki NODE 3 Ke Kf Kh Kj Kk Kl Km Kn Ko Kp Kq Kr 17
18. NAIVE HASHING K * (NN - 1) / NN => K • K = # OF KEYS • NN = # OF NODES • AS NN GROWS FACTOR ESSENTIALLY BECOMES 1, THUS ALL KEYS MOVE 18
19. CONSISTENT HASHING PARTITION # = HASH(KEY) % PARTITIONS • # PARTITIONS REMAINS CONSTANT • KEY ALWAYS MAPS TO SAME PARTITION • NODES OWN PARTITIONS • PARTITIONS CONTAIN KEYS • EXTRA LEVEL OF INDIRECTION 19
20. P9 P6 P3 P8 P5 P2 P7 P4 P1 CONSISTENT HASHING NODE 0 NODE 1 NODE 2 Ka Kb Kc Kd Ke Kf Kg Kh Ki Kj Kk Km Kl Kp Kn Ko Kq Kr 20
21. P9 P6 P3 P8 P5 P2 P7 P4 P1 CONSISTENT HASHING NODE 0 NODE 1 NODE 2 Ka Kb Kc Kd Ke Kf Kg Kh Ki Kj Kk Km Kl Kp Kn Ko Kq Kr NODE 3 21
22. CONSISTENT HASHING NN * K/Q => K/Q • K = # OF KEYS • NN = # OF NODES • Q = # OF PARTITIONS • AS K GROWS NN BECOMES CONSTANT, THUS K/Q KEYS MOVE 22
23. CONSISTENT HASHING • EVENLY DIVIDES KEYSPACE • LOGICAL PARTITIONING SEPARATED FROM PHYSICAL PARTITIONING • UNIFORM HASH GIVES UNIFORM DISTRIBUTION 23
24. THE RING P1 P2 P3 P4 P5 P6 P7 P8 24
25. THE RING P1 P2 P3 P4 P5 P6 P7 P8 ND0 ND1 ND2 ND0 ND1 ND2 ND0 ND1 25
26. THE RING P1 P2 P3 P4 P5 P6 P7 P8 ND3 ND1 ND2 ND0 ND3 ND2 ND0 ND1 26
27. THE RING • GOSSIPED BETWEEN NODES • EPOCH CONSENSUS BASED • MASTERLESS - ANY NODE CAN SERVICE ANY REQUEST 27
28. WRITES (INDEX) NODE 0 NODE 1 NODE 2 Ia Id Ig Ij Im Ip Ib Ie Ih Ik In Iq Ic If Ii Il Io Ir P7 P4 P1 Ka Kd Kg Kj Km Kp P8 P5 P2 Kb Ke Kh Kk Kn Kq P9 P6 P3 Kc Kf Ki Kl Ko Kr 28
29. READS (QUERY) NODE 0 NODE 1 NODE 2 Ia Id Ig Ij Im Ip Ib Ie Ih Ik In Iq Ic If Ii Il Io Ir Q Q + SHARDS 29
30. HIGH AVAILABILITY Hochverfügbarkeit 30
31. UPTIME IS A POOR METRIC 31
32. “IF THE SYSTEM IS ‘DOWN’ AND NO ONE MAKES A REQUEST, IS IT REALLY DOWN?” ~ ME 32
33. HARVEST VS YIELD 33
34. YIELD QUERIES COMPLETED QUERIES OFFERED 34
35. HARVEST DATA AVAILABLE COMPLETE DATA 35
36. DURING FAILURE OR OVERLOAD - FOR A GIVEN QUERY - YOU MUST DECIDE BETWEEN HARVEST OR YIELD 36
37. MAINTAIN HARVEST VIA REPLICATION 37
38. REPLICATION • N VALUE - # OF REPLICAS TO STORE • DEFAULT OF 3 • MORE REPLICAS TRADES IOPS + SPACE FOR MORE HARVEST 38
39. P9 P8 P7 P6 P5 P4 P3 P2 P1 WRITES NODE 0 NODE 1 NODE 2 K I K 39
40. P9 P8 P7 P6 P5 P4 P3 P2 P1 REPLICATED WRITES NODE 0 NODE 1 NODE 2 K1 K2 K3 I1 I2 I3 K 40
41. QUERY + REPLICATION • NOT ALL NODES NEED TO BE QUERIED • FIND COVERING SUBSET OF PARTITIONS/NODES • YOKOZUNA BUILDS THE COVERAGE PLAN - SOLR EXECUTES THE DISTRIBUTED QUERY • NO USE OF SOLR CLOUD 41
42. SLOPPY QUORUM • N REPLICAS IMPLIES IDEA OF “PREFERENCE LIST” • SOME PARTITIONS ARE THE “PRIMARIES” - OTHERS ARE “SECONDARY” • SLOPPY = ALLOW NON-PRIMARY TO STORE REPLICAS • 100% YIELD - BUT POTENTIALLY DEGRADED HARVEST 42
43. TUNABLE QUORUM • R - # OF PARTITIONS TO VERIFYREAD • W - # OF PARTITIONS TO VERIFY WRITE • PR/PW - # OF PARTITIONS WHICH MUST BE PRIMARY • ALLOWS YOU TO TRADE YIELD FOR HARVEST - PER REQUEST 43
44. SIBLINGS • NO MASTER TO SERIALIZE OPS • CONCURRENT ACTORS ON SAME KEY • OPERATIONS CAN INTERLEAVE • USE VCLOCKS TO DETECT CONFLICT • CREATE SIBLINGS - LET CLIENT FIX • INDEX ALL SIBLINGS 44
45. SELF HEALING Selbstheilung 45
46. HINTED HANDOFF • WHEN NODES GO DOWN DATA WRITTEN TO SECONDARY PARTITIONS • WHEN NODES COMES BACK NEED TO GIVE THE DATA TO PRIMARY OWNER • AS DATA IS HANDED OFF INDEX IT ON DESTINATION NODE 46
47. P1 P2 P3 P4 P5 P6 P7 P8 ND3 ND1 ND2 ND0 ND3 ND2 ND0 ND1 K K1 K2 K3 WRITE HINTED HANDOFF 47
48. P1 P2 P3 P4 P5 P6 P7 P8 ND3 ND1 ND2 ND0 ND3 ND2 ND0 ND1 K K1 K2 K3 WRITE HINTED HANDOFF 48
49. P1 P2 P3 P4 P5 P6 P7 P8 ND3 ND1 ND2 ND0 ND3 ND2 ND0 ND1 K K1 K2 K3 WRITE HINTED HANDOFF 49
50. P1 P2 P3 P4 P5 P6 P7 P8 ND3 ND1 ND2 ND0 ND3 ND2 ND0 ND1 K K1 K2 K3 WRITE K2 HINTED HANDOFF 50
51. READ REPAIR • REPLICAS MAY NOT AGREE • REPLICAS MAY BE LOST • CHECK REPLICA VALUES DURING READ • FIX IF THEY DISAGREE • SEND NEW VALUE TO EACH REPLICA 51
52. ACTIVE ANTI- ENTROPY • 2 SYSTEMS (RIAK & SOLR) - GREATER CHANCE FOR INCONSISTENCY • FILES CAN BECOME TRUNCATED/ CORRUPTED • ACCIDENTAL RM -RF • SEGFAULT AT THE RIGHT TIME • ETC 52
53. MYRAID OF FAILURE SCENARIOS - FROM OBVIOUS TO NEARLY INVISIBLE 53
54. AAE - MERKLE TREES 54
55. AAE - MERKLE TREES EACH SEGMENT IS LIST OF KEY-HASH PAIRS 55
56. AAE - MERKLE TREES HASH OF HASHES IN SEGMENT 56
57. AAE - MERKLE TREES HASH OF HASHES OF HASHES OF HASHES :) 57
58. AAE - MERKLE TREES • IT’S A HASH TREE • IT’S ABOUT EFFICIENCY • BILLIONS OF OBJECTS CAN BE COMPARED AT COST OF COMPARING 2 HASHES (WIN!) 58
59. AAE - EXCHANGE 59
60. AAE - EXCHANGE TOP HASHES DON’T MATCH - SOMETHING IS DIFFERENT 60
61. AAE - EXCHANGE NARROW DOWN THE DIVERGENT SEGMENT 61
62. AAE - EXCHANGE NARROW DOWN THE DIVERGENT SEGMENT CONT... 62
63. AAE - EXCHANGE ITER FINAL LIST OF HASHES TO FIND DIVERGENT KEYS 63
64. AAE - EXCHANGE REPAIR (RE-INDEX) KEYS THAT ARE DIVERGENT (RED) 64
65. AAE • DURABLE TREES • UPDATED IN REAL TIME • NON-BLOCKING • PERIODICALLY EXCHANGED • INVOKE READ-REPAIR AND RE-INDEX ON DIVERGENCE • PERIODICALLY REBUILT 65
66. CODE FOR DETECTION AND REPAIR - NOT PREVENTION 66
67. DEMONSTRATION Vorführung 67
68. CREATE CLUSTER 68
69. START 5 NODES 69
70. JOIN NODES 70
71. CREATE PLAN 71
72. COMMIT PLAN 72
73. CHECK MEMBERSHIP 73
74. STORE SCHEMA 74
75. CREATE INDEX 75
76. INDEX SOME DATA • COMMIT LOG HISTORY OF VARIOUS BASHO REPOS • INDEX REPO NAME AND COMMIT AUTHOR, DATE, SUBJECT, BODY • USED BASHO BENCH TO LOAD DATA 76
77. QUERY 77
78. QUERY 78
79. QUERY • QUERY FROM ANY NODE • USE SOLR SYNTAX • RETURN SOLR RESULT VERBATIM • CAN USE EXISTING SOLR CLIENTS (FOR QUERY, NOT WRITE) 79
80. WHAT HAPPENS IF YOU TAKE 2 NODES DOWN? 80
81. DOWN 2 NODES 81
82. VERIFY DOWN 82
83. QUERY (DOWN) 83
84. QUERY (DOWN) • INDEX REPLICATION ALLOWS FOR QUERY AVAILABILITY • JUST NEED 1 REPLICA OF INDEX • IF TOO MANY NODES GO DOWN YOKOZUNA WILL REFUSE QUERY • PREFERS 100% HARVEST 84
85. WHAT HAPPENS IF YOU WRITE DATA WHILE NODES ARE DOWN? 85
86. VERIFY 0 RESULTS 86
87. ADD NEW DATA 87
88. QUERY NODE 1 88
89. DISABLE HANDOFF 89
90. START NODE 4 & 5 90
91. QUERY SOLR NODE 4 91
92. ENABLE HANDOFF 92
93. TAIL LOGS 93
94. QUERY SOLR NODE 4 94
95. QUERY NODE 4 95
96. WHAT HAPPENS IF YOU LOSE YOUR INDEX DATA? 96
97. QUERY SOLR NODE 4 NOTICE NUM FOUND IS 6747 97
98. RM -RF THE INDEX 98
99. KILL -9 JVM 99
100. YOKO RESTART JVM 100
101. QUERY SOLR NODE 4 NUM FOUND 0 BECAUSE INDEX WAS DELETED 101
102. AAE DETECT/REPAIR 102
103. QUERY SOLR NODE 4 NUM FOUND IS 6747 AGAIN THANKS TO AAE 103
104. Danke sehr! HTTP://GITHUB.COM/BASHO/YOKOZUNA 104