Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Riak 2.0における全文検索 と整合性の問題
Search
UENISHI Kota
November 19, 2014
Technology
2
810
Riak 2.0における全文検索 と整合性の問題
WebDB Forum 2014 のBashoジャパンの発表スライドです
UENISHI Kota
November 19, 2014
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Storage Systems in Preferred Networks
kuenishi
0
22
Metadata Management in Distributed File Systems
kuenishi
2
500
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
390
Apache Ozone behind Simulation and AI Industries
kuenishi
0
360
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.2k
A Few Ways to Accelerate Deep Learning
kuenishi
0
1.1k
Introducing Retz
kuenishi
5
1.1k
Introducing Retz and how to develop practical frameworks
kuenishi
3
720
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.4k
Other Decks in Technology
See All in Technology
MCP Clientを活用するための設計と実装上の工夫
yudai00
1
810
Java で学ぶ 代数的データ型
ysknsid25
1
540
大手企業のAIツール導入の壁を越えて:サイバーエージェントのCursor活用戦略
gunta
19
5.4k
2025advance01
minamizaki
0
130
ITエンジニアを取り巻く環境とキャリアパス / A career path for Japanese IT engineers
takatama
4
1.5k
Swiftは最高だよの話
yuukiw00w
2
290
やさしいClaude Code入門
minorun365
PRO
32
25k
Digitization部 紹介資料
sansan33
PRO
1
3.8k
Javaアプリケーションの配布とパッケージング / Distribution and packaging of Java applications
hogelog
1
250
大規模PaaSにおける監視基盤の構築と効率化の道のり
lycorptech_jp
PRO
0
180
令和トラベルQAのAI活用
seigaitakahiro
0
520
データ戦略部門 紹介資料
sansan33
PRO
1
3.1k
Featured
See All Featured
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Practical Orchestrator
shlominoach
188
11k
The Language of Interfaces
destraynor
158
25k
Designing for Performance
lara
608
69k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
KATA
mclloyd
29
14k
Build The Right Thing And Hit Your Dates
maggiecrowley
35
2.7k
Into the Great Unknown - MozCon
thekraken
39
1.8k
Navigating Team Friction
lara
186
15k
Transcript
Riak 2.0ʹ͓͚Δશจݕࡧ ͱ߹ੑͷ WebDB Forum 2014 Basho Japan ্ 2014/11/19
ࣗݾհ • ࢄγεςϜྺ6 • Bashoδϟύϯͷํ͔Βདྷ·ͨ͠ • Riak CSͷ։ൃ • ͦͷଞຊͷ͜ͱ
BashoͱRiak •ࢄσʔλϕʔεʁ •RiakΛ͍ͬͯΔʁ •BashoΛ͍ͬͯΔʁ
We are hiring •࣮ੈքͷࢄγεςϜͷ ʹڵຯ͋Δਓʂ •@BashoJapan •
[email protected]
•APࢦͷσʔλϕʔεʢ݁Ռ߹ੑʣ •҆ఆੑɺ༧ଌՄೳੑ •ʮσʔλΛઈରʹͳ͘͞ͳ͍ʯ •৽͍͠ݕࡧΤϯδϯͱͯ͠SolrΛόϯυϧ
͜Μͳͱ͜ΖͰ ಈ͍͍ͯ·͢Riak •Rovio (Angry Birds) •Yahoo!JAPAN ͷΫϥυετϨʔδ •NHS (ΠΪϦε ࠃຽอݥαʔϏε)
•League of Legends (MMORPG) •ۜߦɺήʔϜɺখചɺηϯαʔɺetc…
CAPఆཧͱཧͷDB •ͲΜͳނোʹରͯ͠ (partition tolerance) •σʔλৗʹ߹͓ͯ͠Γ (consistency) •γεςϜ͕ࢭ·Δ͜ͱͳ͍ (availability) ͜ͷ3ͭΛಉ࣌ʹຬͨ͢γεςϜଘࡏ͠ͳ͍
CAP Theorem • C: ෳͷAtomic Objectʹର͢Δ࿈ ଓͨ͠ૢ࡞ (w1, w3, w4,
….) ͕શ ͯಉҰͰ͋Δ͜ͱ (linearizable) • A: Atomic Objectʹૢ࡞ w1, w2, …Λ࣮ߦͰ͖Δ͜ͱ • P: ωοτϫʔΫ͕ΕΔͱAtomic Objectͷϝοηʔδ͕౸ୡ͠ͳ͍
Is the network reliable?
None
None
How Riak Works (mostly about search)
run Riak with Search $ which java $ sudo apt-get
install riak $ echo “search = on” >> /etc/riak/riak.conf $ riak start $ riak-admin bucket-type create tt $ riak-admin bucket-type activate tt $ curl -XPUT http://localhost:8098/search/index/t $ curl -XPUT -H ‘content-type: application/json’ \ http://localhost:8098/types/tt/props \ -d ‘{“props”:{“search_index”:”t”}}’
Consistent Hashing • 160-bit Ωʔۭؒ • ۭؒΛ͢Δ • ύʔςΟγϣϯϊʔ υ͕ݸผཧ
• ϨϓϦΧNݸͷύʔ ςΟγϣϯʹίϐʔ͞ ΕΔ OPEF OPEF OPEF OPEF hash(“meetups/spamham”) N=3
σʔλϕʔεͷݕࡧ • DBࠐܕ • groongaͳͲ • “LIKE” • Pros •
߹ੑΛҡ͍࣋͢͠ • Cons • ີ݁߹ʹͳΔ • ֎෦࿈ܞܕ • Solr • Elasticsearch • Pros • ૄ݁߹ʹͰ͖Δ • Cons • ߹ੑΛҡ࣋͠ʹ͍͘
DBͱΠϯσοΫεͷ߹ੑ • DBͷσʔλຊମͱɺΠϯσοΫεΛ࣌ʹ҆શʹߋ ৽͢ΔͨΊʹτϥϯβΫγϣϯ͕ඞཁ • ີ݁߹ͷγεςϜͰ͋ΕϩοΫͳͲΛͬͯ߹ੑ Λอো͍͢͠
ૄ݁߹ʹ͢ΔϝϦοτ • ಉ͡ݕࡧΤϯδϯͰ͍ΖΜͳDBʹରԠͰ͖Δ • ӡ༻ϊϋπʔϧɺ։ൃΛूதͰ͖Δ • όʔδϣϯ߹ͷ੍͕؇͞ΕΔ • ยํ͕ނো͍ͯͯ͠͏ยํಈ࡞Ͱ͖Δ •
֦ு͘͢͠ͳΔ
྆ํͱ ͍͍ͱ͜ͲΓ͍ͨ͠
Yokozuna
as Riak Search 2.0 = + σʔλͷӬଓԽ Ϋϥελͷཧ γϯϓϧͳૢ࡞ RVFSZ
JOEFYJOH
Riak Search 2.0 • RiakʹSolrΛόϯυϧ • RiakͷKVΛߋ৽͢Δͱσʔλ͕ࣗಈతʹΠϯσο ΫεԽ͞ΕΔ • SolrͷϓϥάΠϯεΩʔϚɺΫΤϦΛͦͷ··
͏͜ͱ͕Ͱ͖Δ • ԽɺϊʔυՃআRiakͷ୲
ૄ݁߹ʁີ݁߹ʁ • ύοέʔδಉࠝ • ϓϩηεͱͯ͠ૄ݁߹ • ΠϯσοΫεͱσʔλؒ݁Ռ߹ੑΛอো • ΠϯσοΫεͷԽ݁Ռ߹ੑΛอো
Solrͱૄ݁߹ • Riak ϓϩηε͕ Solr ͷϓϩηεΛ fork ͢Δ • ཧɺࢹશͯRiak͕ߦ͏
• ผϓϩηεɺϝϞϦۭؒผ
ΠϯσοΫεԽͷྲྀΕ • Riakͷߋ৽ͱҰ৺ಉମʢdocument-based indexingʣ • ΠϯσοΫεԽʹࣦഊ͢ΔͱPUT͕ࣦഊ
ݕࡧΫΤϦͷॲཧ • Riakͷ2iMapReduceͱಉछͷΫΤϦ͕Δ • 3͓͖ͭͷ vnode ʹΞΫηε͢ΔͷͰɺॏෳͳ͍ ू
߹ੑͷ ͱҟৗܥ ߹ੑҡ࣋ͷͨΊʹߋ৽ϩάΛอ࣋ͯ͠ ϦΧόϦ͢Δͱ͍͏ํ๏ͰɺՄ༻ੑͱ அੑΛ୲อͰ͖ͳ͍ɻRiakͰଞͷ ํ๏Ͱσʔλͷ߹ੑΛҡ࣋͢Δɻ
Α͋͘Δҟৗܥ #1 •ϊʔυނোͰෳΛ3ͭ࡞Εͳ͍ •෮چͯ͠ɺ2ͭͷ··ʹͳͬͯ͠·͏
Α͋͘Δҟৗܥ #1 • ෮چͨ͋͠ͱʹRead Repair͢Δ • GETͨ͠ͱ͖ʹɺσʔλʹ͕ܽؕ͋ͬͨΒ RiakଆͰPUT͢͠ v2 v2
not found v2
Read Repair͕ ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ YES
ΫϥΠΞϯτ͔Β GET͕ݺΕͳ͍ͱ σʔλ͕Βͳ͍ʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
Active Anti Entropy • APࢦͷDBͷσʔλྼԽΛ͙ ͨΊͷόοΫάϥϯυॲཧ • Merkle-TreeΛͬͯύʔςΟγϣ ϯຖͷʮνΣοΫαϜʯΛܭࢉ •
ࠩΛݟ͚ͭͨΒͦ͜ΛRead Repair͢Δ hash(vnode=0, pid=0) hash(vnode=1, pid=0) hash(vnode=2, pid=0)
Α͋͘Δҟৗܥ #2 • Solr͕ΠϯσοΫεԽ͢Δલʹམͪͨ • Key, Valueอଘ͞Ε͕ͨΠϯσοΫεͳ͍
Read Repair͕ ݺΕΕΠϯσο Ϋε͕म෮͞ΕΔʁ NO
AAE (Active Anti Entropy) https://www.flickr.com/photos/51pct/7507525118/
YZ Active Anti Entropy • ݕࡧΠϯσοΫεʹ͍ͭͯ Merkle TreeΛ࡞ • ύʔςΟγϣϯຖʹKey-Valueͷ
Treeͱൺֱ͠ɺ͕ࠩ͋ΕΠϯ σοΫεΛमਖ਼ hash(vnode=0, pid=0, kv) hash(vnode=0, pid=0, yz)
Α͋͘Δҟৗܥ #2 •Searchઐ༻AAE͕ಈ࡞͍ͯ͠Δ •ΠϯσοΫεͱσʔλͷෆ߹Λݟ͚ͭͯ म෮͢Δ AAE
Α͋͘Δҟৗܥ #3 • ωοτϫʔΫ͕Ε͍ͯͨͱ͖ʹ྆ଆͰߋ৽͕ى ͖ͨʢSplit Brainʣ • Hinted HandoffʹΑΓͲͪΒॻ͖ࠐΈޭ
Α͋͘Δҟৗܥ #3 • ෮چ࣌ʹHandoffͰฦ͞Εɺ྆ํͷΛอ࣋ • GET·ͨAAEʹΑͬͯRead Repair͞ΕΔ
ҟৗܥ #1, #2, #3 •Ͳͷҟৗɺಛʹۓٸͷରॲඞཁͳ͍ •߹ੑݕࠪͱम෮͕ࣗಈͰߦΘΕΔ •ϧʔνϯϫʔΫͷӡ༻࡞ۀ͕΄ͱΜͲͳ͍
·ͱΊ • σʔλϕʔεͱશจݕࡧͷ • ߹ੑΛͱΔ͔ɺૄ݁߹ΛͱΔ͔ͷ • ৽͍͠Riak Search 2.0 •
ਫฏࢄͯ͠εέʔϧΞτͰ͖Δݕࡧ • ݁Ռ߹ੑΛબͿ͜ͱʹΑΓૄ݁߹ԽͱࢄԽ ʹޭʢ͔͠APܕʹʂʣ • ݁Ռ߹ੑΛબͨ͠ઃܭʹΑΓӡ༻͕؆୯ʹ
Questions?
FAQ •ΦϯϥΠϯͷεΩʔϚมߋʁ •μΠφϛοΫεΩʔϚʁ
outline • σʔλϕʔεʹೖ͍ͬͯΔσʔλΛશจݕࡧ͍ͨ͠→ࢁͷΠϯσοΫε͕Ͱ͖ΔɺFKeyͰ ΠϯσοΫε…ͱ͍͏͚ͩͰ • શจݕࡧΤϯδϯͷछྨ • ֎෦ܕʢ߹ੑΛҡ࣋͠ʹ͍͕͘Ԡ༻͕ޮ͘ɺރΕͨιϑτΣΞϓϥάΠϯ͕ଟ͍ʣ • ΈࠐΈܕʢ߹ੑΛҡ͍͕࣋͢͠ɺϝϯςφϯε͕͍͠ʣ
• Riakͦͷதؒɹόϯυϧͯ͠ΔͷͰָɺ͚ͩͲ࣮ମSolrɺ߹ੑΛࣗಈతʹҡ࣋͢Δ • όϯυϧͷํ๏ɺ߹ੑɺࢄͷํ๏ • ݕࡧΠϯσοΫε͚ͩΛࢄͤ͞Δͷ͍͠ • DBͱΠϯσοΫεͷ߹ɺϨϓϦΧͱϨϓϦΧͷ߹ɺJepsenͷͳ͠