Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
A New Concept of Consistency in Distributed Dat...
Search
UENISHI Kota
November 28, 2013
Technology
10
3.6k
A New Concept of Consistency in Distributed Database and Implementation in Riak
Web+DB forum 技術報告 by Basho
UENISHI Kota
November 28, 2013
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Metadata Management in Distributed File Systems
kuenishi
2
460
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
320
Apache Ozone behind Simulation and AI Industries
kuenishi
0
290
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.1k
A Few Ways to Accelerate Deep Learning
kuenishi
0
990
Introducing Retz
kuenishi
5
1k
Introducing Retz and how to develop practical frameworks
kuenishi
3
670
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.2k
Mesos Frameworkの作り方 (How to Make Mesos Framework)
kuenishi
7
2.3k
Other Decks in Technology
See All in Technology
来年もre:Invent2024 に行きたいあなたへ - “集中”と“つながり”で楽しむ -
ny7760
0
470
現地でMeet Upをやる場合の注意点〜反省点を添えて〜
shotashiratori
0
520
IaC運用を楽にするためにCDK Pipelinesを導入したけど、思い通りにいかなかった話
smt7174
1
110
グローバル展開を見据えたサービスにおける機械翻訳プラクティス / dp-ai-translating
cyberagentdevelopers
PRO
1
150
ネット広告に未来はあるか?「3rd Party Cookie廃止とPrivacy Sandboxの効果検証の裏側」 / third-party-cookie-privacy
cyberagentdevelopers
PRO
1
130
生成AIの強みと弱みを理解して、生成AIがもたらすパワーをプロダクトの価値へ繋げるために実践したこと / advance-ai-generating
cyberagentdevelopers
PRO
1
180
新卒1年目が向き合う生成AI事業の開発を加速させる技術選定 / ai-web-launcher
cyberagentdevelopers
PRO
7
1.5k
カメラを用いた店内計測におけるオプトインの仕組みの実現 / ai-optin-camera
cyberagentdevelopers
PRO
1
120
【若手エンジニア応援LT会】AWSで繋がり、共に成長! ~コミュニティ活動と新人教育への挑戦~
kazushi_ohata
0
180
omakaseしないための.rubocop.yml のつくりかた / How to Build Your .rubocop.yml to Avoid Omakase #kaigionrails
linkers_tech
3
740
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
49k
MAMを軸とした動画ハンドリングにおけるAI活用前提の整備と次世代ビジョン / abema-ai-mam
cyberagentdevelopers
PRO
1
110
Featured
See All Featured
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
Fontdeck: Realign not Redesign
paulrobertlloyd
81
5.2k
Building a Scalable Design System with Sketch
lauravandoore
459
33k
Statistics for Hackers
jakevdp
796
220k
Designing Experiences People Love
moore
138
23k
A better future with KSS
kneath
238
17k
Testing 201, or: Great Expectations
jmmastey
38
7k
Designing for humans not robots
tammielis
249
25k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
355
29k
Building Better People: How to give real-time feedback that sticks.
wjessup
363
19k
YesSQL, Process and Tooling at Scale
rocio
167
14k
Transcript
ࢄσʔλϕʔεʹ͓͚Δ ৽͍͠߹ੑϞσϧͱ Riakʹ͓͚Δ࣮ 2013 / 11 / 28 WebDB Forum
Basho ্߁ଠ
ࢄσʔλϕʔεʹ͓͚Δ ݹͯ͘৽͍͠߹ੑϞσϧͱ Riakʹ͓͚Δ࣮ 2013 / 11 / 28 WebDB Forum
Basho ্߁ଠ
BashoͱRiak •ࢄσʔλϕʔεʁ •RiakΛ͍ͬͯΔʁ •BashoΛ͍ͬͯΔʁ
CAPఆཧͱཧͷDB •ͲΜͳނোʹରͯ͠ (partition tolerance) •σʔλৗʹ߹͓ͯ͠Γ (consistency) •γεςϜ͕ࢭ·Δ͜ͱͳ͍ (availability) ͜ͷ3ͭΛಉ࣌ʹຬͨ͢γεςϜଘࡏ͠ͳ͍
•Մ༻ੑ (Availability) ͕ಛͷσʔλ ϕʔε •ӡ༻͍͢͠ɺେ͖ͳσʔλͰೖΔ •҆ఆੑɺ༧ଌՄೳੑ •ʮσʔλΛઈରʹͳ͘͞ͳ͍ʯ
͜Μͳͱ͜ΖͰ ಈ͍͍ͯ·͢Riak •Rovio (Angry Birds) •Yahoo!JAPAN ͷΫϥυετϨʔδ •NHS (ΠΪϦε ࠃຽอݥαʔϏε)
•Bump (=>Google) •ۜߦɺήʔϜɺখചɺηϯαʔɺetc…
How Riak Works
Consistent Hashing • 160-bit Ωʔۭؒ • ۭؒΛ͢Δ • ύʔςΟγϣϯϊʔ υ͕ݸผཧ
• ϨϓϦΧNݸͷύʔ ςΟγϣϯʹίϐʔ͞ ΕΔ OPEF OPEF OPEF OPEF hash(“meetups/spamham”) N=3
Consistency͍͠ •ߋ৽ΛࢭΊΔʢAvailabilityΛԼ͛Δʣ͔ɺߋ৽ͷ্ॻ͖Λ ڐ͢ʢσʔλΛࣦ͏ʣ͔͔͠બࢶ͕ͳ͍ Server2 Server1 Server3 PUT V=42 PUT V=0
V=?
ConsistencyͷΘΓʹ •ͱΓ͋͑ͣෳͷόʔδϣϯͷڞଘΛڐ͢ •Ͳͷόʔδϣϯ͕ਖ਼͍͔͠ɺ͘͠Ϛʔδ͢Δ͔ΛRead࣌ʹܾఆ Server2 Server1 Server3 PUT V=42 PUT V=0
V=0 or 42 V=0 V=0 or 42 V=42
APΛ࣮ݱ •ωοτϫʔΫஅ͕ى͖͍ͯͯͱΓ͋͑ͣॻ͖ࠐΈΛڐ͢ Server2 Server1 Server3 PUT V=42 PUT V=0 Server4
෮چͨ͠Βॻ͖͢ ྆ํ͓࣋ͬͯ͘
γϣοϐϯάΧʔτͷྫ •UnionΛͱΕΑ͍ Server2 Server1 Server3 PUT cart=[a,b,d] PUT cart=[a,b,c] union([a,b,c],
[a,b,d]) => [a,b,c,d] [a,b,c] [a,b,c] or [a,b,d] [a,b,d]
ෳόʔδϣϯΛ ڐ͢͜ͱͷ •ϓϩάϥϛϯά͕͍͠ʢτϥϯβΫγϣϯૉ Β͍͠ʣ •ݱ࣮ੈքγϣοϐϯάΧʔτͱΧϯλʔ͚ͩ Ͱͳ͍ •҆શͳMerge, update͕Ͱ͖ΔσʔλߏΛຖճ ߟ͑ͳ͚ΕͳΒͳ͍ •͍ͬͯΔ͏ͪʹࣅͨΑ͏ͳϥΠϒϥϦ͕͋ͪ͜
ͪͰग़དྷ্͕Δ
ͳ͍ͥ͠ͷ͔ʁ •σʔλͷWriteͱWrite͕ೖΕସΘΓ͏ ΔʹSerializableͲ͜Ζ͔WriteҰ؏ ͨ͠ঢ়ଶʹͰ͖ͳ͍ Server2 Server1 Server3 w1 w2 w1
w2 w2 (w1 lost)
Logical Monoticity •σʔλʹର͢ΔՄͳૢ࡞ͷΈΛڐ͢ʂ Data = update(w2, update(w1, Data0)) = update(w1,
update(w2, Data0)) Data = merge(update(w2, Data0), Data)
͑: CRDT •ʮෳՄೳͳՄσʔλܕʯ •Conflict-Free Replicated Data Types •Commutative Replicated Data
Types •… •(Going to be included in Riak 2.0) ) CRDTͷ࡞ऀLogical Monotinicy ͱ͍͏ݴ༿͍ͬͯͳ͍
CRDT in Riak 2.0 •KVSͷVʹʮܕʯΛ࣋ͨͤͯɺܕʹΑͬͯ UpdateͱMergeͷϩδοΫΛܾΊΔ •Read࣌ʹMerge͕αʔόʔଆͰࣗಈతʹ࣮ ߦ͞ΕΔ •ΞϓϦέʔγϣϯܕΛࢦఆ͢Δ͚ͩͰΑ͘ɺ ෳόʔδϣϯͷϋϯυϦϯά͕ෆཁʹͳΔ
CRDT example •PN-Counter •Set •OR-sets •LWW-register •Graph…
PN-Counter •σϞ
PN-Counter • merge • {a: {1,-1}, b: {1,0}, c: {2,0}}
• {a: {0,0}, b: {2, 0}, c: {0, -2}} • => {a: {1,-1}, b:{2,0}, c:{2,-2}} => 2 • update • a͕ {increment, 3} Λड͚͚Δͱ • {a: {4,-1}, b: {1,0}, c: {2,0}}
OR-Sets • merge • {a:{“foo”:true}, b:{“bar”:false}} • + {a:{“foo”:true}, b:{“foo”:false,
“bar”:false}} • => {a:{“foo”:true}, b:{“foo”:false, “bar”:true}} • => [“bar”] • update • add: {a:{}} => +”foo” => {a:{“foo”:false}} • remove: {a: {“foo”:false}} => {a: {“foo”:true}}
OR-Sets •σϞ
Ϣʔεέʔε •ΫϦοΫͷΧϯτ (G-counter) • riak-server/types/counters/buckets/likes/datatypes/basho.com -d 1 •γϣοϐϯάΧʔτ (OR-sets) •ϩάΠϯϢʔβʔ
(PN-counter) •͜ΕΒͷΈ߹Θͤ (map & LWW-register, boolean) •{ name : “basho.com”, likes: 20000, users: 3000, links: [ “basho.co.jp”, “basho.co.uk” ], cool: true }
Ͱ͖ͳ͍͜ͱ •ʮ0Ҏ্ʯͷPN-counter •ϢχʔΫͳIDൃߦ •ͦͷଞCAS͕ඞཁͳσʔλߏͱૢ࡞
·ͱΊ •RiakՄ༻ੑͷ͋Δࢄσʔλϕʔε •ෳͷόʔδϣϯΛಉ࣌ʹอ࣋͢ΔͷΛ ڐ͢͜ͱͰՄ༻ੑΛ୲อ •ΞϓϦ։ൃͷқ͕՝ •CRDTͱ͍͏ܕͷಋೖʹΑΓ؆୯͔ͭ σʔλͷͳ͘ͳΒͳ͍ΈΛ࡞ͬͨ
Questions? •Riak 2.0 Λָ͠Έʹ͍ͯͩ͘͠͞ •Web: http://basho.co.jp •Twitter: @BashoJapan •Me:
[email protected]
•ML:
[email protected]
Useful links http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf http://arxiv.org/pdf/1210.3368.pdf https://gist.github.com/russelldb/f92f44bdfb619e089a4d http://gsd.di.uminho.pt/members/cbm/ps/scadt3.pdf http://arxiv.org/abs/1011.5808