Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
月間10億PVから学んだMongoDBアンチパターン
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
yujiosaka
November 11, 2014
Technology
5.7k
19
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
月間10億PVから学んだMongoDBアンチパターン
ZenClerkが月間10億PVを支えるまでの 過程で経験したアンチパターンを紹介
yujiosaka
November 11, 2014
Transcript
87 slides
݄ؒ10ԯPV͔ΒֶΜͩ MongoDBΞϯνύλʔϯ db tech showcase ౦ژ 2014 speaker: “Yuji Isobe”
history: [ “violinist” “engineer” “security specialist” “socket.io contributor” “startup member
of ZenClerk ” ] name: “Yuji Isobe”
questions: -> MongoDBΛ༻ͨ͜͠ͱ͕͋Γ·͔͢ʁ -> ຊ൪ڥͰMongoDBΛ༻͍ͯ͠·͔͢ʁ
goal: -> ZenClerk͕݄ؒ10ԯPVΛࢧ͑Δ·Ͱͷ աఔͰܦݧͨ͠ΞϯνύλʔϯΛհ -> ओʹύϑΥʔϚϯενϡʔχϯάͷ ->ʮ݄ؒ10ԯPVΛࢧ͑ΔMongoDBʯͷͦͷઌ IUUQXXXTMJEFTIBSFOFUZVKJPTBLBQWNPOHPEC
None
ϦΞϧλΠϜ
ϦΞϧλΠϜੳ Ϗοάσʔλ
db.ZenClerk .stats() -> ݄ؒ10ԯPV -> ඵؒ400PV -> ಉ࣌ଓ5ສ -> ݄ؒ10TBอଘ
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
architecture: front-end: [ “AngularJS” “Ruby on Rails” “MySQL” “memcached” “AWS”
] back-end: [ “Node.js” “MongoDB” “socket.io” “Redis” “AWS” ]
i2.2xlarge m1.large m3.large db.ZenClerk .aggregate() r3.large
ʙ1000ສPV mogngod
1000ສʙ1ԯPV delayed replica replica set
1ԯPVʙ10ԯPV replica set shard mongos replica set shard replica set
shard delayed replica delayed replica delayed replica mongoc
͜͜·ͰΞϓϦέʔγϣϯͷมߋ ΄ͱΜͲͳ͠ʢҰ෦ΫΤϦΛ࠷దԽʣ
we.use(“MongoDB”).explain() -> εΩʔϚϨε -> τϥϯβΫγϣϯෆཁ ʢ࠷ݶͷΞτϛοΫૢ࡞ʣ -> ॊೈͳΫΤϦ -> ༰қͳεέʔϧΞτ
-> Node.jsͱͷ૬ੑͷྑ͞
we.use(“MongoDB”).explain(true) -> ಋೖͷ༰қ͞ʢϗεςΟϯάαʔϏε༗ແʣ -> ֶशͷ༰қ͞ʢΤϯδχΞҎ֎ʹͱͬͯʣ
ࠓ͢ͷɾɾɾ ϕετϓϥΫςΟε Ξϯνύλʔϯ
ͪ͜Βʂ ϕετϓϥΫςΟε Ξϯνύλʔϯ
Ͱͦͷલʹɾɾɾ
ϕετϓϥΫςΟε
εΩʔϚσβΠϯ -> TwitterͷλΠϜϥΠϯΛอଘ͢Δ -> RDBMSͳΒਖ਼نԽ -> MongoDBͷϕετϓϥΫςΟεʁ
-> ಡΈࠐΈॏࢹ -> 1ΫΤϦ1ώοτ -> ίϝϯτʹԠͯ͡ ॻ͖ࠐΈ͕ॏ͘ͳΔ υΩϡϝϯτΛຒΊࠐΉ session: _id:
“session_id” device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”} ]
-> ॻ͖ࠐΈॏࢹ -> 2ΫΤϦN+1ώοτ -> ॻ͖ࠐΈεέʔϧ͢Δ υΩϡϝϯτΛຒΊࠐ·ͳ͍ session: _id: “session_id”
device: “iPhone” page_views: [ {url: “/items/1”} ] timeline: _id: “yujiosaka” tag: “#dbtechshowcase” ! tweet: _id: 1234 user_id: “yujiosaka” text: “MongoDB Rocks!”
timeline: _id: “yujiosaka-20141111” tag: “#dbtechshowcase” tweets: [ {text: “MongoDB Rocks!”}
] -> όϥϯεܕ -> 1ΫΤϦώοτ -> ॻ͖ࠐΈ1·Ͱ ͔͠ॏ͘ͳΒͳ͍ ϋΠϒϦου
εΩʔϚϨε ≠ εΩʔϚఆ͕ٛෆཁ RDBMSΑΓબࢶ͕ଟ͍͘͠ tips: IUUQXXXNPOHPECDPNQSFTFOUBUJPOTTDIFNBEFTJHOTDBMF
Ξϯνύλʔϯ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶃΠϯσοΫεͷషΓա͗
MongoDBϝϞϦͷ͍ํ͕ ͋·Γݡ͋͘Γ·ͤΜ
before: user = new mongoose.Schema first: {type: String, required: true,
index: true} last: {type: String, required: true, index: true} city: {type: String, required: true, index: true} ... created_at: {type: Date, default: Date.now, index: true} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
after: user = new mongoose.Schema first: {type: String, required: true}
last: {type: String, required: true} city: {type: String, required: true} ... created_at: {type: Date, default: Date.now} ! ॻ͖ࠐΈ͕٘ਜ਼ʹͳͬͯಡΈࠐΈ͕ૣ͚Ε͍͍͡ΌΜʂ
ಡΈࠐΈɺΠϯσοΫε͕ ϝϞϦʹΒͳ͚Ε͘ͳΔ
tips: όʔδϣϯ2.8ʹPluggable Storage Engine͕༧ఆ͞Ε͍ͯΔͷͰɺ ݡ͍Τϯδϯ͕બΔΑ͏ʹͳΔ https://blog.compose.io/the-coming-of-the-mongodb-storage-engines/
ᶄจࣈྻͷΠϯσοΫε
ΠϯσοΫεͷߏ -> RDBMSͱ΄ͱΜͲಉ͡ -> ΠϯσοΫεʹB-TreeΛ࠾༻ -> B-TreeৗʹฏߧΛอ͍ͬͯΔ
จࣈྻͷΠϯσοΫε -> ϗοτσʔλ͕ࢄ -> ϝϞϦޮ͕ѱ͍
࣌ؒɾObjectIdͷΠϯσοΫε -> ϗοτσʔλ͕ूத -> ϝϞϦޮ͕ྑ͍
tips: Ͱ͖Δ͚ͩObjectIdΛ͍·͠ΐ͏ ޙ͔Βͷஔ͖͑େมͰ͢
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ
Ϣʔβʔใͷ͏ͪ ࢢใ͚͕ͩཉ͍͠ user: {_id: “yujiosaka”, first: “Yuji”, last: “Isobe”, city:
“Tokyo”} index: {_id:1, city: 1} // compound index
ϑΟʔϧυࢦఆͳ͠ -> SQLͰͷʮselect *ʯ -> ෆཁͳωοτϫʔΫίετ -> ෆཁͳσΟεΫΞΫηε var user
= db.users.findOne({_id: “yujiosaka”});
ϑΟʔϧυࢦఆ͋Γ -> SQLͰͷʮselect cityʯ -> ωοτϫʔΫίετͷઅ -> covered index ͕׆༻Ͱ͖Δ
var user = db.users.findOne({_id: “yujiosaka”}, {city: 1});
tips: covered indexΛ༗ޮ׆༻͢Εɺ σΟεΫͷΞΫηεҰͳ͠ʹ ϝϞϦ͚͔ͩΒใΛಡΈࠐΊΔ
ᶆίʔϧυσʔλͷΫΤϦ
MongoDBͷಛ -> ϗοτσʔλͷΞΫηεߴ -> ίʔϧυσʔλͷΞΫηε͍ -> શσʔλͷΞΫηεۤख
ZenClerkʹ͓͚Δूܭ -> 5͝ͱʹσʔλΛཁ -> 1࣌ؒ͝ͱʹ౷ܭσʔλΛूܭ -> 1͝ͱʹ౷ܭσʔλΛूܭ
tips: ৗʹϗοτσʔλΛҙࣝͨ͠ ͍ํΛ͠·͠ΐ͏
ᶇϓϥΠϚϦʹภͬͨΫΤϦ
ϨϓϦΧηοτ -> ΫΤϦΛࢄͤ͞Δ͜ͱ͕Ͱ͖Δ -> ηΧϯμϦʹΫΤϦΛ͛Δ͔ ϓϩάϥϚ͕ܾΊΒΕΔ -> σϑΥϧτͰϓϥΠϚϦʹ͛ΒΕΔ
σϑΥϧτ -> ϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλ͕อো͞ΕΔ var user = db.users.find({city: “Tokyo”});
ηΧϯμϦΛࢦఆ -> ηΧϯμϦ͕͋ΕηΧϯμϦ͔Βɺ ͳ͚ΕϓϥΠϚϦ͔Βಡ·ΕΔ -> ࠷৽ͷσʔλอো͞Εͳ͍ var user = db.users.find({city:
“Tokyo”}) .readPref("secondaryPreferred")
tips: σʔλ͕࠷৽Ͱͳͯ͘Α͍߹ ಡΈࠐΈΛηΧϯμϦʹ͚Δ͜ͱͰ ΫΤϦ͕ࢄ͢Δ IUUQTTQFBLFSEFDLDPNNPOHPECQSBDUJDBM TDBMJOHBOETIBSEJOHFMJPUIPSPXJU[HFO
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶈඇޮͳΞοϓσʔτ
Α͘ݟ͔͚Δίʔυᶃ -> ඞͣΫΤϦ͕࣮ߦ͞ΕΔ -> ແବͳωοτϫʔΫίετ -> ඇΞτϛοΫͳॲཧ var user =
db.users.findOne({_id: “yujiosaka”}); user.count += 1; user.save();
Α͘ݟ͔͚Δίʔυᶃ -> updateͱ$incΛͬͯ·ͱΊΒΕΔ -> 1ճͷΞοϓσʔτͰࡁΉ -> ΞτϛοΫͳॲཧ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}});
Α͘ݟ͔͚Δίʔυᶄ -> σʔλ͕ݟ͔ͭͬͨΒߋ৽͠ɺݟ͔ͭΒͳ͔ͬͨΒ࡞͢Δ -> Ұݟෳࡶͳίʔυ var user = db.users.findOne({_id: “yujiosaka”});
if (user) { user.count += 1; user.save(); } else { db.users.insert({_id: “yujiosaka”, count: 1}); }
-> {upsert: true} Λ͏ͱ·ͱΊΒΕΔ -> ΑΓγϯϓϧͳίʔυ Α͘ݟ͔͚Δίʔυᶄ db.users.update({_id: “yujiosaka”}, {$inc:
{count: 1}}, {upsert: true});
ᶉංେԽ͢ΔυΩϡϝϯτ
Ϛεͷ࠲ඪσʔλΛ อଘ͢Δ behavior: points: [{x: 151, y: 100} {x: 151,
y: 179} {x: 151, y: 266} … {x: 151, y: 340}]
MongoDBͷۤख -> ϑΟʔϧυ͕ංେԽ͢ΔΞοϓσʔτ͕͍ -> σʔλ͕ҰఆͷαΠζʹऩ·Βͳ͘ͳΔͱ σΟεΫͰσʔλͷҠಈ͕ى͖Δ -> RedisͰMongoDBͷۤखΛิ͏
Redis -> ΦϯϝϞϦܕͷKVS -> ಡΈࠐΈɾॻ͖ࠐΈ͕ૣ͍ -> جຊతʹσʔλͷӬଓԽ͠ͳ͍ -> σʔλܕ͕ඇৗʹ๛ ʢϦετɾηοτɾϋογϡʣ
RedisΛΘͳ͍߹ db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 100}}); db.col.update({_id:
1234}, {$push: {points: {x: 151, y: 179}}); db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 266}}); … db.col.update({_id: 1234}, {$push: {points: {x: 151, y: 340}}); -> ࠲ඪ͕Ҡಈ͢ΔͨͼʹMongoDBʹॻ͖ࠐ·ΕΔ -> αΠζ͕ऩ·ΓΒͳ͍ͱɺσʔλͷҠಈ͕ى͖Δ
RedisΛͬͨ߹ db.col.insert({points: [ {x: 151, y: 100}, {x: 151, y:
179}, {x: 151, y: 266}, … {x: 151, y: 340} ]}); -> Ұ͔͠MongoDBʹॻ͖ࠐ·ͳ͍ -> υΩϡϝϯυͷαΠζ͜ΕҎ্େ͖͘ͳΒͳ͍
tips: ͯ͢ΛMongoDB͚ͩͰΖ͏ͱͤͣɺ ۤखͳ෦ଞͷٕज़Ͱิ͏͜ͱ͕େ
ΠϯσοΫεฤ ΫΤϦฤ Ξοϓσʔτฤ ΞʔΩςΫνϟฤ
ᶊ٧ΊࠐΈա͗DB
σʔλΛҰͭͷDBʹ٧ΊࠐΜͰ ྑ͍͜ͱʢࠓͷͱ͜Ζʣ͋·Γͳ͍
ͳͥDBΛ͚Δͷ͔ʁ -> Reader-WriterϩοΫ͕ΘΕ͍ͯΔ -> 2.2.0Ҏ߱DB୯ҐͰϩοΫ͢Δ -> ϚγϯΛ͚ΕϝϞϦ͕૿͑Δ
ZenClerk ʹ͓͚ΔDBͷ༻్ -> ϝΠϯσʔλʢHotʣ -> ϝΠϯσʔλʢColdʣ -> ੜσʔλ -> ܭࢉ݁Ռσʔλ
-> ౷ܭσʔλ -> αΠτଐੑσʔλ
tips: όʔδϣϯ2.8ʹdocument-level locking 3.0Ҏ߱ʹpartitioned joins͕༧ఆ͞Ε͍ͯΔ ʢޙऀ͋·Γ͋ͯʹ͠ͳ͍ํ͕ແͦ͏ʣ http://www.slideshare.net/tetsutarowatanabe/mongodb-world-2014-37081258
ᶃΠϯσοΫεͷషΓա͗ ᶄจࣈྻͷΠϯσοΫε ᶅϑΟʔϧυΛࢦఆ͠ͳ͍ΫΤϦ ᶆίʔϧυσʔλͷΫΤϦ ᶇϓϥΠϚϦʹภͬͨΫΤϦ ᶈඇޮͳΞοϓσʔτ ᶉංେԽ͢ΔυΩϡϝϯτ ᶊ٧ΊࠐΈա͗DB Ξϯνύλʔϯ͓͞Β͍
Ξϯνύλʔϯʹ ؕΒͳ͍ͨΊʹ
̏ͭͷࢹ -> ϦΞϧλΠϜࢹ -> ݟฦͨ͢Ίͷࢹ -> ͍ΫΤϦͷࢹ
ϦΞϧλΠϜࢹ
-> ComposeʢMongoHQʣμογϡϘʔυ -> ݱࡏͷ݈߁νΣοΫ༻ -> ࣾʹઐ༻σΟεϓϨΠΛઃஔ -> ҟৗ͕͋Εʮ୭͔Mongoࡴͨ͠ʁʯ ϦΞϧλΠϜࢹ
ݟฦͨ͢Ίͷࢹ
-> MMSμογϡϘʔυ -> ఆظతͳ݈߁அ -> োൃੜ࣌ʹݟฦͯ͠ ɹʮͳΜͰ͜ΜͳϩοΫߴ͘ͳͬͯΜͷʁʯ ݟฦͨ͢Ίͷࢹ
͍ΫΤϦͷࢹ
-> explain(true) -> system.profile.find() -> Dex -> Compose (MongoHQ) Slow
Queries ͍ΫΤϦͷࢹ
tips: ObjectIdΛͬͨΫΤϦʹ 100msҎ্͔͔ͬͨΒةݥ৴߸
my.presentation.aggregate() -> MongoDBબࢶ͕ଟ͍ -> Ξϯνύλʔϯ͍͔ͭ͘ଘࡏ͢Δ -> ΞϯνύλʔϯʹؕΒͳ͍ͨΊʹࢹ͢Δ -> ࠷৽ใͷΩϟονΞοϓॏཁ
͍͞͝ʹ
we.are “hiring! :)” https://www.zenclerk.com/recruit.html
any.questions?
More Decks by yujiosaka
See All by yujiosaka
I was understanding WASM all wrong! 🤯
yujiosaka
2
370
Machine Learning with JavaScript
yujiosaka
0
250
JavaScriptでも機械学習がやりたかった話
yujiosaka
2
510
ヘッドレスChromeでクローラを作った後の話
yujiosaka
3
770
俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな
yujiosaka
4
1.5k
『XXX』のための管理画面
yujiosaka
1
1.5k
Enjoy Deep Learning by JavaScript
yujiosaka
1
420
ひたすら楽してディープラーニング
yujiosaka
20
13k
technology x business
yujiosaka
3
630
Other Decks in Technology
See All in Technology
DevOps Agentにあらゆるセキュリティインシデントのトリアージを任せた件
jirtosterone
1
180
お風呂場をサーバルームにしよう
yusuke427
0
300
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
570
並列度を下げたら速くなった — go build の重複コンパイルを実測で見る
tomoaki25
2
1.3k
これからのデータエンジニアの役割について考える / Rethinking the Role of Data Engineers in the AI Era
okiyuki99
1
530
Context parameters で作る!はじめての型クラス
kworkdev
PRO
1
150
パスワードのない鍵で、 データを手元に置く
ginjihayashi
0
270
SSFの現在地 ― CAEP/RISCからAI Identityへ
oidfj
0
460
HTTP通信に使おうとすると実はめんどくさいfile_get_contents
maguroalternative
0
180
Claude Securityを使ってみた / claude-security-claude-code-meetup-nagoya
tnir
0
140
コピーオンライト型ファイルシステムと ランダム書き込み
sat
PRO
2
120
目を覚ませ!僕らの認証情報が何者かに侵略されてるぞ
tkito
2
440
Featured
See All Featured
brightonSEO & MeasureFest 2025 - Winning Strategies for Black Friday CRO & PPC - Christian Goodrich
cargoodrich
3
900
Leadership Guide Workshop - DevTernity 2021
reverentgeek
1
430
Unsuck your backbone
ammeep
672
58k
[RailsConf 2023] Rails as a piece of cake
palkan
59
7.1k
The agentic SEO stack - context over prompts
schlessera
0
1k
Agile Actions for Facilitating Distributed Teams - ADO2019
mkilby
0
340
Reflections from 52 weeks, 52 projects
jeffersonlam
356
22k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
31
10k
svc-hook: hooking system calls on ARM64 by binary rewriting
retrage
2
680
Six Lessons from altMBA
skipperchong
29
4.6k
[SF Ruby Conf 2025] Rails X
palkan
3
1.5k
What’s in a name? Adding method to the madness
productmarketing
PRO
24
4.2k