Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Database scaling patterns
Search
Mariusz Gil
April 12, 2013
Programming
2
510
Database scaling patterns
Mariusz Gil
April 12, 2013
Tweet
Share
More Decks by Mariusz Gil
See All by Mariusz Gil
Aspect Oriented Programming
mariuszgil
1
280
Designing and implementing GraphQL API
mariuszgil
1
39
Discovering unknown with EventStorming ConFoo
mariuszgil
0
250
Game of Developer Life... Deconstructed
mariuszgil
1
150
Back to forgotten roots
mariuszgil
1
350
Go micro with microservices
mariuszgil
5
530
Machine Learning for the rescue
mariuszgil
0
340
Discovering graph structures
mariuszgil
3
520
Introduction to Aerospike with PHP
mariuszgil
8
720
Other Decks in Programming
See All in Programming
Jakarta EE meets AI
ivargrimstad
1
520
Google Sign-inの移行から始めるCredential Manager活用
clockvoid
0
310
Scala におけるコンパイラエラーとの付き合い方
chencmd
2
430
connect-go で面倒くささと戦う / 2024-08-27 #newmo_layerx_go
izumin5210
2
650
Pythonで改めて考える「クラス(class)」の使いどころ
os1ma
1
510
Increased Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
380
状態管理ライブラリZustandの導入から運用まで
k1tikurisu
3
470
GenU導入でCDKに初挑戦し、悪戦苦闘した話
hideg
0
160
RAGの回答精度評価用のQAデータセットを生成AIに作らせた話
kurahara
0
250
Understand the mechanism! Let's do screenshots tests of Compose Previews with various variations / 仕組みから理解する!Composeプレビューを様々なバリエーションでスクリーンショットテストしよう
sumio
3
790
Android開発以外のAndroid開発経験の活かしどころ
konifar
2
1k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
0
120
Featured
See All Featured
Large-scale JavaScript Application Architecture
addyosmani
508
110k
Designing Experiences People Love
moore
138
23k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
29
2.6k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.2k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
Gamification - CAS2011
davidbonilla
79
5k
Thoughts on Productivity
jonyablonski
66
4.2k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
123
18k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
32k
The Brand Is Dead. Long Live the Brand.
mthomps
53
38k
VelocityConf: Rendering Performance Case Studies
addyosmani
322
23k
Transcript
Database scaling patterns Mariusz Gil the we a way
ABOUT ME
EXPERIENCE from 1 to 100+ DB servers
EXPERIENCE from 1 to 100+ DB servers
What i it a ABOUT ?
DATABASE IS MAIN PROBLEM IN 99% OF WEB APPS
TYPICAL ISSUES? NORMALIZED TO xNF INEFFICIENT SCHEMA, BAD INDEXES AND
QUERIES, UNUSED SLOW-LOG
shared setup
!" # $ %% & ' !( !" # !" # shared server
cache all the things, wisely...
!" # $ %% & ' !( !" # !" # shared server
separated servers
!" # $ %% & ' !( !" # !" # DB server
scale up
scale up
denormalization
DB server
!" # $ %% & ' !( !" # !" # !" # $ %% & ' !( !" # !" # DB server
DB server
!" # $ %% & ' !( !" # !" # DB server !" # $ %% & ' !( !" # ) !" # )
DB server DB server
!" # $ %% & ' !( !" # ) !" # ) # $ %% & ' !( !" # !" # ID SENDER_ID RECEIVER_ID ... 1 10 1 2 20 2 3 10 1 4 20 3 5 10 1 6 20 4 7 10 1 USER_ID MESSAGE_ID 1 1 1 3 1 5 1 7 2 2 3 4 4 6 USER_ID MESSAGE_ID 10 1 10 3 10 5 10 7 20 2 20 4 20 6
replication
MASTER-SLAVE scale reads, but not writes
DB master
!" # $ %% & ' !( !" # ) !" # ) DB slave !" # $ %% & ' !( !" # ) !" # ) DB slave !" # $ %% & ' !( !" # ) !" # ) ... INSERT... UPDATE... DELETE... SELECT... SELECT...
MASTER-MASTER just for HA, not for scaling
DB master
!" # $ %% & ' !( !" # ) !" # ) DB master !" # $ %% & ' !( !" # ) !" # ) INSERT... UPDATE... DELETE... SELECT...
sharding
DB shard $ %% &
' !( !" # ) !" # ) DB shard !" # $ %% & ' !( !" # ) !" # ) DB shard !" # $ %% & ' !( !" # ) !" # ) INSERT... UPDATE... DELETE... SELECT... INSERT... UPDATE... DELETE... SELECT... INSERT... UPDATE... DELETE... SELECT...
Sharding i HARD !
Resharding i HARDER !
JOINS FORGET ABOUT IT data could be splitted into many
shards
ALTERS THINK TWICE ABOUT IT altering many shards could be
a nice challenge
fault tolerant multi-master sharding
SCALE WHEN NECESSARY NOT FOR FUN
Alternative TOOLS ?
None
None
None
None
None
Thanks! Any questions? @mariuszgil
[email protected]