Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Small Data: Storage For The Rest Of Us
Search
Andrew Godwin
May 26, 2015
Programming
680
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Small Data: Storage For The Rest Of Us
A talk I gave at PyWaw Summit 2015.
Andrew Godwin
May 26, 2015
Transcript
36 slides
Andrew Godwin @andrewgodwin SMALL DATA STORAGE FOR THE REST OF
US
Andrew Godwin Hi, I'm Django Core Developer Senior Engineer at
Far too many hobbies
BIG DATA What does it mean?
BIG DATA What does it mean? What is 'big'?
1,000 rows? 1,000,000 rows? 1,000,000,000 rows? 1,000,000,000,000 rows?
Scalable designs are a tradeoff: NOW LATER vs
Small company? Agency? Focus on ease of change, not scalability
You don't need to scale from day one But always
leave yourself scaling points
Rapid development Continuous deployment Hardware choice Scaling 'breakpoints'
Rapid development It's all about schema change overhead
Explicit Schema ID int Name text Weight uint 1 2
3 Alice Bob Charles 76 84 65 Implicit Schema { "id": 342, "name": "David", "weight": 44, }
Silent Failure { "id": 342, "name": "David", "weight": 74, }
{ "id": 342, "name": "Ellie", "weight": "85kg", } { "id": 342, "nom": "Frankie", "weight": 77, } { "id": 342, "name": "Frankie", "weight": -67, }
Continuous deployment It's 11pm. Do you know where your locks
are?
Add NULL and backfill 1-to-1 relation and backfill DBMS-supported type
changes
Hardware choice ZOMG RUN IT ON THE CLOUD
VMs are TERRIBLE at IO Up to 10x slowdown, even
with VT-d.
Memory is king Your database loves it. Don't let other
apps steal it.
Adding more power goes far Especially with PostgreSQL or read-only
replicas
Scaling Breakpoints
Sharding point Datasets paritioned by primary key
Vertical split Entirely unrelated tables
Denormalisation It's not free!
Consistency leeway Can you take inconsistent views?
Load Shapes
Read-heavy Write-heavy Large size
Read-heavy Write-heavy Large size Wikipedia TV show website Minecraft Forums
Amazon Glacier Eventbrite Logging
Read-heavy Write-heavy Large size Offline storage Append formats In-memory cache
/ flat files Many indexes Fewer indexes
Extremes
Extreme Reads Heavy Replication Extreme Writes Sacrifice ordering or consistency
Extreme Size Sacrifice query time
Extreme Longevity Flash in cold storage Extreme Survivability Rad-hardened Flash
Extreme Auditability True append only storage
SSDs Magnetic Tape Hard Drives Consumer Flash CDs/DVDs Long-life Flash
Metal-Carbon DVDs 3-6 months 5-10 years 3-5 years 100+ years Approximate time to bit flip, unpowered at room temperature
Big Data isn't one thing It depends on type, size,
complexity, throughput, latency...
Focus on the current problems Future problems don't matter if
you never get there
Efficiency and iterating fast matters The smaller you are, the
more time is worth
Good architecture affects product You're not writing a system in
a vacuum
Thanks. Andrew Godwin @andrewgodwin
More Decks by Andrew Godwin
See All by Andrew Godwin
Working With Risk: Lessons From Mountain Rescue
andrewgodwin
1
38
Reconciling Everything
andrewgodwin
1
430
Django Through The Years
andrewgodwin
0
360
Writing Maintainable Software At Scale
andrewgodwin
0
560
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
450
Async, Python, and the Future
andrewgodwin
2
780
How To Break Django: With Async
andrewgodwin
1
860
Taking Django's ORM Async
andrewgodwin
0
890
The Long Road To Asynchrony
andrewgodwin
0
800
Other Decks in Programming
See All in Programming
느린 네트워크와 저사양 디바이스 사용자를 위한 웹뷰 로딩 시간 개선기
kmkim
1
270
[Kotlin Fest 2026] eBPF Tracing for Kotlin
takehaya
1
400
AIが生成したコードは読むべきか?抽象化設計とコードの品質保証 / abstraction-design-and-quality
minodriven
17
9.5k
ローカルLLMの基本と近況 2611 / Local LLM basics and recent situation 2611
kishida
3
660
Math.sumPreciseから学ぶ、ECMAScript仕様の数値の種類
syumai
0
120
How SRE Grew Out of Server-Side Work and Became the Team's Own / サーバサイドから始まったSREを チームに渡すまで
maaaato
0
210
マルチプロダクトSaaSの アプリ一覧を速くする——複数プロダクトにまたがる「権限行脚」を減らすまで #techramen26conf
mktakuya
1
190
The Cost of the DOM
ypresto
2
220
Enforcing Your Architecture with Unit Tests in the AI Era (Swift Connection X)
steliosf
PRO
1
190
眠らない開発チームの作り方
biwakonbu
0
110
Materialが消えた日 〜真っ白なFlutterキャンバスにRenderObjectで絵を描く〜
umigishiaoi
0
170
Agentic AI with the Power of Jakarta EE
ivargrimstad
0
190
Featured
See All Featured
Facilitating Awesome Meetings
lara
57
7.2k
Building a A Zero-Code AI SEO Workflow
portentint
PRO
0
850
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
10
1.4k
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
aleyda
1
2.3k
Technical Leadership for Architectural Decision Making
baasie
4
660
Money Talks: Using Revenue to Get Sh*t Done
nikkihalliwell
0
570
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
tomoyanonymous
3
1.5k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
56
3.5k
A designer walks into a library…
pauljervisheath
211
25k
Documentation Writing (for coders)
carmenintech
77
5.7k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
konakalab
0
680
Impact Scores and Hybrid Strategies: The future of link building
tamaranovitovic
0
490