Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Thoughts About Normal and Abnormal Data (PyCon ...
Search
Markus H
October 27, 2017
Technology
0
13k
Thoughts About Normal and Abnormal Data (PyCon UK 2017)
Speaker notes at
https://markusholtermann.eu/2017/10/thoughts-about-normal-and-abnormal-data/
Markus H
October 27, 2017
Tweet
Share
More Decks by Markus H
See All by Markus H
🐍 ❤️ 🦀 — Python loves Rust
markush
0
230
Knock! Knock! Who's There?
markush
0
64
An Introduction To Kubernetes ☸
markush
0
91
Writing Safe Database Migrations (DjangoCon Europe 2021)
markush
0
14k
A Pony On The Move: How Migrations Work In Django 🐎
markush
0
13k
All Hands on Deck — Handling Security Issues
markush
0
14k
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon UK 2019)
markush
0
60
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)
markush
1
200
Logging Rethought 2: The Actions of Frank Taylor Jr. (DjangoCon Europe 2019)
markush
0
13k
Other Decks in Technology
See All in Technology
英語は話せません!それでも海外チームと信頼関係を作るため、対話を重ねた2ヶ月間のまなび
niioka_97
0
120
Green Tea Garbage Collector の今
zchee
PRO
2
390
10年の共創が示す、これからの開発者と企業の関係 ~ Crossroad
soracom
PRO
1
190
PLaMo2シリーズのvLLM実装 / PFN LLM セミナー
pfn
PRO
2
980
実装で解き明かす並行処理の歴史
zozotech
PRO
1
350
AIが書いたコードをAIが検証する!自律的なモバイルアプリ開発の実現
henteko
1
340
リーダーになったら未来を語れるようになろう/Speak the Future
sanogemaru
0
280
AI Agentと MCP Serverで実現する iOSアプリの 自動テスト作成の効率化
spiderplus_cb
0
500
BirdCLEF+2025 Noir 5位解法紹介
myso
0
200
Shirankedo NOCで見えてきたeduroam/OpenRoaming運用ノウハウと課題 - BAKUCHIKU BANBAN #2
marokiki
0
150
GopherCon Tour 概略
logica0419
2
190
stupid jj tricks
indirect
0
8k
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Why Our Code Smells
bkeepers
PRO
339
57k
The Straight Up "How To Draw Better" Workshop
denniskardys
237
140k
Visualization
eitanlees
148
16k
It's Worth the Effort
3n
187
28k
Large-scale JavaScript Application Architecture
addyosmani
514
110k
Speed Design
sergeychernyshev
32
1.1k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Side Projects
sachag
455
43k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
960
Measuring & Analyzing Core Web Vitals
bluesmoon
9
610
Transcript
Thoughts About Normal and Abnormal Data Markus Holtermann @m_holtermann markusholtermann.eu
@m_holtermann I am Markus Holtermann • Senior Software Engineer at
LaterPay • Django Core Developer
@m_holtermann How do we store our data?
@m_holtermann Files CC-BY-NC 2.0 by Tim Gee https://flic.kr/p/rZm63
@m_holtermann Document Stores CC-BY-SA 4.0 by Susan Gerbic https://commons.wikimedia.org/wiki/File%3AArchive_Room.JPG
@m_holtermann Copyright Geek Batman https://www.youtube.com/watch?v=gPDx_IwdYMY
@m_holtermann Name Home planet Gender Padmé Naboo Female Luke Tatooine
Male Leia Alderaan, Naboo Female
@m_holtermann First Normal Form (1NF)
@m_holtermann PersonID Name Home planet Gender 1 Padmé Naboo Female
2 Luke Tatooine Male 3 Leia Alderaan Female 3 Leia Naboo Female
@m_holtermann PersonID Name Home planet Gender 3 Leia Alderaan Female
3 Leia Naboo Male Update Anomalies
@m_holtermann Second Normal Form (2NF)
@m_holtermann PersonID Name Home planet Gender 1 Padmé Naboo Female
2 Luke Tatooine Male 3 Leia Alderaan Female 3 Leia Naboo Female
@m_holtermann PersonID Planet Name 1 Naboo 2 Tatooine 3 Alderaan
3 Naboo PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female
@m_holtermann PersonID Planet Name 1 Naboo 2 Tatooine 3 Alderaan
3 Naboo ??? Dagobah Insert Anomalies
@m_holtermann Deletion Anomalies PersonID Planet Name 1 Naboo 2 Tatooine
3 Alderaan 3 Naboo PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female
@m_holtermann Third Normal Form (3NF)
@m_holtermann PlanetID Name Water 10 Naboo 85% 11 Tatooine 1%
12 Alderaan 78% 13 Dagobah 88% PersonID Name Gender 1 Padmé Female 2 Luke Male 3 Leia Female PersonID PlanetID 1 10 2 11 3 10 3 12
@m_holtermann Database normalization is great!
@m_holtermann Always?
@m_holtermann Yet Another Wiki
@m_holtermann Page + PageID Name Slug Revision + RevisionID PageID
Text Date Database Schema
@m_holtermann Task 1: Fetch a single page and its current
revision
@m_holtermann Task 2: Fetch all page titles and the date
of their current revision
Task 1: Fetch a single page SELECT * FROM page
INNER JOIN revision ON page.page_id = revision.page_id WHERE page.slug = 'some-slug' ORDER BY revision.date DESC LIMIT 1;
Task 2: Fetch all pages SELECT page.name, last_revs.date FROM page
INNER JOIN ( SELECT revision.page_id, MAX(revision.date) date FROM revision GROUP BY revision.page_id ) last_revs ON page.page_id = last_revs.page_id;
@m_holtermann Benchmark Environment • Intel i7-6600U, 2.60GHz • 8 GB
Memory • PostgreSQL 9.6.5 • 10k pages, 6m revisions
@m_holtermann Task 1: Fetch a single page Concurrent queries 10
Pages per connection 1000 Queries per page 10 Queries total 100000
@m_holtermann Task 2: Fetch all pages Concurrent queries 1 Queries
per connection 10 Queries total 10
@m_holtermann Task 1: Fetch a single page
@m_holtermann Task 2: Fetch all pages
@m_holtermann Rae Knowler https://speakerdeck.com/bellisk/unsafe-at-any-speed-pycon-uk-26th-october-2017
@m_holtermann Database Schema Page + PageID Name Slug LastRevision Revision
+ RevisionID PageID Text Date
Task 1: Fetch a single page SELECT * FROM page
INNER JOIN revision ON page.last_revision_id = revision.revision_id WHERE page.slug = 'some-slug';
Task 2: Fetch all pages SELECT page.name, revision.date FROM page
INNER JOIN revision ON page.last_revision_id = revision.revision_id;
@m_holtermann Task 1: Fetch a single page
@m_holtermann Task 2: Fetch all pages
@m_holtermann Conclusion
Thanks Markus Holtermann @m_holtermann markusholtermann.eu