Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
You Have Control
Search
Andrew Godwin
June 04, 2018
Programming
0
620
You Have Control
My keynote from PyCon Israel 2018.
Andrew Godwin
June 04, 2018
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
320
Django Through The Years
andrewgodwin
0
210
Writing Maintainable Software At Scale
andrewgodwin
0
450
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
360
Async, Python, and the Future
andrewgodwin
2
680
How To Break Django: With Async
andrewgodwin
1
730
Taking Django's ORM Async
andrewgodwin
0
730
The Long Road To Asynchrony
andrewgodwin
0
660
The Scientist & The Engineer
andrewgodwin
1
780
Other Decks in Programming
See All in Programming
プロパティベーステストによるUIテスト: LLMによるプロパティ定義生成でエッジケースを捉える
tetta_pdnt
0
210
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
260
機能追加とリーダー業務の類似性
rinchoku
2
1.1k
AIと私たちの学習の変化を考える - Claude Codeの学習モードを例に
azukiazusa1
0
360
基礎から学ぶ大画面対応(Learning Large-Screen Support from the Ground Up)
tomoya0x00
0
270
Swift Updates - Learn Languages 2025
koher
2
460
フロントエンドのmonorepo化と責務分離のリアーキテクト
kajitack
2
160
TDD 実践ミニトーク
contour_gara
1
290
ProxyによるWindow間RPC機構の構築
syumai
3
1.1k
Microsoft Orleans, Daprのアクターモデルを使い効率的に開発、デプロイを行うためのSekibanの試行錯誤 / Sekiban: Exploring Efficient Development and Deployment with Microsoft Orleans and Dapr Actor Models
tomohisa
0
240
MCPとデザインシステムに立脚したデザインと実装の融合
yukukotani
4
1.4k
CJK and Unicode From a PHP Committer
youkidearitai
PRO
0
110
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Embracing the Ebb and Flow
colly
87
4.8k
Done Done
chrislema
185
16k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.6k
The Straight Up "How To Draw Better" Workshop
denniskardys
236
140k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
What's in a price? How to price your products and services
michaelherold
246
12k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Transcript
None
Hi, I’m Andrew Godwin • Django core developer • Senior
Software Engineer at • Private + Instrument pilot
Content Warning
Software is difficult.
By Derek Lowe "Things I won't work with"
On Hexanitrohexaazaisowurtzitane "...a more stable form of it, by mixing
it with TNT. Yes, this is an example of something that becomes less explosive as a one-to-one cocrystal with TNT."
On “Sand Won’t Save You This Time” "...the operator is
confronted with the problem of coping with a metal-fluorine fire. For dealing with this situation, I have always recommended a good pair of running shoes."
Unicode Locales Time Calendars Geography Money
Network latency Hardware unreliability Deadlocks Bit flips Ambiguous specifications No
documentation
We just move faster and hit them at higher speed.
Not unique to software
None
Who's solved this? Aviation.
A Boeing 747 has six million parts
…and a 0.000006% accident rate A Boeing 747 has six
million parts
Airplane Car Walking Train 220 130 30.8 Deaths per billion
hours (Per passenger, UK 1990-2000) 30
People matter as much as machines
Pilot 76% Aviation Accident Causes (2005 Nall report) 9% Other
16% Mechanical
And how we can apply them to software. Let's look
at some aviation principles
Principle #1 Hard Failure
If something is wrong it turns itself off Autopilots, engines,
air conditioning, and more
This only works if you have redundancy All of these
systems have a backup that lets you land.
"We'll ignore errors so the site doesn't crash!" "Save the
invalid data and we'll fix it later"
These are great ways to ensure you never fix something.
No accident or outage has a single cause. Stop your
code getting into odd states.
Fail hard if anything unexpected happens Validate all your data
strictly in and out Deploy changes early and often
Single points of failure can be good Only one place
to look when things go wrong!
None
Principle #2 Good Alerting
Cockpits are incredibly selective about what sets off an audio
alarm
Alert fatigue is real. Avoid at all costs.
Never, ever, put all errors in the same place
Critical Normal Background
Critical Normal Background Wakes someone up. Actionable.
Critical Normal Background Wakes someone up. Actionable. Fixed over the
next week.
Critical Normal Background Wakes someone up. Actionable. Fixed over the
next week. Metrics, not errors.
Have you been ignoring an error for weeks? Then turn
off its error reporting.
Principle #3 Find your limits
Everything will fail. You should know when.
Copyright Boeing
What's your Minimum Equipment List? What can you run the
system without?
Lavatory ashtrays Air conditioning Seatbelt signs Passenger video screens Fuel
caps Weather radar REQUIRED OPTIONAL
Did you load test? Did you fuzz test?
You don't have to perfectly scale. But you do have
to know where your limits are.
Risk is fine when you're informed! Unknowns are the most
dangerous thing.
Principle #4 Build for failure
No single thing in an aircraft can fail and take
it down.
We all want this for our code, but the way
to do it is to build for failure.
Kill your application randomly Practice server network failures Develop on
unreliable connections
The majority of pilot training is handling emergencies.
None
Use checklists. Don't rely on memory.
If you practice failure, you'll be ready when the inevitable
happens.
Pilot 76% Aviation Accident Causes (2005 Nall report) 9% Other
16% Mechanical
Principle #5 Communicate well
"You have control" "I have control" "You have control"
Complex software means separate teams.
As you grow, communication becomes exponentially harder.
None
None
None
Clear communication is vital.
Write everything down. Written specs = less time in meetings.
Have a clear chain of command.
Make decisions. They don't have to be perfect, just good
enough.
Principle #6 No blame culture
How do I know all these aviation stats?
Every incident is reported and investigated.
There is never a single cause of a problem.
Make it very difficult to do again. Why did your
software let this happen? What's the UX of your admin tools like?
None
None
Encourage reporting. Don't blame anyone for a mistake. They're unlikely
to make it again.
Reward maintenance as well as firefighting It's easy to look
good when you ship broken and are always heroically fixing it.
None
In aviation, every rule is written in blood.
Software is not yet there. But we are getting closer.
Margaret Hamilton Her error detection code saved Apollo 11
Patriot Missile Floating-point bug killed 28
Therac-25 Killed 3, severely injured at least 3 more
Uber Autonomous Vehicle Saw a pedestrian and chose to hit
her
None
Hard failure Good alerting Find your limits Build for failure
Communicate well No blame culture
Thanks. Andrew Godwin @andrewgodwin aeracode.org