Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Keynote - Dylan Richard
Search
Monitorama
September 19, 2013
1
250
Berlin 2013 - Keynote - Dylan Richard
Monitorama
September 19, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
550
PDX 2017 - Pedro Andrade
monitorama
0
630
PDX 2017 - Roy Rapoport
monitorama
4
870
PDX 2017 - Julia Evans
monitorama
0
410
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
650
Berlin 2013 - Session - Alex Petrov
monitorama
6
630
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
570
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
490
Berlin 2013 - Session - David Goodlad
monitorama
0
380
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Fontdeck: Realign not Redesign
paulrobertlloyd
80
5.1k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
Being A Developer After 40
akosma
84
590k
Fireside Chat
paigeccino
31
2.9k
No one is an island. Learnings from fostering a developers community.
thoeni
18
2.9k
4 Signs Your Business is Dying
shpigford
179
21k
Imperfection Machines: The Place of Print at Facebook
scottboms
263
13k
Building Flexible Design Systems
yeseniaperezcruz
325
38k
Web development in the modern age
philhawksworth
205
10k
Building Better People: How to give real-time feedback that sticks.
wjessup
359
19k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
123
18k
Transcript
Monitoring the Obama Campaign
me: Director of Engineering, OFA typical political operative...
what I’m gonna talk about: •campaign context •how we monitored
•what I wish we had
Crate&Barrel
threadless
None
rebuild
None
None
while it’s in heavy use
18 months
7 days a week
18 hours a day
~40 engineers
7 teams
~300 repos
~200 deployed products
~3,000 servers
some million views a day
1,000,000+ volunteers
~8,000 staff
1 metric that matters
270
campaigns in 3 minutes
a campaign: •plan •build •execute •GOTV
PLAN!
dreams
not fast
for tech: start building
BUILD!
staff/teams
volunteer base
$
technology
for tech: keep building, make money
EXECUTE!
talk to voters
get volunteers
do ALL the things!
for tech: keep building, get feedback!
GOTV!
get out the vote
this is it
for tech:
scale.
scale!
SCALE!
None
None
also...
None
so.
so many apps
so many eyes
hiccup in our apps => press, legal, election?
keep the hiccups to a minimum
2 ways
more monitoring (to keep the hiccups small)
less things to fail
architectural simplicity
if you make things that look like this:
None
None
out of small, well understood pieces, you’re going to have
an easier time
make the easy things easy
monitoring what and how?
monitor ALL the things
we used some tools:
loggly
graylog
nagios
statsd
pingdom
cloudwatch
seyren
graphite
newrelic
google analytics
chartbeat
cloudability
pagerduty
...more
we were indiscriminate about the tools that we used to
monitor.
why?
[story time]
monitoring is only worth it if people are... monitoring.
alerts & dashboards!
alerting effectively is hard
my inbox was a terrible place
None
alerting needs appropriate severity
YELLING!
phone calls
calls from pagerduty are nice
calls from your boss at 2am are less nice, but
more powerful
<secret sauce>people</secret sauce>
your power users often know LONG before you do.
Craig
there are friendly people already monitoring your site
they find things that can be hard to monitor
they know how to make it your priority
they only monitor downstream
you need to know what that means upstream.
look at ALL your monitors when you get the downstream
alert
drill down into base level monitors.
recognize problems by their side affects!
make custom dashboards
dreams
1. monitor people
ಠ_ಠ [insert NSA joke]
my team was overworked, frustrated, unhealthy, tired, cranky, underfed, etc.
how can I monitor the health of my team like
we monitored the health of the systems?
aggregate fitbit dashboard?
simple feedback input while tests are running/things are building?
> how are you?
>
sentiment analysis of commit messages?
2. automate power users
what're they doing?
looking at the site and seeing things change.
it's not super sophisticated
what are people who are smarter than me doing?
Brett Slatkin (smarter than me)
dpxdt p-diffs for release testing
why not p-diffs for monitoring changes?
None
Summary:
monitor the things that matter any way you can
learn what downstream looks like
listen to your people, they know what's up
be careful about crying wolf with your alerts.
dreams:
why aren't we watching ourselves?
how do we enable that? [hint: I don't know.]
automate your power users
maybe p-diffs?
thanks!
Dylan Richard t: @dylanr e:
[email protected]