Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Keynote - Dylan Richard
Search
Monitorama
September 19, 2013
1
250
Berlin 2013 - Keynote - Dylan Richard
Monitorama
September 19, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
560
PDX 2017 - Pedro Andrade
monitorama
0
650
PDX 2017 - Roy Rapoport
monitorama
4
890
PDX 2017 - Julia Evans
monitorama
0
420
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
660
Berlin 2013 - Session - Alex Petrov
monitorama
6
640
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
570
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
490
Berlin 2013 - Session - David Goodlad
monitorama
0
380
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
3
370
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Building Adaptive Systems
keathley
38
2.2k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
92
16k
What's new in Ruby 2.0
geeforr
342
31k
A designer walks into a library…
pauljervisheath
202
24k
Designing Experiences People Love
moore
138
23k
4 Signs Your Business is Dying
shpigford
180
21k
Unsuck your backbone
ammeep
668
57k
Transcript
Monitoring the Obama Campaign
me: Director of Engineering, OFA typical political operative...
what I’m gonna talk about: •campaign context •how we monitored
•what I wish we had
Crate&Barrel
threadless
None
rebuild
None
None
while it’s in heavy use
18 months
7 days a week
18 hours a day
~40 engineers
7 teams
~300 repos
~200 deployed products
~3,000 servers
some million views a day
1,000,000+ volunteers
~8,000 staff
1 metric that matters
270
campaigns in 3 minutes
a campaign: •plan •build •execute •GOTV
PLAN!
dreams
not fast
for tech: start building
BUILD!
staff/teams
volunteer base
$
technology
for tech: keep building, make money
EXECUTE!
talk to voters
get volunteers
do ALL the things!
for tech: keep building, get feedback!
GOTV!
get out the vote
this is it
for tech:
scale.
scale!
SCALE!
None
None
also...
None
so.
so many apps
so many eyes
hiccup in our apps => press, legal, election?
keep the hiccups to a minimum
2 ways
more monitoring (to keep the hiccups small)
less things to fail
architectural simplicity
if you make things that look like this:
None
None
out of small, well understood pieces, you’re going to have
an easier time
make the easy things easy
monitoring what and how?
monitor ALL the things
we used some tools:
loggly
graylog
nagios
statsd
pingdom
cloudwatch
seyren
graphite
newrelic
google analytics
chartbeat
cloudability
pagerduty
...more
we were indiscriminate about the tools that we used to
monitor.
why?
[story time]
monitoring is only worth it if people are... monitoring.
alerts & dashboards!
alerting effectively is hard
my inbox was a terrible place
None
alerting needs appropriate severity
YELLING!
phone calls
calls from pagerduty are nice
calls from your boss at 2am are less nice, but
more powerful
<secret sauce>people</secret sauce>
your power users often know LONG before you do.
Craig
there are friendly people already monitoring your site
they find things that can be hard to monitor
they know how to make it your priority
they only monitor downstream
you need to know what that means upstream.
look at ALL your monitors when you get the downstream
alert
drill down into base level monitors.
recognize problems by their side affects!
make custom dashboards
dreams
1. monitor people
ಠ_ಠ [insert NSA joke]
my team was overworked, frustrated, unhealthy, tired, cranky, underfed, etc.
how can I monitor the health of my team like
we monitored the health of the systems?
aggregate fitbit dashboard?
simple feedback input while tests are running/things are building?
> how are you?
>
sentiment analysis of commit messages?
2. automate power users
what're they doing?
looking at the site and seeing things change.
it's not super sophisticated
what are people who are smarter than me doing?
Brett Slatkin (smarter than me)
dpxdt p-diffs for release testing
why not p-diffs for monitoring changes?
None
Summary:
monitor the things that matter any way you can
learn what downstream looks like
listen to your people, they know what's up
be careful about crying wolf with your alerts.
dreams:
why aren't we watching ourselves?
how do we enable that? [hint: I don't know.]
automate your power users
maybe p-diffs?
thanks!
Dylan Richard t: @dylanr e:
[email protected]