Designing For Disaster - Preparing Your Code For Emergencies

A presentation by @stuherbert  for @GanbaroDigital Designing For Disaster Prepare
Your Code For Emergencies

Industry veteran: architect, engineer, leader, manager, mentor F/OSS contributor since
1994 Talking and writing about PHP since 2004 Chief Software Archaeologist Building Quality @GanbaroDigital About Stuart

Follow me I do tweet a lot about non-tech stuff
though :) @stuherbert

@GanbaroDigital This is a follow-on from my EdPUG talk from
June 2017.

@GanbaroDigital Ganbaro Digital on YouTube.com

@GanbaroDigital I’m here to talk to you about disasters in
Prod ...

@GanbaroDigital ... and how I prepare my code for them.

@GanbaroDigital “An emergency is the worst possible time to be
ﬁxing processes and practices.

@GanbaroDigital Three Desired Outcomes 1. Know That There Is A
Problem 2. Know What The Problem is 3. Make Handling Disasters Normal

@GanbaroDigital “ Guesswork is for amateurs.

@GanbaroDigital “ Assumptions are the mother of all screwups.

@GanbaroDigital “ Under pressure, people revert to habit.

@GanbaroDigital This is just my experience. I’m here to learn
from you too!

@GanbaroDigital Know That There Is A Problem

@GanbaroDigital Passive Monitoring

@GanbaroDigital ?? ?? How do you ﬁnd out if something’s
wrong in Prod?

@GanbaroDigital https://unsplash.com/photos/28v9cq7ytNU Everything On Fire

@GanbaroDigital ?? ?? Do you use your website / app
regularly enough to notice when things are down?

@GanbaroDigital https://ﬂic.kr/p/83Z8Kp Angry Customers!

@GanbaroDigital ?? ?? Will customers tell you there’s a problem
... ... or will they simply go elsewhere?

@GanbaroDigital ?? ?? Are you aware when customers do report
problems?

@GanbaroDigital Who Talks To Customers? • ... different part of
the building • ... different building • ... different timezone • ... different agenda

@GanbaroDigital “ To reliably know when there’s a problem we
need to cut out the middle man.

@GanbaroDigital https://unsplash.com/photos/LkD_IH8_K8k Monitoring

@GanbaroDigital You might be thinking ... “isn’t monitoring a devops*
responsibility?”

@GanbaroDigital * devops is not a job title!

@GanbaroDigital Log Monitoring

@GanbaroDigital ?? ?? Do you monitor your logs?

@GanbaroDigital I don’t (any more).

@GanbaroDigital Modern-day log messages are fundamentally unsuited for monitoring.

@GanbaroDigital “ As an industry, we have stopped treating log
messages as part of our interfaces.

@GanbaroDigital Log messages aren’t documented these days.

@GanbaroDigital Log messages, they come and they go from version
to version.

@GanbaroDigital “ Logs aren’t machine readable.

@GanbaroDigital Structured log messages are rare.

@GanbaroDigital Even structured logs aren’t additive facts. They’re statements.

@GanbaroDigital “ Logs aren’t reliable.

@GanbaroDigital ?? ?? If you stopped receiving log messages, would
your monitoring notice?

@GanbaroDigital During emergencies, logs get lost.

@GanbaroDigital Logs are transient. At scale, it’s not practical to
keep them for very long.

@GanbaroDigital ... and, at scale, we have to reduce logging
to avoid IO starvation.

@GanbaroDigital ?? ?? What can you monitor?

@GanbaroDigital “ Use metrics for monitoring.

@GanbaroDigital Metrics

@GanbaroDigital Metrics have: a hierarchical name, and a numerical value.

@GanbaroDigital Common Metrics • Counters - how many times has
an event happened? • Timers - how long is something taking to happen?

@GanbaroDigital Some metrics can come from monitoring log ﬁles.

@GanbaroDigital Web Server Metrics • Incoming requests • Response codes
(HTTP 500 etc) • Response timings • Response sizes

@GanbaroDigital Your web server doesn’t understand your app. Or your
business.

@GanbaroDigital Build your apps to emit metrics that mean something
to your business.

@GanbaroDigital composer require league/statsd

@GanbaroDigital https://www.slideshare.net/brianbrazil/an-introduction-to- prometheus-grafanacon-2016

@GanbaroDigital https://unsplash.com/photos/hOf9BaYUN88 Dashboards

@GanbaroDigital https://ﬂic.kr/p/7iMNDQ Warning Lights

@GanbaroDigital https://unsplash.com/photos/dLTpk6N31Fc Alarms!

@GanbaroDigital If we’re not monitoring app logs, what’s the point
of them?

@GanbaroDigital Know What The Problem Is

@GanbaroDigital Why We Log

@GanbaroDigital ?? ?? Do you rely on intuitive leaps to
solve problems in Prod?

@GanbaroDigital Intuition works well ... ... until intuition ﬁnds a
better paid job somewhere else!

@GanbaroDigital “ Intuition-driven production problem solving is not sustainable.

@GanbaroDigital Someone is going to be left holding the baby.
That someone might be future you.

@GanbaroDigital ?? ?? What can we replace intuition with?

@GanbaroDigital https://ﬂic.kr/p/cwK7UN Science!

@GanbaroDigital https://ﬂic.kr/p/9gh9EA Evidence!

@GanbaroDigital https://unsplash.com/photos/nYIQYg8cQVc Follow The Evidence!

@GanbaroDigital https://unsplash.com/photos/pe_R74hldW4 Clarity

@GanbaroDigital https://ﬂic.kr/p/9b24xc We Want To See The Internals

@GanbaroDigital “ Use logs for investigations.

@GanbaroDigital Consumable Logs

@GanbaroDigital Log collection and management is based on UNIX syslog.

@GanbaroDigital 1 line of text == 1 complete log message

@GanbaroDigital PHP Fatal error: Uncaught Exception: Something went wrong in
/tmp/ test.php:3 Stack trace: #0 {main} thrown in /tmp/test.php on line 3

@GanbaroDigital Use a custom formatter in Monolog.

@GanbaroDigital Convert line breaks and tabs into \n and \t
strings.

@GanbaroDigital

@GanbaroDigital Structured Logs

@GanbaroDigital

@GanbaroDigital datetime || log-level || message || data || IDs

@GanbaroDigital Datetime is complicated. Common practice is to log everything
in UTC to create consistent timelines.

@GanbaroDigital This has been standardised for decades. Don’t invent your
own.

@GanbaroDigital 1. emerg 2. alert 3. crit 4. err 5.
warning 6. notice 7. info 8. debug RFC 5424 Log Levels

@GanbaroDigital vnsprintf() format strings work really well for this.

@GanbaroDigital

@GanbaroDigital Make this the params array used to build the
‘message’ ﬁeld ... + any additional items you need.

@GanbaroDigital ?? ?? Can you identify the relevant log messages?

@GanbaroDigital Three Tracking Tokens 1. All log messages from a
single business process 2. All log messages from a single request 3. All log messages from a single user

@GanbaroDigital X-Request-ID GUID Header passed in by the API user

@GanbaroDigital (Request) UID Random hash Generated at the start of
each PHP request.

@GanbaroDigital

@GanbaroDigital Auth token User ID || OAuth Token You need
to know who did what.

@GanbaroDigital This structure works well with the search / ﬁlter
features in log management tools.

@GanbaroDigital Make Handling Disasters Normal

@GanbaroDigital Are we prepared?

@GanbaroDigital “ Under pressure, people revert to habit.

@GanbaroDigital Build those habits into your daily dev work.

@GanbaroDigital Debugging In Dev

@GanbaroDigital ?? ?? How many of you use Xdebug during
development?

@GanbaroDigital ?? ?? How many of you run Xdebug in
prod?

@GanbaroDigital You can’t run XDebug in prod. It’s just not
practical.

@GanbaroDigital ?? ?? What would you use when you can’t
use Xdebug?

@GanbaroDigital Use metrics and logs to debug in development.

@GanbaroDigital Effective logging is an iterative process. Start iterating in
dev.

@GanbaroDigital Logs complement your tests. They’re not a replacement.

@GanbaroDigital Shipping Under Pressure

@GanbaroDigital ?? ?? How quickly can you ship to prod?

@GanbaroDigital ?? ?? What can you safely bypass when you
have to ship in an emergency?

@GanbaroDigital ?? ?? Which day-to-day safeguards will stop you shipping
in an emergency?

@GanbaroDigital ?? ?? Do you have the ability to bypass
the things you need to?

@GanbaroDigital ?? ?? Can you force your emergency build to
top priority on your CI box?

@GanbaroDigital ?? ?? Can you build and ship from your
dev box if you had to?

@GanbaroDigital ?? ?? How would you tailor your testing?

@GanbaroDigital Just some food for thought.

@GanbaroDigital

Thank You Any Questions? A presentation by @stuherbert  for @GanbaroDigital

Designing For Disaster - Preparing Your Code Fo...

Designing For Disaster - Preparing Your Code For Emergencies

More Decks by Stuart Herbert

Other Decks in Programming

Featured

Transcript