Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The OPS side of DEV

The OPS side of DEV

I think we as developers need to be more attentive towards the operational aspects of our code. Presented at Crash & Burn conference in Stockholm.

Approx 40 minutes.

Mårten Gustafson

March 02, 2012
Tweet

More Decks by Mårten Gustafson

Other Decks in Programming

Transcript

  1. * Huge files * Messy log format * Hard to

    filter * Hard to correlate * Might as well...
  2. framework pick a framework that’s: * makes sense * is

    de-facto standard? * is flexible * is lightweight * is easy to use
  3. consistent try to log in a consistent manner * think

    of log messages in terms of operation * what’s an error (should somebody be woken up?) * what’s a trace (who’s the audience?) * etc
  4. resources are finite * rotate your log files * put

    an upper bound on the size of one log file
  5. define window of interest * for the local disc: toss

    anythings that’s older than X, or: * compress (and then toss when they’re even older) * will you ever look in compressed log files?
  6. 0 [main] INFO Main - foo 0 [main] WARN Main

    - bar 0 [doer] ERROR Worker - gah java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13)
  7. 0 [main] INFO Main - foo 0 [main] WARN Main

    - bar 0 [doer] ERROR Worker - gah java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13) 0 [main] INFO Main - foo 0 [main] WARN Main - bar 0 [doer] ERROR Worker - gah java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13) 0 [main] INFO Main - foo 0 [main] WARN Main - bar 0 [doer] ERROR Worker - gah java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13) 0 [main] INFO Main - foo 0 [main] WARN Main - bar 0 [doer] ERROR Worker - gah java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13)
  8. INFO [2012-02-25 20:24:03] foo.Main - foo WARN [2012-02-25 20:24:03] foo.Main

    - bar ERROR [2012-02-25 20:24:03] foo.Worker - gah ! java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13)
  9. INFO [2012-02-25 20:24:03] foo.Main - foo WARN [2012-02-25 20:24:03] foo.Main

    - bar ERROR [2012-02-25 20:24:03] foo.Worker - gah ! java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13) INFO [2012-02-25 20:24:03] foo.Main - foo WARN [2012-02-25 20:24:03] foo.Main - bar ERROR [2012-02-25 20:24:03] foo.Worker - gah ! java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13) INFO [2012-02-25 20:24:03] foo.Main - foo WARN [2012-02-25 20:24:03] foo.Main - bar ERROR [2012-02-25 20:24:03] foo.Worker - gah ! java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13) INFO [2012-02-25 20:24:03] foo.Main - foo WARN [2012-02-25 20:24:03] foo.Main - bar ERROR [2012-02-25 20:24:03] foo.Worker - gah ! java.lang.NullPointerException: gah ! at Doer.worker(Doer.java:13)
  10. * when this is your reality, you don’t really only

    want log files on local machine disk
  11. * when this is your reality, you don’t really only

    want log files on local machine disk
  12. (beware of sensitive data) * security sensitive: keys, passwords, etc

    * integrity sensitive: whatever you’re users might provide that’s not for everyone’s eyes
  13. per environment * have a configuration that automatically adapts to

    the environment * log everything to stdout in local development * log everything to file in test * log X to Y and Z to FOO in prod
  14. re:configurable * don’t require a deploy to change a log

    level * provide an API * use JMX * so that you can tweak logging (enable tracing) right in production when you need to
  15. ...or whatever makes sense for you ...and yes, it’s Comic

    Sans for BAYEUX ...and yes, it’s Helvetica for JMX
  16. JMX JSON XML HTTP XMPP AMQP THRIFT BAYEUX RMI CSV

    ...or whatever makes sense for you ...and yes, it’s Comic Sans for BAYEUX ...and yes, it’s Helvetica for JMX
  17. * put all your values into your tools and services

    * BUT DON’T FORGET THE AD-HOC, LOCAL, USAGE (ie JMX)
  18. @Override protected Result check() throws Exception { if (database.ping()) {

    return Result.healthy(); } return Result.unhealthy("Can't ping database"); } * databases * other services * other dependencies * make them explicitly invokable
  19. or load externalized configuration * DNS * ZooKeeper * CouchDB

    * Doozer * External property/YML/JSON/whatever files ** in one sane specified location (preferably the working directory)
  20. aka

  21. 1. The network is reliable 2. Latency is zero 3.

    Bandwidth is infinite 4. The network is secure 5. Topology doesn't change 6. There is one administrator 7. Transport cost is zero 8. The network is homogeneous (- James Gosling) Fallacies of distributed computing - Peter Deutsch
  22. 1. The network is reliable 2. Latency is zero 3.

    Bandwidth is infinite 4. The network is secure 5. Topology doesn't change 6. There is one administrator 7. Transport cost is zero 8. The network is homogeneous (- James Gosling) Fallacies of distributed computing - Peter Deutsch
  23. 1. The network is reliable 2. Latency is zero 3.

    Bandwidth is infinite 4. The network is secure 5. Topology doesn't change 6. There is one administrator 7. Transport cost is zero 8. The network is homogeneous (- James Gosling) Fallacies of distributed computing - Peter Deutsch ...this * reliability (overall, geo location, connectivity) * security (communication, retention) * cost (of using, of not being available)
  24. YOU

  25. YOU

  26. YOU

  27. YOU

  28. YOU

  29. YOU

  30. YOU