The UnNamed Talk

@highgroove tech talk - 04/06/12

Transcript

1. The Unnamed Talk @highgroove Tech Talk 04/06/2012
2. Alternate Titles • “Operations Eye for the Dev Person” • “Operations 101” • “Everything you want to know about Ops but were afraid to ask” • “Foreign Exchange for make benefit of glorious Nation of Operations” • “Care and Feeding of your Ops team” • “Ops is from Venus, Development is from Mars”
3. But first a word from our “sponsors” • Hosted or On-premise cloud management • Abstracts all the Things • Governance • You should ask me about it
4. Who the hell am I? • @lusis every-effing-where • Atlanta Native • DevOps-y kind of guy • Does Ruby (most of the time) • Does Python (other times) • Does Java (well, once like 2 days ago) • #monitoringsucks • Hate Driven Development
5. Let's talk about Ops
6. Ops Tradition
7. Ops Tradition You shall not pass! (lol)
8. @shitallspawsays “If you care at all about these things, you're doing Ops” • Metrics collection • PaaS/IaaS evaluation/investigation • Automation (auto-build, auto-recovery) • Fault tolerance • Availability • Monitoring • Performance • Capex and Opex forecasting • Outage response
9. @shitallspawsays “If you care at all about these things, you're doing Ops” • Metrics collection • PaaS/IaaS evaluation/investigation • Automation (auto-build, auto-recovery) • Fault tolerance • Availability • Monitoring • Performance • Capex and Opex forecasting • Outage response
10. What is “The Metric” “an analytical measurement intended to quantify the state of a system” - W. I. Pedia
11. Does this look familiar? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
12. Does this look familiar? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
13. So that's great for humans but • what if you want to do this?
14. Well you could.... • grep ms | awk | sed | grep | sort | cry • Performant? • Realistic? • Scalable? • Do you seriously want to do this?
15. What about Business Metrics? • Failed Logins • Shopping Cart Abandon Rates • New Account creations • <insert shit the business cares about here>
16. What about Business Metrics? • Failed Logins • Shopping Cart Abandon Rates • New Account creations • <insert shit the business cares about here> AND DID ANY OF IT GET BETTER OR WORSE WITH A NEW CODE RELEASE?
17. Let's talk about Ops
18. Baking operations into your code (easy) • Visibility – Implement a “ping” page • Low-impact • Exercise the full stack (call DB, perform some safe action, Easy to Parse, Easy to Check) • Please don't expect folks to robot a login session
19. Baking operations into your code (awesome) • Visibility – Push Metrics • StatsD + Graphite (DIY) • Librato, New Relic (aaS) • Read this - http://codeascraft.etsy.com/2011/02/15/measure-an • Watch This - http://pivotallabs.com/talks/139-metrics-metrics-e • Try This - https://github.com/johnewart/ruby-metrics
20. Let's talk about Logs
21. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
22. Logstash • Started by @jordansissel and @fetep ~2004 • Inputs → Filters → Outputs • Jruby (and increasingly better YARV support) • UNIX pipe with Radioactive Super Powers
23. Works like this
24. More like this
25. Can give you this
26. Config language Not Ruby. It'll be okay.
27. Basic Config
28. Slightly Advanced Config
29. Filters all up in your shit
30. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
31. Lookie here
32. Annoying to parse though • Try this: https://github.com/mattmatt/lograge
33. Remember This? Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)
34. Lookie here GET /jobs/833552.json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26 • yes that's a single line! • No more multiline • No more awkward parsing
35. Remember this?
36. Lookie here
37. Logs need to be readable by humans AND machines • Single lines are better • JSON is readable by humans and Machines • Every language has JSON support
38. Baking operations into your code • Take little effort • Has great value • Makes everyone happy • Don't forget the non-functional requirements (The “ilities”) • Stop inventing Timestamp formats (ISO8601 is a thing) • Turn Kittens into PopTarts
39. Ops AMA time! • Packaging? Artifacts? - FPM • Monitoring? - Nagios, Sensu, Riemann • Logging? - Loggly, Papertrail, Logstash (woot), Graylog2 • Alerting? - Pagerduty
40. Thank you so much!