Metrics Driven Development

A talk given first at Nordic Ruby, then polished and refactored a bit for Scottish Ruby Conf. It talks about the process we use @paperlessdev / @paperlesspost for driving our development of features and systems with metrics and 'moving the needle'

Transcript

1. Metrics drIVEN DEVELOPMENT Aaron Quint / Scottish Ruby Conf / June 2012 Friday, June 29, 12
2. In a data-less world Friday, June 29, 12
3. It’s me, @AQ HeYO Friday, June 29, 12
4. Friday, June 29, 12
5. • Food Obsessed (CodeFoodBar) Friday, June 29, 12
6. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) Friday, June 29, 12
7. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) Friday, June 29, 12
8. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here Friday, June 29, 12
9. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here • twitter: @aq Friday, June 29, 12
10. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • Bicoastal (Brooklyn, NY <-> Berkeley, CA) • Very excited to be here • twitter: @aq • github: quirkey Friday, June 29, 12
11. CTO of Friday, June 29, 12
12. Chief Taco Officer CTO of Friday, June 29, 12
13. www.paperlesspost.com Friday, June 29, 12
14. Friday, June 29, 12
15. A lot of these ideas and implementations are theirs. Amazing team. Friday, June 29, 12
16. METRICS METRICS METRICS Friday, June 29, 12
17. If you’re not improving your product, what are you doing? System metrics == PRoduct Metrics Friday, June 29, 12
18. The spiel Friday, June 29, 12
19. Measure Experiment Pivot Metrics Friday, June 29, 12
20. Measure Experiment Pivot Metrics #winning Friday, June 29, 12
21. WTF is going on MEASURE Friday, June 29, 12
22. Make changes, users are like WTF Experiment Friday, June 29, 12
23. Take the best result and change WTF is going on Pivot Friday, June 29, 12
24. The data you collect in _every_ step Metrics Friday, June 29, 12
25. The Way Friday, June 29, 12
26. Moving the needle Friday, June 29, 12
27. Continuous Delivery Friday, June 29, 12
28. Continuous Process Delivery Friday, June 29, 12
29. Continuous Process DeliveryTM Friday, June 29, 12
30. Moving the needle Friday, June 29, 12
31. Moving the needle 1. Measure Everything Friday, June 29, 12
32. Moving the needle 1. Measure Everything 2. Set Goal metrics attached to each feature/deploy Friday, June 29, 12
33. Moving the needle 1. Measure Everything 2. Set Goal metrics attached to each feature/deploy 3. Build & Roll out in chunks Friday, June 29, 12
34. Moving the needle 1. Measure Everything 2. Set Goal metrics attached to each feature/deploy 3. Build & Roll out in chunks 4. Regroup and analyze Friday, June 29, 12
35. Seriously, everything 1. Measure everything Friday, June 29, 12
36. What happened vs What’s happening Absolute vs Relative Accounting vs Trending Friday, June 29, 12
37. Key Performance Indicators Accounting = KPI’s Friday, June 29, 12
38. It’s in your database the data is there Friday, June 29, 12
39. Friday, June 29, 12
40. KPI’s • From the existing database • PLPSQL • Run nightly, offline • Ad Hoc • Returns fixed (absolute) data, saved back to a database • About the numbers Friday, June 29, 12
41. Pretty pictures. TRends Friday, June 29, 12
42. Friday, June 29, 12
43. TRends • Redis backed • Low impact • Time series, fading resolution • GRAPHS • About the relative differences, actual numbers may be incorrect Friday, June 29, 12
44. NEW SHIT or new to us and maybe you Friday, June 29, 12
45. Statsd + Friday, June 29, 12
46. Statsd + Graphite + Friday, June 29, 12
47. graphiti Statsd + Graphite + Friday, June 29, 12
48. gh://etsy/statsd gh://paperlesspost/statsd gh://github/statsd-ruby Statsd Friday, June 29, 12
49. 1 Paperless::Statsd.increment "dashboard.logins" 2 Paperless::Statsd.timing "media_file.convert", time_in_ms Friday, June 29, 12
50. Friday, June 29, 12
51. gh://quirkey/resque-metrics Friday, June 29, 12
52. An alternate frontend for graphite, optimized for play and discovery gh://paperlesspost/graphiti Graphiti Friday, June 29, 12
53. Friday, June 29, 12
54. Graphiti Friday, June 29, 12
55. Graphiti • Sinatra, Sammy, Redis Friday, June 29, 12
56. Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis Friday, June 29, 12
57. Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis • Ace Editor for Play/editing Friday, June 29, 12
58. Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis • Ace Editor for Play/editing • Named Dashboards are collections of graphs Friday, June 29, 12
59. Graphiti • Sinatra, Sammy, Redis • Graphs stored as JSON in Redis • Ace Editor for Play/editing • Named Dashboards are collections of graphs • WIP Friday, June 29, 12
60. Friday, June 29, 12
61. Friday, June 29, 12
62. Attached to Specs/Issues 2. SET Goal metrics Friday, June 29, 12
63. Terminology TANGENT Friday, June 29, 12
64. Terminology TANGENT Friday, June 29, 12
65. • Accounts create Events Terminology TANGENT Friday, June 29, 12
66. • Accounts create Events • Events have a Card Terminology TANGENT Friday, June 29, 12
67. • Accounts create Events • Events have a Card • Card [customizations] are paid for with Coins Terminology TANGENT Friday, June 29, 12
68. • Accounts create Events • Events have a Card • Card [customizations] are paid for with Coins • Events send to Guests Terminology TANGENT Friday, June 29, 12
69. • Accounts create Events • Events have a Card • Card [customizations] are paid for with Coins • Events send to Guests • Guests Reply to Events Terminology TANGENT Friday, June 29, 12
70. Quantitative vs Qualitative Friday, June 29, 12
71. How good your product is. qualitative Friday, June 29, 12
72. Events created per Account Guests added per Event Coins used per Event A single value measured on a per n axis Friday, June 29, 12
73. Quality is the velocity of our product growth Friday, June 29, 12
74. How big your funnel is. Quantitative Friday, June 29, 12
75. % of Accounts that enter the create tool % of Guests that leave a comment % of Events sent with coins A number value measured in a % change over time Friday, June 29, 12
76. QUANTITY is the ACCELERATION of our product growth Friday, June 29, 12
77. Quantitative vs Qualitative Friday, June 29, 12
78. Quantitative X Qualitative GROWTH Friday, June 29, 12
79. A feature can touch both, but in practice it doesn’t. Pick ONE type And a FEW metrics Friday, June 29, 12
80. Attach them to the spec or issue Friday, June 29, 12
81. Friday, June 29, 12
82. Friday, June 29, 12
83. Duh, you guys know Ruby, this part is easy. 3. Build & Roll OUt in Chunks Friday, June 29, 12
84. if current_account_has_feature? :hybrid # show the hybrid Friday, June 29, 12
85. Friday, June 29, 12
86. Friday, June 29, 12
87. Daily, Weekly, Obsessive Checking Report along the way Friday, June 29, 12
88. Friday, June 29, 12
89. Was our hunch correct. 4. roundup and report Friday, June 29, 12
90. And no negative change. Numbers are important but most important is any positive change Friday, June 29, 12
91. Making sure everyone knows what did go right, and what opportunities there are to improve. Roundup meetings Friday, June 29, 12
92. 5. ProfIt? Friday, June 29, 12
93. Still early days Friday, June 29, 12
94. The good Friday, June 29, 12
95. The good • For certain features, it can bring laser focus to the goals. Friday, June 29, 12
96. The good • For certain features, it can bring laser focus to the goals. • The separation of the Q’s has brought a lot of clarity to what we’re actually trying to change. Friday, June 29, 12
97. The good • For certain features, it can bring laser focus to the goals. • The separation of the Q’s has brought a lot of clarity to what we’re actually trying to change. • Can inspire and give a clear goal to developers. Friday, June 29, 12
98. The opportunities Friday, June 29, 12
99. The opportunities • Can be dangerous if people focus on the wrong goals. Friday, June 29, 12
100. The opportunities • Can be dangerous if people focus on the wrong goals. • Hard to actually test features in isolation. Friday, June 29, 12
101. The opportunities • Can be dangerous if people focus on the wrong goals. • Hard to actually test features in isolation. • Some metrics are much harder to measure accurately. Friday, June 29, 12
102. The opportunities • Can be dangerous if people focus on the wrong goals. • Hard to actually test features in isolation. • Some metrics are much harder to measure accurately. • Teams need more mid-process visibility in to goals. Friday, June 29, 12
103. We’re gathering metrics about how metrics driven works NOW Friday, June 29, 12
104. IT’s going pretty well. Friday, June 29, 12
105. github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS! Friday, June 29, 12