Performance and Maintainability with Continuous Experimentation

Transcript

1. Performance and Maintainability with Continuous Experimentation @sethwalker sethwalker.me/talks/continuous-experimentation
2. @sethwalker Seth Walker Allison McKnight Jonathan Klein @sethwalker @aemcknig @jonathanklein Daniel Espeset @danielespeset @dxna Daniel Na Lara Swanson @laraswanson
4. @sethwalker
5. @sethwalker 175 committers
6. @sethwalker 100 committers touch CSS & JS
7. @sethwalker
8. @sethwalker Confidence?
9. Operations
10. @sethwalker Feature Flags
11. @sethwalker $cfg['save_everything'] = 'staff'; // ... snip ... if (isEnabled('save_everything')) { writeToDB($everything); } else { throwAway($everything); }
12. @sethwalker $cfg['save_everything'] = '1%'; // ... snip ... if (isEnabled('save_everything')) { writeToDB($everything); } else { throwAway($everything); }
13. @sethwalker $cfg['save_everything'] = '5%'; // ... snip ... if (isEnabled('save_everything')) { writeToDB($everything); } else { throwAway($everything); }
14. @sethwalker $cfg['save_everything'] = '1%'; // ... snip ... if (isEnabled('save_everything')) { writeToDB($everything); } else { throwAway($everything); }
15. @sethwalker $cfg['save_everything'] = '5%'; // ... snip ... if (isEnabled('save_everything')) { writeToDB($everything); } else { throwAway($everything); }
16. @sethwalker $cfg['save_everything'] = '10%'; // ... snip ... if (isEnabled('save_everything')) { writeToDB($everything); } else { throwAway($everything); }
17. @sethwalker $cfg['jquery_1_11'] = 'staff'; // ... snip ... if (isEnabled("jquery_1_11")) { <script src="/js/lib/jquery-1.11.0.js"></script> } else { <script src="/js/lib/jquery-1.8.2.js"></script> }
18. @sethwalker $cfg['jquery_1_11'] = '5%'; // ... snip ... if (isEnabled("jquery_1_11")) { <script src="/js/lib/jquery-1.11.0.js"></script> } else { <script src="/js/lib/jquery-1.8.2.js"></script> }
19. @sethwalker $cfg['jquery_1_11'] = '50%'; // ... snip ... if (isEnabled("jquery_1_11")) { <script src="/js/lib/jquery-1.11.0.js"></script> } else { <script src="/js/lib/jquery-1.8.2.js"></script> }
25. @sethwalker $cfg['jquery_1_11'] = '5%'; // ... snip ... if (isEnabled("jquery_1_11")) { <script src="/js/lib/jquery-1.11.0.js"></script> } else { <script src="/js/lib/jquery-1.8.2.js"></script> }
30. @sethwalker $cfg['jquery_1_11'] = '50%'; // ... snip ... if (isEnabled("jquery_1_11")) { <script src="/js/lib/jquery-1.11.0.js"></script> } else { <script src="/js/lib/jquery-1.8.2.js"></script> }
31. @sethwalker $cfg['jquery_1_11'] = '100%'; // ... snip ... if (isEnabled("jquery_1_11")) { <script src="/js/lib/jquery-1.11.0.js"></script> } else { <script src="/js/lib/jquery-1.8.2.js"></script> }
33. @sethwalker
34. @sethwalker window.Etsy.Context = { // ... snip ... "feature":{ "conversations.rejuvination":true, "activity.dynamic_load_on_scroll":true, "overlays.buttery_smooth":true, "collections.perf_render_fix":true, } };
35. @sethwalker if (Etsy.Context.feature["overlays.buttery_smooth"]) { doSmooth(); } else { doJanky(); }
36. @sethwalker TODO: Feature Flags for the asset pipeline
37. @sethwalker $cfg['js_debug_build'] = 'staff'; // ... snip ... if (isEnabled("js_debug_build")) { <script src="/js-debug/main.js"></script> } else { <script src="/js/main.js"></script> }
38. @sethwalker + Analytics
39. Continuous Experimentation
40. @sethwalker Small, measurable changes Dan McKinley - http://mcfunley.com/design-for-continuous-experimentation
41. @sethwalker Validate hypotheses
42. @sethwalker Invalidate hypotheses
45. @sethwalker
47. :(
48. @sethwalker 100+ active flags
49. @sethwalker 60+ active experiments
50. @sethwalker 14 experimental iterations on listing page redesign
64. @sethwalker N inactive experiments
65. @sethwalker N inactive experiments N > 0
66. @sethwalker Cruft
67. @sethwalker if (isEnabled("new_listing_page")) { <script src="/js/listing/new-listing.js"></script> } if (isEnabled("new_favorite_button")) { <script src="/js/new-favorites.js"></script> } if (isEnabled("cart2")) { <script src="/js/add-to-cart2.js"></script> }
68. @sethwalker $cfg["new_listing_page"] = "100%"; $cfg["new_favorite_button"] = "5%"; $cfg["cart2"] = "50%";
69. @sethwalker define("foo", ["logger"], function(Logger) { Logger.warn("foo is deprecated!"); doFoo(); });
70. @sethwalker <link rel="stylesheet" href="/css/base.css" /> <link rel="stylesheet" href="/css/listing.css" /> if (isEnabled("new_listing_page")) { <link rel="stylesheet" href="/css/new-listing.css" /> } if (isEnabled("new_favorite_button")) { <link rel="stylesheet" href="/css/new-favorites.css" /> } if (isEnabled("cart2")) { <link rel="stylesheet" href="/css/add-to-cart2.css" /> }
71. @sethwalker Let's clean up!
72. Fear
73. @sethwalker Throw away
74. @sethwalker Throw away ability
75. @sethwalker Design for Throwaway-ability Bill Scott - http://www.looksgoodworkswell.com/the-experimentation-layer/
76. @sethwalker Tools
77. @sethwalker Needs More Visibility
85. @sethwalker selectorText = document.styleSheets[i].cssRules[j].selectorText; selected = document.querySelector(selectorText); results.push({ selector: selectorText, found: !!selected ? 1 : 0}); // ... snip ... log(results);
86. @sethwalker Existing code - Seen in the wild = SUSPECT
87. @sethwalker Instrumentation
88. @sethwalker Increasing confidence
89. @sethwalker Adapting tools and process
90. Thanks! @sethwalker sethwalker.me/talks/continuous-experimentation