What do you mean you don't test in production - we experiment daily

by Andrew Thorpe

Step through real life examples of the build, measure, learn process that Open Universities Australia put in place after their MVP launch of a new business Open Training Institute. Learn the actual A/B testing process that was used at Open Training Institute - Selecting a test, building & executing, measuring and productionising a successful test.

Using the actual A/B tests conducted at Open Training Institute we will demonstrate how each stage of the process works, which you can then use to begin your own experimentation journey.

1. 1 awthorpe   stevemac  
2. 2 awthorpe   stevemac   How to confidently test in Prod
3. 3 awthorpe   stevemac   How it all started
4. 4 awthorpe   stevemac   Our evolution of Testing in Production 1. Learn 2. Understanding 3. Self Confidence 4. Wider Audience 5. Responsibility
5. 5 awthorpe   stevemac   1. Learn •  State: –  Attempting to introduce A/B –  No experience –  No tool –  No team –  No confidence
6. 6 awthorpe   stevemac   Phase 1 - Test •  Start/enroll button
7. 7 awthorpe   stevemac   Learn
8. 8 awthorpe   stevemac   Key Learning •  Learn how to test, don’t worry about successful tests •  Use an existing tool •  Apply rigor of science, set success criteria before •  Start very small – no impact to business •  Successful test – what now? –  Optimizely 100%
9. 9 awthorpe   stevemac   2. Understanding •  State: chaotic –  Had a tool –  Could measure basic metrics –  Some idea of what success is •  Adhoc selection of experiments •  Adhoc running of tests •  Lack of rigor around the testing process •  No regularity of the tests
10. 10 awthorpe   stevemac  
11. 11 awthorpe   stevemac   Understanding Failure
12. 12 awthorpe   stevemac   Key Learning •  Focus the test to a specific issue/goal •  Create cadence -> Weekly meeting (Process) •  Defined length of all tests -> Focus •  Develop a backlog -> process of refining & prioritising •  Successful test, still what to do? –  BA involvement earlier
13. 13 awthorpe   stevemac   3. Self Confidence •  State: –  Knew how to run the testing machine –  Basic governance process –  Running out team capacity –  Tests getting more complex, •  Bigger effort to create •  Larger business impact
14. 14 awthorpe   stevemac   Self Confidence •  Simple test –  Breadcrumbs
15. 15 awthorpe   stevemac   Key learning •  Complex tests require dedicated resources •  Optimization of process •  Focus not just on sales uplift •  Successful test, how do we get to Prod. faster?? - Priority implementation lane
16. 16 awthorpe   stevemac   Need repeated success (or you’re going no where)
17. 17 awthorpe   stevemac   4. Wider audience •  State: –  Limited set of people involved & aware (insular) –  Achieving worthwhile success –  Didn’t have authority to change riskier elements (high business impact) –  Gaining wider business visibility, but not the whole picture –  Negative external view, cowboys testing in prod
18. 18 awthorpe   stevemac   Wider audience
19. 19 awthorpe   stevemac   Key learning •  Sell success outside the immediate team –  Gain business and exec support •  Sell failure as much as success –  Learning more from failure •  High visibility & transparency –  Wiki, A3 •  Cheaper to test than full dev build
20. 20 awthorpe   stevemac   5. Responsibility •  State: –  We are are here today –  Understand our tools •  Documentation, prioritisation, what tests –  Defined practice •  Confidence in ability to deliver value •  We have external stakeholder buy in
21. 21 awthorpe   stevemac   Phase 5 – Test •  Test example –  Home redesign •  External agency design options –  Check out redesign –  Smoke and mirrors promo discount
22. 22 awthorpe   stevemac   Phase 5 – Test •  Test example –  Home redesign •  External agency design options –  Check out redesign –  Smoke and mirrors promo discount
23. 23 awthorpe   stevemac   Key learning •  Multiple experiments •  Authority to impact high value assets •  Accountability to spend $$$ on external design work
24. 24 awthorpe   stevemac   What it looks like now – 
25. 25 awthorpe   stevemac   Team Structure •  Cross functional team - Business Stakeholder - Business Analyst - Technical Implementation •  Experiment developer, Tester, UI/UX - Marketing - Analytics - Product Owner •  Dedicated team
26. 26 awthorpe   stevemac   Meeting •  Structure: - Review and accept/reject current, - Prioritize proposed tests - Review test implementation •  Time: 30min – 1hr , set time and location •  Ensure meeting is sign off only •  Responsibilities: - Rules: can say yes, BUT have to be there - No external blockers
27. 27 awthorpe   stevemac   Weekly activities •  Review and design of new –  BA input into proposed test –  Implement next tests on dev environment •  Monitoring of current tests –  Kill switch – regular review –  Early identification of success (heads up for BA) •  Implementation of successful (productionise) –  Optimizely 100% –  Fast lane to dev, but not everything
28. think sharp The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. Thank  you   awthorpe   stevemac  
29. think sharp Innovation, Agile Transformation, Project Delivery? Looking for a career in We’re hiring! Visit: elabor8.com.au/careers
30. 30 awthorpe   stevemac   Image References •  Nowhere [image] Available at: http://4.bp.blogspot.com/-dntZUkFcHPw/UgPVOQ6rCxI/ AAAAAAAAAtQ/Y_IkR6dyjfA/s1600/8220nowhere8221-by-endre-to%CC%81t-1980.png [Accessed 31st August] •  20th Century Fox, (2015) Where are we today – Futurama Fry [image] Available at: http:// memegenerator.net/instance2/1565256 [Accessed 31st August]