On AB Testing

My slides on my presentation on AB Testing.
As presented in NSSpain 2016.

Logroño, Spain.

Transcript

1. AB Testing Hector Zarate @ChocoChipset
2. AB Testing Hector Zarate @ChocoChipset
3. Hector Zarate (@ChocoChipset) Software Engineer at Spotify Boxun Zhang Data Scientist at Spotify
4. 1. What?
7. YOU ARE PART OF AN EXPERIMENT
8. AB Testing
10. Samson, 3
11. Max, 30 Samson, 3 +5% 
 matches
12. Share of Matches 20% 23.75% 27.5% 31.25% 35% Week 0 Week 1 Week 2 Week 3 Week 4 With Sunglasses Sunglasses
13. Share of Matches 20% 23.75% 27.5% 31.25% 35% Week 0 Week 1 Week 2 Week 3 Week 4 With Sunglasses Control Sunglasses
14. Share of Premium Conversion 20% 23.75% 27.5% 31.25% 35% Week 0 Week 1 Week 2 Week 3 Week 4 Variation A Control Sunglasses
15. User Retention 20% 23.75% 27.5% 31.25% 35% Week 0 Week 1 Week 2 Week 3 Week 4 Variation A Control Sunglasses
17. spotify:user:chocochipset 92b2976bb15d26c9008
18. 1 2 3 4 5 6 7 8 9 10 p ( x ) x 1 / n
19. 1 2 3 4 5 6 7 8 9 10 p ( x ) x 1 / n
20. 1 2 3 4 5 6 7 8 9 10 p ( x ) x 1 / n 0% 60%
21. Markets
22. Demographics
23. User Attributes
25. Login resolveABFlags() response(ABFlags) ABBA Cache AB Values Load Cached or Default AB Flags
26. Example button.color = [UIColor spotifyGreen]; BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) { // alternate path here: button.color = [UIColor spotifyPink]; } Key Value charts Enabled pink-buttons 0 buffer-quality low gallery-artist Control
27. Key Value charts Enabled pink-buttons 0 buffer-quality low gallery-artist Control Example button.color = [UIColor spotifyGreen]; BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) { // alternate path here: button.color = [UIColor spotifyPink]; } Key Value charts Enabled pink-buttons 1 buffer-quality low gallery-artist Control pink-button 1 Variation A Control
28. Control Variation
29. Analytics impressions and interactions
30. * just a personal preference
31. 2. How?
32. 1. Hypothesis 2. Design 3. Run 4. Analysis
33. 1. Formulate a Hypothesis
34. 2. Design the Test
35. # shuffle plays 2.1 Target Metrics
36. # matches Samson, 3 # replies # walks in the park
37. 2.2 Test Group more test bandwith, less confidence, smaller effect in business less test bandwith, more confidence, bigger effect in business Small Large
38. 2.3 Duration
39. 3. Run the test
40. 1 2 3 4 5 6 7 8 9 10 p ( x ) x 1 / n Test A Test A Test B Test B Test A / Test B Watch out for conflicting tests!
41. Don’t cut them short! Bruno Cesar 48’ Ronaldo 89’ Morata 90’ + 4’ 2 -1
42. 0 25 50 75 100 D1 D2 D3 D4 D5 D6 D7 D8 Don’t cut them short!
43. 4. Analyze the results
44. 1. Formulate 
 a Hypothesis new
45. 3. Case Studies
46. 3.1. Losing Calories
53. Test: Tab Bar Navigation Hypothesis: By switching to TBN, we expect an increase in the share of users who click at least one menu item. 1
54. Test: Tab Bar Navigation Hypothesis: By switching to TBN, we expect an increase in the share of users who click at least one menu item. 2.1
55. 10% New Users 1% Existing Users 2.2
56. 2.3 8 day period
57. 4 Clicks on Menu Items 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 110% 120% 130% Control Tab Bar 130% 100%
58. 4 Clicks Overall 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 110% 120% 130% Control Tab Bar 109% 100%
59. 4 1. Increased clicks in Tab Items 2. Decreased clicks in non-tab items.
60. WHY AB TEST?
 QUANTIFY THE IMPACT OF A SPECIFIC CHANGE
61. 3.2. Home in Mexico
62. My Rock Your Rock
63. My Home Your Home
64. My Home Your Home
65. Test: Home Hypothesis: We will measure an increased second week retention by using the new ‘Home’ as start page. 1
66. Test: Home Hypothesis: We will measure an increased second week retention by using the new ‘Home’ as start page. 2.1
67. US UK Germany Austria Mexico 4
68. 1. Is the localization good? 2. Are recommendations relevant for the market? 3. Technical restrictions we are not aware of? 4
69. 12.34 Mbps US 13.70 Mbps UK 13.42 Mbps Germany 15.48 Mbps Austria 7.4 Mbps Mexico * 3.7 Mb 4
70. 1. Hypothesis 2. Design 3. Run 4. Analysis
71. Test: Home Hypothesis: By compressing the data sent for Home, more users will play on day one, two and be active during their second week. 1
72. 4. Pitfalls
73. NO DATA
74. Max, 30 TOO MANY CHANGES
75. Max, WOLF AT WALL STREET TOO MANY CHANGES
76. DONT
 LISTEN TO YOUR HEART (LISTEN TO THE DATA)
77. 5. Wrap Up
83. KNOWLEDGE IS YOUR ROI
84. A test is an investment Design to maximize learning
85. NEGATIVE RESULTS ARE STILL POSITIVE
86. TRY AB TESTING
88. Gracias! Hector Zarate @ChocoChipset spotify.com/jobs
89. Preguntas! Hector Zarate @ChocoChipset spotify.com/jobs