Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On AB Testing

Hector Zarate
September 16, 2016

On AB Testing

My slides on my presentation on AB Testing.
As presented in NSSpain 2016.

Logroño, Spain.

Hector Zarate

September 16, 2016
Tweet

More Decks by Hector Zarate

Other Decks in Technology

Transcript

  1. Share of Matches 20% 23.75% 27.5% 31.25% 35% Week 0

    Week 1 Week 2 Week 3 Week 4 With Sunglasses Sunglasses
  2. Share of Matches 20% 23.75% 27.5% 31.25% 35% Week 0

    Week 1 Week 2 Week 3 Week 4 With Sunglasses Control Sunglasses
  3. Share of Premium Conversion 20% 23.75% 27.5% 31.25% 35% Week

    0 Week 1 Week 2 Week 3 Week 4 Variation A Control Sunglasses
  4. User Retention 20% 23.75% 27.5% 31.25% 35% Week 0 Week

    1 Week 2 Week 3 Week 4 Variation A Control Sunglasses
  5. 1 2 3 4 5 6 7 8 9 10

    p ( x ) x 1 / n
  6. 1 2 3 4 5 6 7 8 9 10

    p ( x ) x 1 / n
  7. 1 2 3 4 5 6 7 8 9 10

    p ( x ) x 1 / n 0% 60%
  8. Example button.color = [UIColor spotifyGreen]; BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]);

    if (isButtonPink) { // alternate path here: button.color = [UIColor spotifyPink]; } Key Value charts Enabled pink-buttons 0 buffer-quality low gallery-artist Control
  9. Key Value charts Enabled pink-buttons 0 buffer-quality low gallery-artist Control

    Example button.color = [UIColor spotifyGreen]; BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) { // alternate path here: button.color = [UIColor spotifyPink]; } Key Value charts Enabled pink-buttons 1 buffer-quality low gallery-artist Control pink-button 1 Variation A Control
  10. 2.2 Test Group more test bandwith, less confidence, smaller effect

    in business less test bandwith, more confidence, bigger effect in business Small Large
  11. 1 2 3 4 5 6 7 8 9 10

    p ( x ) x 1 / n Test A Test A Test B Test B Test A / Test B Watch out for conflicting tests!
  12. 0 25 50 75 100 D1 D2 D3 D4 D5

    D6 D7 D8 Don’t cut them short!
  13. Test: Tab Bar Navigation Hypothesis: By switching to TBN, we

    expect an increase in the share of users who click at least one menu item. 1
  14. Test: Tab Bar Navigation Hypothesis: By switching to TBN, we

    expect an increase in the share of users who click at least one menu item. 2.1
  15. 4 Clicks on Menu Items 10% 20% 30% 40% 50%

    60% 70% 80% 90% 100% 110% 120% 130% Control Tab Bar 130% 100%
  16. 4 Clicks Overall 10% 20% 30% 40% 50% 60% 70%

    80% 90% 100% 110% 120% 130% Control Tab Bar 109% 100%
  17. Test: Home Hypothesis: We will measure an increased second week

    retention by using the new ‘Home’ as start page. 1
  18. Test: Home Hypothesis: We will measure an increased second week

    retention by using the new ‘Home’ as start page. 2.1
  19. 1. Is the localization good? 2. Are recommendations relevant for

    the market? 3. Technical restrictions we are not aware of? 4
  20. 12.34 Mbps US 13.70 Mbps UK 13.42 Mbps Germany 15.48

    Mbps Austria 7.4 Mbps Mexico * 3.7 Mb 4
  21. Test: Home Hypothesis: By compressing the data sent for Home,

    more users will play on day one, two and be active during their second week. 1