AB Testing - Working out your apps to make more $$$

An AB Testing talk given at CakeFest 2014 that covers AB testing as an idea, as well as a real-world case study from SeatGeek.com and introducing an open-source ab testing framework.

Transcript

1. working out your apps to make more $$$ Ab Testing
2. GET /ME • operations @ seatgeek.com • cakephp core developer • blog @ josediazgonzalez.com • github @ josegonzalez • irc @ savant • tweet @ savant
3. How do we make the most money possible? How do we grow as quickly as possible? goals for a site
4. Both ways involve changing how you do things
5. • redesigns • new features • removing features • simplifying user flows things we change
6. how do you know your changes are making an impact?
7. test all the things
8. definitions • Element: A discrete item on a page (button, link, text) • Page: a web page or app view containing your test • Variation: a version of the page that has some changes • Conversion: when a user performs a desired page action
9. what is an ab test?
10. source: wikipedia A/B testing is jargon for a randomized experiment with two variants, A and B, which are the control and treatment in the controlled experiment
11. ab testing is a way to randomly test whether one version of a thing is better than another version
12. ab testing is comparing a control version to an alternate version and seeing what differences there are in conversion
13. ab testing allows you to compare two different sets of visitors and measure conversion changes from each set of users.
14. ab testing is a way to test in production
15. • To ensure your changes are effecting success • Data driven development • To ensure your efforts are minimally wasted why ab test?
16. • Be targeted at a percentage of your traffic • Require less traffic to see effects • Be used in conjunction with heat maps, click tracking etc. ab tests can
17. Things we test • button sizes • colors • page layout • page text • user interaction (some of the)
18. • Adding video to your landing page • Raising conversion points above the fold • Nouns, ie. Masculine vs Feminine vs Unspecified • Positive vs Negative • Form Length, required vs non-required fields Things you can test
19. • You need a representative sample size • Must control for daily/seasonal traffic differences • Must control for skews in available data how long should i run tests for?
20. 1. Brainstorm potential testing targets 2. Create test variants 3. Launch tests and drive traffic to your pages 4. Promote a test winner (and analyze results) 5. Start a new test testing workflow
21. jUDGING TEST RESULTS I understand the results I do not understand the results Conversion Improvement Woo Hoo! You are a genius. Start your next test. ! (Pass) Really? Ok, do a test to figure out the next step. ! (Pass, kinda) Conversion Decline Good save. Stop the test and try a new hypothesis. ! (Pass) Seriously, a kitten just died because of you. ! (Fail) source: http://files.meetup.com/5254712/ultimate-AB-testing-guide.pdf
22. • each variant has a certain (unknown) conversion rate • alternatives can be “chosen” via exploitation or exploration routes • given more traffic to a variant when it has a higher conversion rate Multi-armed bandit
23. • each variant more than one element changed across • requires more traffic to reach a confidence level • harder to integrate with 3rd-party tracking tools • failures are noticeable multi-variate tests
24. a real life case study Testing Mobile checkouts
25. mobile user funnel As a User, I will: • click an event link • click a ticket link • click out to market • purchase a ticket
26. first test had two variations
27. Test 1 • Add a view from seat • Larger buy button • More structured ticket info • Centered map at bottom
28. and the old shitty version soundly trounced the new version
29. and it continued to do so for 4 tests
31. the old version was something i designed drunk
32. i am not a smart man
33. test 5 was a slaughter
34. test 6 was similarly embarrassing
35. we gave up at round 7
36. This continues to be completely confounding. Maybe it has something to do with the event info being at the top? - Jack G, SeatGeek Co-Founder
37. ab tests do not need to confirm your suspicions
38. ab tests can just as easily invalidate your hypothesis
39. how do i start?
40. ab testing on a variety of platforms ! github.com/seatgeek/sixpack sixpack
42. • useful for feature flags • json api • multiple kpis features
43. use SeatGeek\Sixpack\Session\Base as Sixpack; ! $sp = new Sixpack; $test = $sp->participate('test', [‘Buy', ‘Download’]); $alternative = $test->getAlternative(); ! $this->Html->link($alternative, ‘/some/link’); start a test in PHP
44. var session = new sixpack.Session(); session.convert("test", function (err, res) { if (err) throw err; $(“.convert”).html(JSON.stringify(res)); }); Convert in js
45. start testing today!
46. buy my book https://bit.ly/phpcakebook 20% off: CAKEFEST2014
47. thanks