Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AB Testing - Working out your apps to make mor...

AB Testing - Working out your apps to make more $$$

An AB Testing talk given at CakeFest 2014 that covers AB testing as an idea, as well as a real-world case study from SeatGeek.com and introducing an open-source ab testing framework.

Jose Diaz-Gonzalez

August 24, 2014
Tweet

More Decks by Jose Diaz-Gonzalez

Other Decks in Technology

Transcript

  1. GET /ME • operations @ seatgeek.com • cakephp core developer

    • blog @ josediazgonzalez.com • github @ josegonzalez • irc @ savant • tweet @ savant
  2. How do we make the most money possible? How do

    we grow as quickly as possible? goals for a site
  3. definitions • Element: A discrete item on a page (button,

    link, text) • Page: a web page or app view containing your test • Variation: a version of the page that has some changes • Conversion: when a user performs a desired page action
  4. source: wikipedia A/B testing is jargon for a randomized experiment

    with two variants, A and B, which are the control and treatment in the controlled experiment
  5. ab testing is a way to randomly test whether one

    version of a thing is better than another version
  6. ab testing is comparing a control version to an alternate

    version and seeing what differences there are in conversion
  7. ab testing allows you to compare two different sets of

    visitors and measure conversion changes from each set of users.
  8. • To ensure your changes are effecting success • Data

    driven development • To ensure your efforts are minimally wasted why ab test?
  9. • Be targeted at a percentage of your traffic •

    Require less traffic to see effects • Be used in conjunction with heat maps, click tracking etc. ab tests can
  10. Things we test • button sizes • colors • page

    layout • page text • user interaction (some of the)
  11. • Adding video to your landing page • Raising conversion

    points above the fold • Nouns, ie. Masculine vs Feminine vs Unspecified • Positive vs Negative • Form Length, required vs non-required fields Things you can test
  12. • You need a representative sample size • Must control

    for daily/seasonal traffic differences • Must control for skews in available data how long should i run tests for?
  13. 1. Brainstorm potential testing targets 2. Create test variants 3.

    Launch tests and drive traffic to your pages 4. Promote a test winner (and analyze results) 5. Start a new test testing workflow
  14. jUDGING TEST RESULTS I understand the results I do not

    understand the results Conversion Improvement Woo Hoo! You are a genius. Start your next test. ! (Pass) Really? Ok, do a test to figure out the next step. ! (Pass, kinda) Conversion Decline Good save. Stop the test and try a new hypothesis. ! (Pass) Seriously, a kitten just died because of you. ! (Fail) source: http://files.meetup.com/5254712/ultimate-AB-testing-guide.pdf
  15. • each variant has a certain (unknown) conversion rate •

    alternatives can be “chosen” via exploitation or exploration routes • given more traffic to a variant when it has a higher conversion rate Multi-armed bandit
  16. • each variant more than one element changed across •

    requires more traffic to reach a confidence level • harder to integrate with 3rd-party tracking tools • failures are noticeable multi-variate tests
  17. mobile user funnel As a User, I will: • click

    an event link • click a ticket link • click out to market • purchase a ticket
  18. Test 1 • Add a view from seat • Larger

    buy button • More structured ticket info • Centered map at bottom
  19. This continues to be completely confounding. Maybe it has something

    to do with the event info being at the top? - Jack G, SeatGeek Co-Founder
  20. use SeatGeek\Sixpack\Session\Base as Sixpack; ! $sp = new Sixpack; $test

    = $sp->participate('test', [‘Buy', ‘Download’]); $alternative = $test->getAlternative(); ! $this->Html->link($alternative, ‘/some/link’); start a test in PHP
  21. var session = new sixpack.Session(); session.convert("test", function (err, res) {

    if (err) throw err; $(“.convert”).html(JSON.stringify(res)); }); Convert in js