A/B Testing to Create Happy Users

We barely know our users. We know what technology they use, where they live, what they look at on our site - but we don't know their personality or decision-making process. However, A/B testing can give you insight into what works best for your audience on your site. It can make your stakeholders happy, it'll make your users happy, and it'll definitely make you happy.

Transcript

1. A/B Testing to Create Happy Users dyn.com/abtest for slides, links and extra stu f
2. @laraswanson
3. @laraswanson We have no idea what's going on in our users' heads.
4. @laraswanson We rarely check to see if guesses about our users are right.
5. @laraswanson We have some options. eye-tracking field studies surveys/interviews A/B testing
6. @laraswanson We have some options. eye-tracking field studies surveys/interviews A/B testing
7. @laraswanson illustration by Kevin Cornell
8. @laraswanson illustration by Kevin Cornell Content
9. @laraswanson illustration by Kevin Cornell Content Design
10. @laraswanson illustration by Kevin Cornell Content Design Layout
11. @laraswanson illustration by Kevin Cornell Content Design Layout Action
12. A/B testing may be what you need.
13. @laraswanson no user survey could give you this info
14. @laraswanson subtle, continuous improvements
15. @laraswanson solve a design disagreement
16. @laraswanson unique to your site and users
17. Getting Started
18. @laraswanson Test screams (not whispers)
19. @laraswanson Test screams (not whispers) call to action text form length subject lines 41 shades of blue landing page design
20. @laraswanson Test screams (not whispers) call to action text form length subject lines 41 shades of blue landing page design
21. @laraswanson Define your audience
22. @laraswanson Define your audience undergrad/grad site goals geography desktop/mobile academic program
23. @laraswanson Define success
24. @laraswanson Define success form conversions pages per visit churn rate donations error rate
25. Guess the winner!
26. @laraswanson http://whichtestwon.com/archives/7526 A B
27. @laraswanson http://whichtestwon.com/archives/7526 19% more conversions B
28. @laraswanson Give it enough time me
29. @laraswanson Give it enough time Kill a dead test
30. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A B CALCULATE
31. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 B 180 CALCULATE
32. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 52 CALCULATE
33. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 52 99.9% certain B is the winner!
34. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 24 CALCULATE
35. @laraswanson http://www.usere fect.com/split-test-calculator Visitors Goals A 200 26 B 180 24 no clear winner :(
36. Technical Stu f
37. @laraswanson http://www.whichmvt.com/ A/B testing technology
38. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty
39. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty ease of editing by non-techies
40. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty ease of editing by non-techies page load time
41. @laraswanson http://www.whichmvt.com/ A/B testing technology initial implementation di ficulty ease of editing by non-techies page load time integration with analytics
42. @laraswanson Audience segmentation academic program mobile/desktop geography
43. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer
44. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests
45. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA
46. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA calculates significance
47. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA calculates significance page load time-optimized
48. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA calculates significance page load time-optimized can do multiple tests on one page
49. @laraswanson http://www.whichmvt.com/visual-website-optimizer/ Example: Visual Website Optimizer WYSIWYG view of tests integration with GA calculates significance page load time-optimized can do multiple tests on one page pricing starts at $49/month
50. Guess the winner!
51. @laraswanson http://whichtestwon.com/archives/2450 A B
52. @laraswanson http://whichtestwon.com/archives/2450 73% more conversions A
53. Google Analytics integration
54. @laraswanson Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]);
55. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson custom variable slot custom variable slot
56. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson name of your A/B test name of your A/B test
57. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson which version the user saw which version the user saw
58. Version A _gaq.push(['_setCustomVar',1,'Product Comparison','Control',2]); Version B _gaq.push(['_setCustomVar',1,'Product Comparison','Post It',2]); @laraswanson session-level setting session-level setting
59. @laraswanson Google Analytics > Audience > Custom > Custom Variables
60. @laraswanson
61. @laraswanson
62. @laraswanson
63. Guess the winner!
64. @laraswanson http://whichtestwon.com/archives/8413 A B
65. @laraswanson http://whichtestwon.com/archives/8413 More PhD candidates submitted request for information B
66. A/B tests will make you mad.
67. @laraswanson valuable development time
68. @laraswanson may not reach significant results
69. @laraswanson you'll be excited and then maybe disappointed
70. @laraswanson A/B tests will prove you wrong
71. A/B tests will make your users happy.
72. @laraswanson your site will be easier to use
73. @laraswanson users will accomplish what they came to do
74. A/B tests will make your stakeholders happy.
75. @laraswanson better conversion rates
76. @laraswanson alumni engagement
77. @laraswanson insight for each audience segment
78. A/B tests will make you mad happy.
79. @laraswanson knowing why is better than knowing how
80. @laraswanson impress your stakeholders
81. @laraswanson data to say "NO" to bad ideas
82. Talking to stakeholders
83. @laraswanson Stakeholder: I heard from a large donor that it's hard to donate online.
84. @laraswanson Decoder ring: Online donations are down, someone complained about the online experience, I'm lost when it comes to the web so I need your help.
85. @laraswanson You: I'm ready to help! We can actually test a few di ferent versions of the form to see which gets more donations. Decoder ring: Online donations are down, someone complained about the online experience, I'm lost when it comes to the web so I need your help.
86. @laraswanson Stakeholder: Really? You can't just fix it right now?
87. @laraswanson Decoder ring: I don't understand why we'd wait any longer.
88. @laraswanson You: By running a test we can statistically determine which version of the form is better. Decoder ring: I don't understand why we'd wait any longer.
89. @laraswanson You: It avoids putting up a fix now that may actually make things worse, because we won't have any data about how users respond to it. Decoder ring: I don't understand why we'd wait any longer.
90. @laraswanson Stakeholder: I don't understand. How would people see the di ferent versions?
91. @laraswanson Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!
92. @laraswanson You: Half of visitors will see a new version that I think will make our users happier and more likely to donate. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!
93. @laraswanson You: The other half will see the current form. I'll measure donations from each version so we can see which performs better. Decoder ring: This is sounding very technical. Blurgh. JUST FIX IT!
94. @laraswanson Stakeholder: And then what? Then you'll fix it?
95. @laraswanson Decoder ring: I just want to hear that you're going to fix this.
96. @laraswanson You: You bet. And I can continue to run tests on it to improve it more and more over time. Decoder ring: I just want to hear that you're going to fix this.
97. @laraswanson You: We can make the form better than it's ever been and because we'll have data, we'll be confident in its results. Decoder ring: I just want to hear that you're going to fix this.
98. Guess the winner!
99. @laraswanson http://whichtestwon.com/archives/14918 A B
100. @laraswanson http://whichtestwon.com/archives/14918 57% more leads B
101. @laraswanson http://whichtestwon.com/archives/4781 A B
102. @laraswanson http://whichtestwon.com/archives/4781 A B
103. @laraswanson http://whichtestwon.com/archives/4781 A B
104. @laraswanson http://whichtestwon.com/archives/4781 17% more conversions Wednesday and Thursday a ternoons B
105. @laraswanson http://whichtestwon.com/archives/4781 22% more conversions on weekday and weekend mornings A
106. @laraswanson http://whichtestwon.com/archives/2655 A B
107. @laraswanson http://whichtestwon.com/archives/2655 86% more "Subscribe" clicks A
108. @laraswanson http://whichtestwon.com/archives/7453 A B
109. @laraswanson http://whichtestwon.com/archives/7453 2666% higher CTR B
110. @laraswanson http://whichtestwon.com/archives/3026 A B
111. @laraswanson http://whichtestwon.com/archives/3026 25% more form submissions A
112. How did you do?
113. @laraswanson https://twitter.com/katiedel/status/240136499704778754 "Only 10-30% (of 1000's) of experiments improved the metrics they were designed the improve"
114. That's why we need to test.
115. @laraswanson Alternatives to A/B testing
116. @laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt
117. @laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit: http://bit.ly/multi-armed-bandit
118. @laraswanson Alternatives to A/B testing Multivariate testing: http://bit.ly/smashing-mvt Multi-armed bandit: http://bit.ly/multi-armed-bandit Tons of tools: http://whichtestwon.com/techguide
119. Thanks! Questions? dyn.com/abtest @laraswanson dyn.com/careers - we're hiring UI devs!