The Truth about A/B Experimentation

http://chrisjpowers.com @chrisjpowers The Truth About A/B Experimentation

Chris Powers 10 years of development 5 Years of Experimentation

http://chrisjpowers.com @chrisjpowers

http://chrisjpowers.com @chrisjpowers Why Use Experiments?

http://chrisjpowers.com @chrisjpowers Scientiﬁc Method Create a Hypothesis Run an Experiment
Collect Data Analyze

http://chrisjpowers.com @chrisjpowers What is A/B Experimentation?

http://chrisjpowers.com @chrisjpowers Split Testing (A/B)

http://chrisjpowers.com @chrisjpowers Multivariant Testing VARIATION 2 VARIATION 3

http://chrisjpowers.com @chrisjpowers Multivariate Testing Control White Text Black Text Control
Control Variant 3 Variant 6 Blue BG Variant 1 Variant 4 Variant 7 Green BG Variant 2 Variant 5 Variant 8

http://chrisjpowers.com @chrisjpowers Great Expectations

http://chrisjpowers.com @chrisjpowers Big Lifts 14% 62%

http://chrisjpowers.com @chrisjpowers Enormous Lifts 228%

http://chrisjpowers.com @chrisjpowers Incredible Lifts (Literally) 450%

http://chrisjpowers.com @chrisjpowers Realistic Goals 0.5% 5% => 5.5% 5% =>
5.025%

http://chrisjpowers.com @chrisjpowers $1B * 0.00025 = $250,000

http://chrisjpowers.com @chrisjpowers Example Experiments

http://chrisjpowers.com @chrisjpowers Inline Deal Options 8 Million Samples 46 days
Lift: 0.81% (± 0.69%)

http://chrisjpowers.com @chrisjpowers Free Shipping Text 12.5 Million Samples 63 days
Lift: 0.78% (± 0.67%)

http://chrisjpowers.com @chrisjpowers Top Goods Categories 80 Million Samples 8 days
Lift: 0.07% (± 0.47%)

http://chrisjpowers.com @chrisjpowers Brands Search 14 Million Samples 55 days Lift:
-0.79% (± 0.41%)

http://chrisjpowers.com @chrisjpowers Flat 55% Negative 30% Positive 15%

http://chrisjpowers.com @chrisjpowers Signiﬁcance and Power

http://chrisjpowers.com @chrisjpowers Statistical Signiﬁcance T-Score p < 0.05 95% Conﬁdence
Controls False Positive Results

http://chrisjpowers.com @chrisjpowers 5% False Positive Rate

http://chrisjpowers.com @chrisjpowers Statistical Signiﬁcance “If my p-value is less than
0.05, then I have a 95% chance that my reading is correct.”

http://chrisjpowers.com @chrisjpowers Statistical Signiﬁcance “If my p-value is less than
0.05, then I have a 95% chance that my reading is correct.” Incorrect

http://chrisjpowers.com @chrisjpowers Statistical Signiﬁcance “If my treatment has no actual
effect (null hypothesis) then there’s a 5% chance I will measure a p-value less than 0.05.” correct

http://chrisjpowers.com @chrisjpowers Statistical Power 80% Conﬁdence Controls False Negative Results
Sets required sample size

http://chrisjpowers.com @chrisjpowers 20% False NEGATIVE Rate

http://chrisjpowers.com @chrisjpowers How Many Samples?

http://chrisjpowers.com @chrisjpowers Sample Size Equation Coefﬁcient: 5% False Positive, 20%
False Negative

http://chrisjpowers.com @chrisjpowers Sample Size Equation Sample size scales linearly with
variance.

http://chrisjpowers.com @chrisjpowers Sample Size Equation Sample size gets smaller (squared)
with larger values.

http://chrisjpowers.com @chrisjpowers Sample Size Equation Sample size gets larger (squared)
as you try to measure smaller lifts.

http://chrisjpowers.com @chrisjpowers Sensitivity to Effect Size 20%: 6K Samples per
Treatment 10%: 25K Samples per Treatment 5%: 100K Samples per Treatment 1%: 2.5M Samples per Treatment 0.5%: 10M Samples per Treatment

http://chrisjpowers.com @chrisjpowers Sensitivity to Effect Size 0 2500000 5000000 7500000
10000000 0.5% 1% 5% 10% 20%

http://chrisjpowers.com @chrisjpowers You May Need These Lifts

http://chrisjpowers.com @chrisjpowers Living with Small Samples

http://chrisjpowers.com @chrisjpowers Run experiments at 50/50. Only amateurs run experiments
at 1%. - Ron Kohavi

http://chrisjpowers.com @chrisjpowers The Peeking Problem

http://chrisjpowers.com @chrisjpowers Peeking Problem Peeking regularly and stopping experiments when
signiﬁcance is achieved raises the 5% false positive rate to… 30%

http://chrisjpowers.com @chrisjpowers 30% False Positive Rate

http://chrisjpowers.com @chrisjpowers Flat 55% Negative 30% Positive 15%

http://chrisjpowers.com @chrisjpowers False 17% Flat 39% Negative 30% Positive 15%

http://chrisjpowers.com @chrisjpowers Directionality

http://chrisjpowers.com @chrisjpowers Mitigating Risk

http://chrisjpowers.com @chrisjpowers Losing Money -100 -75 -50 -25 0 Start
Sample Size

http://chrisjpowers.com @chrisjpowers Using a Smoke Test Smoke Test: 5% Sensitivity,
100K Users Main Test: 0.5% Sensitivity, 10M Users

http://chrisjpowers.com @chrisjpowers Group Sequential Analysis How can we peek while
still controlling our false positive rate?

http://chrisjpowers.com @chrisjpowers Organizational Hurdles

http://chrisjpowers.com @chrisjpowers HIghest Paid Person in the Ofﬁce

http://chrisjpowers.com @chrisjpowers One Last Thought

http://chrisjpowers.com @chrisjpowers 40 Experiments

http://chrisjpowers.com @chrisjpowers 2 ^ 40 1,099,511,627,776

http://chrisjpowers.com @chrisjpowers 99.5% of Experiences

http://chrisjpowers.com @chrisjpowers Thank You

Questions?

The Truth about A/B Experimentation

The Truth about A/B Experimentation

More Decks by Chris Powers

Other Decks in Technology

Featured

Transcript