Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
On AB Testing
Search
Hector Zarate
September 16, 2016
Technology
2
430
On AB Testing
My slides on my presentation on AB Testing.
As presented in NSSpain 2016.
Logroño, Spain.
Hector Zarate
September 16, 2016
Tweet
Share
More Decks by Hector Zarate
See All by Hector Zarate
On Debugging
ioscowboy
0
540
Inheritance, Interfaces and Composition
ioscowboy
0
460
iOS @ Spotify
ioscowboy
0
130
Consistent, Dumb and Thin
ioscowboy
3
16k
Other Decks in Technology
See All in Technology
Oracle Cloud Infrastructureデータベース・クラウド:各バージョンのサポート期間
oracle4engineer
PRO
29
13k
OCI Network Firewall 概要
oracle4engineer
PRO
0
4.2k
【令和最新版】AWS Direct Connectと愉快なGWたちのおさらい
minorun365
PRO
5
770
開発生産性を上げながらビジネスも30倍成長させてきたチームの姿
kamina_zzz
2
1.7k
データプロダクトの定義からはじめる、データコントラクト駆動なデータ基盤
chanyou0311
2
350
IBC 2024 動画技術関連レポート / IBC 2024 Report
cyberagentdevelopers
PRO
1
120
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
0
110
個人でもIAM Identity Centerを使おう!(アクセス管理編)
ryder472
4
240
障害対応指揮の意思決定と情報共有における価値観 / Waroom Meetup #2
arthur1
5
490
Platform Engineering for Software Developers and Architects
syntasso
1
520
EventHub Startup CTO of the year 2024 ピッチ資料
eventhub
0
130
【Startup CTO of the Year 2024 / Audience Award】アセンド取締役CTO 丹羽健
niwatakeru
0
1.3k
Featured
See All Featured
Automating Front-end Workflow
addyosmani
1366
200k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
229
52k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
47
2.1k
StorybookのUI Testing Handbookを読んだ
zakiyama
27
5.3k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.3k
Typedesign – Prime Four
hannesfritz
40
2.4k
Agile that works and the tools we love
rasmusluckow
327
21k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
27
840
Building Your Own Lightsaber
phodgson
103
6.1k
Fashionably flexible responsive web design (full day workshop)
malarkey
405
65k
KATA
mclloyd
29
14k
Transcript
AB Testing Hector Zarate @ChocoChipset
AB Testing Hector Zarate @ChocoChipset
Hector Zarate (@ChocoChipset) Software Engineer at Spotify Boxun Zhang Data
Scientist at Spotify
1. What?
None
None
YOU ARE PART OF AN EXPERIMENT
AB Testing
None
Samson, 3
Max, 30 Samson, 3 +5% matches
Share of Matches 20% 23.75% 27.5% 31.25% 35% Week 0
Week 1 Week 2 Week 3 Week 4 With Sunglasses Sunglasses
Share of Matches 20% 23.75% 27.5% 31.25% 35% Week 0
Week 1 Week 2 Week 3 Week 4 With Sunglasses Control Sunglasses
Share of Premium Conversion 20% 23.75% 27.5% 31.25% 35% Week
0 Week 1 Week 2 Week 3 Week 4 Variation A Control Sunglasses
User Retention 20% 23.75% 27.5% 31.25% 35% Week 0 Week
1 Week 2 Week 3 Week 4 Variation A Control Sunglasses
None
spotify:user:chocochipset 92b2976bb15d26c9008
1 2 3 4 5 6 7 8 9 10
p ( x ) x 1 / n
1 2 3 4 5 6 7 8 9 10
p ( x ) x 1 / n
1 2 3 4 5 6 7 8 9 10
p ( x ) x 1 / n 0% 60%
Markets
Demographics
User Attributes
None
Login resolveABFlags() response(ABFlags) ABBA Cache AB Values Load Cached or
Default AB Flags
Example button.color = [UIColor spotifyGreen]; BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]);
if (isButtonPink) { // alternate path here: button.color = [UIColor spotifyPink]; } Key Value charts Enabled pink-buttons 0 buffer-quality low gallery-artist Control
Key Value charts Enabled pink-buttons 0 buffer-quality low gallery-artist Control
Example button.color = [UIColor spotifyGreen]; BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]); if (isButtonPink) { // alternate path here: button.color = [UIColor spotifyPink]; } Key Value charts Enabled pink-buttons 1 buffer-quality low gallery-artist Control pink-button 1 Variation A Control
Control Variation
Analytics impressions and interactions
* just a personal preference
2. How?
1. Hypothesis 2. Design 3. Run 4. Analysis
1. Formulate a Hypothesis
2. Design the Test
# shuffle plays 2.1 Target Metrics
# matches Samson, 3 # replies # walks in the
park
2.2 Test Group more test bandwith, less confidence, smaller effect
in business less test bandwith, more confidence, bigger effect in business Small Large
2.3 Duration
3. Run the test
1 2 3 4 5 6 7 8 9 10
p ( x ) x 1 / n Test A Test A Test B Test B Test A / Test B Watch out for conflicting tests!
Don’t cut them short! Bruno Cesar 48’ Ronaldo 89’ Morata
90’ + 4’ 2 -1
0 25 50 75 100 D1 D2 D3 D4 D5
D6 D7 D8 Don’t cut them short!
4. Analyze the results
1. Formulate a Hypothesis new
3. Case Studies
3.1. Losing Calories
None
None
None
None
None
None
Test: Tab Bar Navigation Hypothesis: By switching to TBN, we
expect an increase in the share of users who click at least one menu item. 1
Test: Tab Bar Navigation Hypothesis: By switching to TBN, we
expect an increase in the share of users who click at least one menu item. 2.1
10% New Users 1% Existing Users 2.2
2.3 8 day period
4 Clicks on Menu Items 10% 20% 30% 40% 50%
60% 70% 80% 90% 100% 110% 120% 130% Control Tab Bar 130% 100%
4 Clicks Overall 10% 20% 30% 40% 50% 60% 70%
80% 90% 100% 110% 120% 130% Control Tab Bar 109% 100%
4 1. Increased clicks in Tab Items 2. Decreased clicks
in non-tab items.
WHY AB TEST? QUANTIFY THE IMPACT OF A SPECIFIC CHANGE
3.2. Home in Mexico
My Rock Your Rock
My Home Your Home
My Home Your Home
Test: Home Hypothesis: We will measure an increased second week
retention by using the new ‘Home’ as start page. 1
Test: Home Hypothesis: We will measure an increased second week
retention by using the new ‘Home’ as start page. 2.1
US UK Germany Austria Mexico 4
1. Is the localization good? 2. Are recommendations relevant for
the market? 3. Technical restrictions we are not aware of? 4
12.34 Mbps US 13.70 Mbps UK 13.42 Mbps Germany 15.48
Mbps Austria 7.4 Mbps Mexico * 3.7 Mb 4
1. Hypothesis 2. Design 3. Run 4. Analysis
Test: Home Hypothesis: By compressing the data sent for Home,
more users will play on day one, two and be active during their second week. 1
4. Pitfalls
NO DATA
Max, 30 TOO MANY CHANGES
Max, WOLF AT WALL STREET TOO MANY CHANGES
DONT LISTEN TO YOUR HEART (LISTEN TO THE DATA)
5. Wrap Up
None
None
None
None
None
KNOWLEDGE IS YOUR ROI
A test is an investment Design to maximize learning
NEGATIVE RESULTS ARE STILL POSITIVE
TRY AB TESTING
None
Gracias! Hector Zarate @ChocoChipset spotify.com/jobs
Preguntas! Hector Zarate @ChocoChipset spotify.com/jobs