A/B Testing: Testing product ideas the "Mad Scientist" way

Most major online platforms these days run A/B tests. A lot of them. Even the wording in e-mails get A/B tested.The idea is simple: If you have a large enough user base, you can roll out two or more different versions of your feature, design or wording to different parts of your user base, and see which one is more successful. This is called „experimentation“ on your user base, the same way pharmaceutical companies run trials of medicine before they sell them to the whole population.The implementation of an online A/B test is very tricky, however. In this talk I want to show when A/B testing a feature is a good idea, and when it is not, how to integrate your platform with an A/B testing framework, and some of the challenges of running a meaningful online experiment.

Transcript

1. Experimenting product ideas the “Mad Scientist” Ellen König (@ellen_koenig) SoundCloud
2. Why run experiments online? A B
3. How can we run experiments online? Backend Frontend SoundCloud platform SoundCloud user Data analyst Data warehouse Frontend Data analyst Set up experiment Collect data Analyze results
4. What is involved in running an A/B test? Set up experiment Collect data Analyze results Run the experiment for the configured time Assign users to experiment variants Track the impact on the selected metrics Select metrics Implement the experiment on the platform Configure the experiment setup Configure the experiment assignment Evaluate the results Run statistical tests Decide based on impact
5. How to integrate your application with an A/B testing framework? GET http://assignments.internal.net/assign?user=123 { "status_code": 200, "error": "", "assignment": { "experiment_id": 1, "experiment_name": "homepage", "variant_id": 1, "variant_name": “new_hp" } } } Step 1: Requesting an assignment Bac Fro Sou Dat Data Fro Step 1
6. GET http://www.soundcloud.com Step 2: Show the homepage variant Bac Fro Sou Dat Data Fro Step 2 B How to integrate your application with an A/B testing framework?
7. GET https://eventlogger.soundcloud.com/ pageview? anonymous_id=23&client_id=42&page_name=homepa ge%3Amain&variant=1 Step 3: Log the user behavior Bac Fro Sou Dat Data Fro Step 3 GET http://www.soundcloud.com How to integrate your application with an A/B testing framework?
8. User traffic segmentation picking representative user segments during assignment Sample sizes Calculating the right traffic ratio and experiment duration UI consistency Preventing experiment combinations that create bad UX when running multiple experiments at the same time Appropriate statistical tests The right test for the right type of metric and sample size Challenges when running A/B tests
9. Details: Preserving UX consistency One solution: Use layers Diane Tang et. al. (Google), 2010 UI layer Search results layer Ad results layer UI layer Onboarding layer Stream algorithms Search algorithms Recommen dations algorithms Ad results layer
10. Feel free to ask for more details :) Ellen König (@ellen_koenig) SoundCloud