Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jessica Stringham - Experiment Assignment on th...

Jessica Stringham - Experiment Assignment on the Web

A popular way of improving websites is to run experiments on it. We split users into groups, show two or more variations of the site, measure how well each one does, and then show the best version to everyone. In this talk, I'll walk through a toy Python program that does the first step: splits users into groups.
A few interesting problems arise: grouping users, whitelists, and scaling. I'll share different ways to address them. I'll also give examples of things that can go terribly wrong when designing experiment assignment code.

https://us.pycon.org/2017/schedule/presentation/389/

PyCon 2017

May 21, 2017
Tweet

More Decks by PyCon 2017

Other Decks in Programming

Transcript

  1. Experiment assignment on the web • Experiments on the web

    • Random assignment from scratch • Assignment in practice
  2. Experiment assignment on the web • Experiments on the web

    • Random assignment from scratch • Assignment in practice
  3. Experiment assignment on the web • Experiments on the web

    • Random assignment from scratch • Assignment in practice
  4. Experiment assignment on the web • Experiments on the web

    • Random assignment from scratch • Assignment in practice
  5. Experiment assignment on the web • Experiments on the web

    • Random assignment from scratch • Assignment in practice
  6. # Different ids get different values. >>> choose_color_assignment(user_id=100) red >>>

    choose_color_assignment(user_id=101) blue >>> choose_color_assignment(user_id=102) red
  7. Experiment assignment on the web • Experiments on the web

    • Random assignment from scratch • Assignment in practice
  8. Experiment assignment on the web • Experiments on the web

    • Random assignment from scratch • Assignment in practice
  9. Shuffling is a tool that can help avoid bias But

    watch out - experiment interactions
  10. Assignment in practice • Independence with salts • Abstraction •

    Additional logic This got complicated! A/A tests are a good idea.
  11. Assignment from scratch def choose_color_assignment(user_id): key = "{}|color".format(user_id) assignment_i =

    hash_func(key) % 20 if assignment_i < 10: return 'red' else: return 'blue'