Why Ben ❤️s Vanity for running A/B Tests

Transcript

1. Why Ben s Vanity for running A/B Tests Why The Pieces: Metrics & Experiments The Dashboard 30 minutes to our first experiment
2. Why: It’s Ruby and runs deep in the application Track and experiment with more than pageviews (e.g. validation errors, pageflows, changes to multiple pages or templates ) Easy for engineers to configure (double-edge sword) Store experiments in source code Ensure variations do not break things using standard CI flow before it gets to production (integrates with Rspec)
3. The pieces Metrics the objectives you want to influence Experiments the variants/changes you want to evaluate
4. The pieces: metrics - Define a metric with Ruby - Can be explicitly called (track!) or use existing ActiveRecord objects
5. The pieces: experiments - Choose a metric to influence - Define a list of variations - Define a success condition
6. The pieces: experiments (cont) - Branch your code
7. The Dashboard
8. 30 minutes to our first experiment 1. Determine a measurable objective to influence 2. Create a Metric 3. Instrument the metric (if necessary) 4. Create an Experiment 5. Define the variants 6. Update tests to run through multiple variants (also helps inform resilient tests) 7.