Experimenting on Humans

How do you know what 55 millions users like? Wix.com is conducting over 1000 experiments per month on production to understand which features our users like and which hurt or improve our business.

In this talk we’ll explain how our engineering team is supporting our product managers in making the right decisions and getting our product roadmap on the right path. We will also present some of the open source tools we developed that help us experimenting our products on humans.

Transcript

1. Experimenting on Humans Aviran Mordo Head of Back-end Engineering @aviranm http://www.linkedin.com/in/aviran http://www.aviransplace.com Talya Gendler Back-end Team Leader www.linkedin.com/in/talyagendler
3. Wix In Numbers  Over 55M users + 1M new users/month  Static storage is >1.5Pb of data  3 data centers + 3 clouds (Google, Amazon, Azure)  1.5B HTTP requests/day  800 people work at Wix, of which ~ 300 in R&D
5.  Basic A/B testing  Experiment driven development  PETRI – Wix’s 3rd generation open source experiment system  Challenges and best practices  Complexities and effect on product Agenda
6. 05:37 A/B Test
8. Home page results (How many registered)
10. This is the Wix editor
11. Our gallery manager What can we improve?
12. Is this better?
14. Product Experiments Toggles & Reporting Infrastructure
15. How do you know what is running?
16. If I “know” it is better, do I really need to test it? Why so many?
18. Sign-up Choose Template Edit site Publish Premium The theory
21.  EVERY new feature is A/B tested  We open the new feature to a % of users ¡ Measure success ¡ If it is better, we keep it ¡ If worse, we check why and improve  If flawed, the impact is just for % of our users Conclusion
22. Start with 50% / 50% ?
24.  New code can have bugs  Conversion can drop  Usage can drop  Unexpected cross test dependencies Sh*t happens (Test could fail)
25.  Language  GEO  Browser  User-agent  OS Minimize affected users (in case of failure) Gradual exposure (percentage of…)  Company employees  User roles  Any other criteria you have (extendable)  All users
26.  First time visitors = Never visited wix.com  New registered users = Untainted users Not all users are equal
27. Start new experiment (limited population)
29. Adding a mobile view
30. First trial failed Performance had to be improved
31. Halting the test results in loss of data. What can we do about it?
32. Solution – Pause the experiment! •  Maintain NEW experience for already exposed users •  No additional users will be exposed to the NEW feature
33. PETRI’s pause implementation  Use cookies to persist assignment ¡ If user changes browser assignment is unknown  Server side persistence solves this ¡ You pay in performance & scalability
34. Decision Keep feature Drop feature Improve code & resume experiment Keep backwards compatibility for exposed users forever? Migrate users to another equivalent feature Drop it all together (users lose data/ work)
36.  Numbers look good but sample size is small  We need more data!  Expand Reaching statistical significance 25% 50% 75% 100% 75% 50% 25% 0% Control Group (A) Test Group (B)
37. Keep user experience consistent Control Group (A) Test Group (B)
38.  Signed-in user (Editor) ¡ Test group assignment is determined by the user ID ¡ Guarantee toss persistency across browsers  Anonymous user (Home page) ¡ Test group assignment is randomly determined ¡ Can not guarantee persistent experience if changing browser  11% of Wix users use more than one desktop browser Keeping persistent UX
40. Always exclude robots Don’t let Google index a losing page Don’t let bots affect statistics
41. 1.  Convert A/B Test to Feature Toggle (100% ON) 2.  Merge the code 3.  Close the experiment Ending successful experiment
43. # of active experiment Possible # of states 10 1024 20 1,048,576 30 1,073,741,824 Possible states >= 2^(# experiments) Wix has ~200 active experiments = 1.606938e+60
44.  Supporting 2^N different users is challenging  How do you know which experiment causes errors? Managing an ever changing production env.
45.  Override options (URL parameters, cookies, headers…)  Near real time user BI tools Specialized tools
46. Integrated into the product
47. Why should product care about the system architecture
48. Share document with other users
49. Document owner is part of a test that enables a new video component
50. What will the other user experience when editing a shared document ? Owner Friend
51. Assignment may be different than owner’s Owner (B) Friend (A)
52.  Enable features by existing content  Enable features by document owner’s assignment  Exclude experimental features from shared documents Possible solutions
54. Petri is more than just an A/B test framework Feature toggle A/B Test Personalization Internal testing Continuous deployment Jira integration Experiments Dynamic configuration QA Automated testing
55.  Expose features internally to company employees  Enable continuous deployment with feature toggles  Select assignment by sites (not only by users)  Automatic selection of winning group*  Exposing feature to #n of users*  Integration with Jira * Planned feature Other things we (will) do with Petri
56. Petri is now an open source project https://github.com/wix/petri
57. Q&A Aviran Mordo Head of Back-end Engineering @aviranm http://www.linkedin.com/in/aviran http://www.aviransplace.com Talya Gendler Back-end Team Leader www.linkedin.com/in/talyagendler https://github.com/wix/petri http://goo.gl/L7pHnd
58. Credits http://upload.wikimedia.org/wikipedia/commons/b/b2/Fiber_optics_testing.jpg http://goo.gl/nEiepT https://www.flickr.com/photos/ilo_oli/2421536836 https://www.flickr.com/photos/dexxus/5791228117 http://goo.gl/SdeJ0o https://www.flickr.com/photos/112923805@N05/15005456062 https://www.flickr.com/photos/wiertz/8537791164 https://www.flickr.com/photos/laenulfean/5943132296 https://www.flickr.com/photos/torek/3470257377 https://www.flickr.com/photos/i5design/5393934753 https://www.flickr.com/photos/argonavigo/5320119828
59.  Modeled experiment lifecycle  Open source (developed using TDD from day 1)  Running at scale on production  No deployment necessary  Both back-end and front-end experiment  Flexible architecture Why Petri
60. PERTI Server Your app Laboratory DB Logs