Complexity In Recommender Systems

In this talk, we will discuss some of the challenges with operating mature recommendations systems while continuing advancements. We will touch on interaction across multiple recommendations systems, challenges with personalization, testing and consistency in the user experience.

Video presentation: https://www.youtube.com/watch?v=YFOVoxQHe7Q

Transcript

1. Complexity in Recommendations Systems Nishan Subedi Head Of Algorithms, VP of Technology Overstock.com
2. Goal: • Highlight complexities with long running recommendations systems in production • Highlight some approaches to mitigation of complexities • Recommend experimentation practices • Outline architecture and similarities to Learning To Rank • Ecosystem effects of recommendations
4. Overstock
5. Who am I • Head of Algorithms, VP of Technology @ Overstock • Products include: ◦ Organic search (user interface, backend systems, machine learning) ◦ Ads (auction, ranking, partner interface) ◦ Recommendations Systems (machine learning & systems) ◦ Pricing Science ◦ CoreML (forecasts, estimates) • Previously scientist @ Etsy on Ranking, experience with Systems Engineering
6. Complexity: • Feedback & bias ◦ Positional & presentation bias ◦ Features reinforce biases • Emergence • Long term effects - deviation and drift • ML Debt ◦ Entanglement ◦ Dependencies ◦ Feature erosion • Each learning algorithm can be considered an agent • Measurements may not generalize
7. Experimentation and measurement challenges: • Long-term experiments and holdouts have big engineering costs. • Proxy metrics • Hard to find HEART (Happiness, Engagement, Adoption, Retention, and Task-success) metrics for recommendations • Heterogeneity in treatment effects • Market effects: ensure similarity in market conditions between variants • Interaction effects • Learning about a model’s ability to learn quickly, not long term convergence
8. Recommendation system https://arxiv.org/abs/1907.12372
9. Formulation as LTR • Re-ranking layer for recommendations • Embedding based retrieval system • Modeling the user and entities as embeddings • Optimizing recommendations for specific goals • Synthetic training data generation • Leveraging session information https://arxiv.org/pdf/2006.11632.pdf https://static.googleusercontent.com/media/re search.google.com/en//pubs/archive/45530.p df
10. Recommender ecosystem
11. Recommender ecosystem Overall evaluation criterion (OEC)
12. Personalization • Your platform is one part in the customer’s overall journey. • Consistency in the customer’s journey ◦ Consistent user representation ◦ Unified training and production infrastructure ◦ Using signals across multiple platforms • Factoring the stage of the user’s journey • Incorporating context from source
13. Thankyou! https://www.linkedin.com/in/nishan-subedi-0b7aa928/