John Langford on Making Contextual Decisions with Low Technical Debt

Applications and systems are constantly faced with decisions that require picking from a set of actions based on contextual information. Reinforcement-based learning algorithms such as contextual bandits can be very effective in these settings, but applying them in practice is fraught with technical debt, and no general system exists that supports them completely. We address this and create the first general system for contextual learning, called the Decision Service. Existing systems often suffer from technical debt that arises from issues like incorrect data collection and weak debuggability, issues we systematically address through our ML methodology and system abstractions. The Decision Service enables all aspects of contextual bandit learning using four system abstractions which connect together in a loop: explore (the decision space), log, learn, and deploy. Notably, our new explore and log abstractions ensure the system produces correct, unbiased data, which our learner uses for online learning and to enable real-time safeguards, all in a fully reproducible manner.

The Decision Service has a simple user interface and works with a variety of applications: we present two live production deployments for content recommendation that achieved click-through improvements of 25-30%, another with 18% revenue lift in the landing page, and ongoing applications in tech support and machine failure handling. The service makes real-time decisions and learns continuously and scalably, while significantly lowering technical debt.

Transcript

1. Papers We Love June 26 https://arxiv.org/abs/1606.03966 Contextual Decision w/ Low Technical Debt
2. Ex: Which news? Repeatedly: 1. Observe features of user+articles 2. Choose a news article. 3. Observe click-or-not Goal: Maximize fraction of clicks
4. … > 25% increase in clicks (without much tuning)
5. (, , ) (, ) , ) arg max 56789:;< (|, ) test fails L
8. Contextual Bandits not Supervised! Repeatedly: 1. Observe features 2. Choose action ∈ 3. Observe reward Goal: Maximize expected reward
9. BL L LZ KLR KLLS ELL HLL
10. Explore Log Learn Deploy What could possibly go wrong?
11. Client Library Or Web API Join Server Online Learning Offline Learning Policy App context decision reward (, , , ) , (, , , )
12. Client Library Or Web API Join Server Online Learning Offline Learning Policy App context decision reward (, , , ) , (, , , ) Explore
13. Client Library Or Web API Join Server Online Learning Offline Learning Policy App context decision reward (, , , ) , (, , , ) Log
14. Client Library Or Web API Join Server Online Learning Offline Learning Policy App context decision reward (, , , ) , (, , , ) Learn
15. Client Library Or Web API Join Server Online Learning Offline Learning Policy App context decision reward (, , , ) , (, , , ) Deploy
16. Client Library Or Web API Join Server Online Learning Offline Learning Policy App context decision reward (, , , ) , (, , , ) Offline Learn Data
18. http://ds.microsoft.com http://aka.ms/mwt http://arxiv.org/abs/1606.03966 http://hunch.net/~vw