People in the loop machine learning: A case Study in news similarity

An overview of some of the currently active machine learning projects at the BBC and a deep dive into one of them: the content-based recommender systems used for article to article recommendations in World Service online news sites, in particular looking at how we align our similarity models with human opinion of what news articles are similar to one another.

Transcript

1. DR BEN FIELDS 21 NOVEMBER 2019 PEOPLE IN THE LOOP MACHINE LEARNING: A CASE STUDY IN NEWS SIMILARITY https://www.flickr.com/photos/woolamaloo_gazette/47571470732/ SLIDES: http://bit.ly/newssimbbc
2. 2 Intro Machine Learning at the BBC Human vs Machine Similarity Content-based News Recommenders Conclusions STRUCTURE
3. 3 MACHINE LEARNING AT THE BBC
4. 4 OVERVIEW ML AT THE BBC Audience data Content data Audience-facing Internal-facing Audience segmentation Starfruit (autotagger) Mango (NER) Topic Segmentor Article Recommendation VoD Recommendation Kids App and Keyboard Content Origin Graph
5. •Other products of BBC use 3rd party solutions •Domain is weird, our product expires! •ML aligned with BBC values ‣Inform, educate and entertain ‣Context of public service algorithm ‣Transparency •Keep editorial control of automated systems •Multiple language support 5 WHY BUILD IN HOUSE? ML AT THE BBC
6. 6 ML AT THE BBC OWN IT
7. 7 ML AT THE BBC OWN IT
8. 8 ML AT THE BBC ENTITIES, TOPICS, AND THINGS, OH MY! Starfruit Mango Named Entity Recogniser Autotagger BBC Things Linked Data Store and Ontology
9. 9 ML AT THE BBC ENTITIES, TOPICS, AND THINGS, OH MY!
10. 10 ML AT THE BBC ENTITIES, TOPICS, AND THINGS, OH MY!
11. 11 ML AT THE BBC WORLD SERVICE NEWS RECS
12. 12 ML AT THE BBC WORLD SERVICE NEWS RECS
13. 13 ML AT THE BBC WORLD SERVICE NEWS RECS
14. 14 HUMAN VS MACHINE SIMILARITY
15. 15 Article similarity can be an effective mean to recommend news to readers
16. 16 Problem: We need computed content similarity to match (mostly) people’s perception of news article similarity
17. 17 HOW DO HUMANS PERCEIVE SIMILARITY?
18. 18 Or rather: how can we efficiently measure the perception of similarity
19. A proposed methodology: 1. Gather a collection of anchor articles from your corpus. 
 2. For each anchor select two additional articles for comparison 
 3. Present each of these triplets in turn to a human evaluator asking the evaluator to decide which of the two articles is most similar to the anchor 19 TRIANGLE TESTS HUMAN PERCEPTION
20. 20 HOW CAN MACHINES COMPUTE SIMILARITY?
21. 21 COMPUTER READABLE REPRESENTATION MACHINE PERCEPTION Article read (a 1 ,a 2 a 3 ,a 4 ,…,a n ) (b1 ,b2 b3 ,b4 ,…,bn ) (c1 ,c2 c3 ,c4 ,…,cn ) (d1 ,d2 d3 ,d4 ,…,dn )
22. 22 LATENT DIRICHLET ALLOCATION MACHINE PERCEPTION Docs 1 2 3 4 5 6 ... The Irish border Brexit backstop 0.7 0 0 0 0.1 0 Scotland to get AI health research centre 0 0 0.9 0 0 0.1 ... Topics Matrix of docs Topics 1 2 3 4 5 6 .... brexit 0.6 0.3 0 0 0 0 hospital 0 0 0.8 0.2 0 0 ... Topics Matrix of topics Words Articles
23. 23 SIMILARITY MEASURES MACHINE PERCEPTION • Discrete probability distributions • Kullback-Leibler divergence or relative entropy • Information gain between distributions Docs 1 2 3 4 5 6 ... The Irish border Brexit backstop 0.7 0 0.2 0 0.1 0 Scotland to get AI health research centre 0 0 0.9 0 0 0.1 KL = 6.74 KL pairwise distances Similar Different
24. 24 A PROTOTYPICAL CASE: CONTENT-BASED NEWS RECOMMENDERS
25. 25 How can we measure alignment between humans and machines?
26. 26 RUNNING TRIANGLE TESTS PROTOTYPICAL CASE a2 a3 a4 a5 KL distribution of base article a1 KL Which article is more similar to a1 ? a2 or a5 Sample of 12 journalists
27. 27 RUNNING TRIANGLE TESTS PROTOTYPICAL CASE
28. 50 topic model Average agreement: 71% % of answers aligned with algorithm per user 28 ALIGNMENT CASE STUDY Random chance: 0.516 30 topic model Average agreement: 54% 70 topic model Average agreement: 62%
29. • Content similarity recommenders: Use LDA for automatic topic scoring pipeline • Potential in capturing alignment between human and machine perception • Tests could be scaled to a much larger population to more formally assess a similarity model 29 CONCLUSIONS AND FUTURE WORK
30. THANKS! LET’S HAVE SOME QUESTIONS! DR BEN FIELDS PEOPLE IN THE LOOP MACHINE LEARNING: A CASE STUDY IN NEWS SIMILARITY HTTP://CEUR-WS.ORG/VOL-2411/PAPER9.PDF SLIDES: bit.ly/newssimbbc HTTPS://PIRET.GITLAB.IO/FATREC2018/PROGRAM/FATREC2018-FIELDS.PDF HTTPS://WWW.BBC.CO.UK/THINGS/