The Case for Public Service Recommender Algorithms

In this position paper we lay out the role for public service organisations within the fairness, accountability, and transparency discourse. We explore the idea of public service algorithms and what role they might play, especially with recommender systems. We then describe a research agenda for public service recommendation systems.

(As given at FATRec2018 https://piret.gitlab.io/fatrec2018/program/ )

Transcript

1. The Case for Public Service Recommender Algorithms Ben Fields (presenting) Rhianne Jones Tim Cowlishaw BBC
2. Motivations and Aims The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
3. Motivations and Aims Data to deliver personalised services has become a key strategic priority for Public Service Media organisations across Europe The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
4. Motivations and Aims Concerns about potential risk for PSM values: • potential to undermine shared and collective media experiences • reinforce audiences’ preexisting preferences • (Public Service) Media becoming more like a goldfish bowl, rather than a window to the world • Lots of examples in recent work: Pasquale 2015, Bulck and Moe 2017, Bennet 2018, Lots 2018, Sørensen and Hutchinson 2018 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
5. Motivations and Aims For traditional commercial applications the goal is a straightforward extension of an organisation’s overall commercial aims https://www.flickr.com/photos/24354425@N03/16593266327 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
6. “inform, educate, and entertain” The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
7. “inform, educate, and entertain” Fairness, accountability, and transparency <=?=> The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
8. A Research Agenda We join wider calls for PSM to do personalisation differently (Bennet 2018, Helberger 2015) We do this from a specific PSM context but with wider relevance in mind The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
9. A Research Agenda Thus far guidance on how PSM should approach personalisation has been vague and not sufficient to drive implementation The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
10. Our position paper asks - how can/should public service algorithms that enshrine the principles of Fairness, Accountability and Transparency (FAT) lead to novel ways to design recommender algorithms? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
11. How do we operationalise public service media (PSM) values as tangible concepts in specific PSM contexts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <1>
12. operationalise PSM • Caveat: Notions of public service inevitably vary across different geo-political and cultural contexts (Helberger 2015) • This is principally about core aspects to a PSM approach/ remits have implications for how we design and evaluate recommenders • Key challenges: operationalising concepts like diversity, surprise, shared experience, etc. The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
13. operationalise PSM “any initiative to promote diversity exposure will first have to deal with ‘the question of what exposure diversity actually is’ as well as how to measure it” (Helberger et al. 2018) The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
14. What are useful metrics for which to optimise (e.g. diversity or serendipity), how should the importance of different metrics be balanced in different PSM contexts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <2>
15. Optimise the metrics Can we select our metrics to explicitly address civic goals: • to counter filter bubble effects (Bozdag and van den Hoven 2015)? • for sociocultural diversity (Sheth et al 2011)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
16. Optimise the metrics How can we broaden the scope of these metrics? e.g serendipity, self-actualisation
 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
17. Optimise the metrics “even if an algorithm is designed with the goal of stimulating ‘diversity’ an assessment of its performance by other measures nullifies these good intentions” (van Es 2017)
 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
18. What data (metadata/audience data) should algorithms work on, what are the limits of this data in its current form and how might awareness of this inform new approaches? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <3>
19. Data selection • Content vs. metadata vs. behavioural? • Privacy/ethical concerns • Regulatory requirements
 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
20. How much accuracy loss is acceptable in pursuit of new metrics, e.g. diversity? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <4>
21. At what (accuracy) cost? • Tradeoffs must be made explicitly to minimise unexpected and undesirable outcomes • Can this question be answered through other research practises? (e.g. audience research, UX methodologies)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
22. How should transparency work - when and to whom is it useful, e.g. regulators? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <5>
23. Transparency for whom? Principles of transparency key to the mission of PSM: They provide the mechanism by which PSM are regulated and held accountable The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
24. Transparency for whom? • Transparency for stakeholders/audiences or regulators? • When we say transparency, when do we mean disclosure? • In a user-facing system this is an issue of enabling consent to be meaningful • However in a stakeholder arrangement this is about disclosure The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
25. To what extent should we be transparent about how we are resolving metric and optimisation complexity (the trade-offs we are making)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <6>
26. Transparency, how much? • What is the effective and maximally useful fidelity of transparency? • Is it optimal/necessary/ideal to exposure users to metric tradeoffs? • How transparent to be about accuracy v. diversity? • Third parties/party platforms/challenges? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
27. How do we design for interpretability and explainability to enable appropriate oversight of how recommenders are making decisions and ensure due accountability? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <7>
28. Design for accountability? • Accountability vital to PSM/BBC • Transparency/full disclosure vs meaningful explanations for a user • Does the desire for transparency push algorithmic design in particular directions? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
29. Design for accountability? • How can complex algorithmic systems designed to be intelligible/interpretable • What types of explanations can a system generate/what’s possible? • What explanations will be sufficient for oversight? • How will explanation needs vary e.g regulators/ stakeholders/editorial/public? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
30. What do emerging approaches in algorithmic auditing offer us in terms of scrutinising recommender systems in the real world? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <8>
31. Algorithmic auditing for recsys? • Does auditing an algorithmic system change the system and by extension the user (experience) • How can we observe and monitor impacts of algorithmic systems? • Is auditing a useful approach to assess bias, unfairness, diversity etc? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
32. What type/level of explanation will be most useful? Will explanations produced for editorial need to vary from the type of explanations PSM may provide to audiences? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <9>
33. Explanation, how much? For whom? • Are we explaining algorithmic decision making to users or to experts (principally: editorial teams)? • What is the distance between these two groups? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
34. How will we determine the value of different potential approaches? How might new methodologies, e.g. multi-method, comparative, or longitudinal research, explore cumulative effects? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <10>
35. Determining value • What does testing look like in a public service context? • Should we consult public on value of different approaches in PSM contexts • Do public service organisations have an obligation to explore longer view and cumulative impacts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
36. Next steps The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
37. better align recommender systems in public service contexts with their underlying value frameworks The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
38. The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
39. Thanks! Let’s have some questions! Ben Fields @alsothings ben.fields@bbc.co.uk http://bit.ly/bbcfatrec18 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018