Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Human-centric evaluation of similarity spaces o...

Human-centric evaluation of similarity spaces of news articles

In this paper we present a practical approach to evaluate similarity spaces of news articles, guided by human perception. This is motivated by applications that are expected by modern news audiences, most notably recommender systems. Our approach is laid out and contextualised with a brief background in human similarity measurement and perception. This is complimented with a discussion of computational methods for measuring similarity between news articles. We then go through a prototypical use of the evaluation in a practical setting before we point to future work enabled by this framework.
paper at http://ceur-ws.org/Vol-2411/paper10.pdf

Ben Fields

July 25, 2019
Tweet

More Decks by Ben Fields

Other Decks in Science

Transcript

  1. CLARA HIGUERA CABAÑES, MICHEL SCHAMMEL, SHIRLEY KA KEI YU, BEN

    FIELDS 25 JULY 2019 NEWSIR WORKSHOP @SIGIR2019 HUMAN-CENTRIC EVALUATION OF SIMILARITY SPACES OF NEWS ARTICLES https://www.flickr.com/photos/woolamaloo_gazette/47571470732/ SLIDES: bit.ly/newsirbbc
  2. 3 Problem: We need computed content similarity to match (mostly)

    people’s perception of news article similarity
  3. A proposed methodology: 1. Gather a collection of anchor articles

    from your corpus. 
 2. For each anchor select two additional articles for comparison 
 3. Present each of these triplets in turn to a human evaluator asking the evaluator to decide which of the two articles is most similar to the anchor 6 TRIANGLE TESTS HUMAN PERCEPTION
  4. 8 COMPUTER READABLE REPRESENTATION MACHINE PERCEPTION Article read (a 1

    ,a 2 a 3 ,a 4 ,…,a n ) (b1 ,b2 b3 ,b4 ,…,bn ) (c1 ,c2 c3 ,c4 ,…,cn ) (d1 ,d2 d3 ,d4 ,…,dn )
  5. 9 LATENT DIRICHLET ALLOCATION MACHINE PERCEPTION Docs 1 2 3

    4 5 6 ... The Irish border Brexit backstop 0.7 0 0 0 0.1 0 Scotland to get AI health research centre 0 0 0.9 0 0 0.1 ... Topics Matrix of docs Topics 1 2 3 4 5 6 .... brexit 0.6 0.3 0 0 0 0 hospital 0 0 0.8 0.2 0 0 ... Topics Matrix of topics Words Articles
  6. 10 SIMILARITY MEASURES MACHINE PERCEPTION • Discrete probability distributions •

    Kullback-Leibler divergence or relative entropy • Information gain between distributions Docs 1 2 3 4 5 6 ... The Irish border Brexit backstop 0.7 0 0.2 0 0.1 0 Scotland to get AI health research centre 0 0 0.9 0 0 0.1 KL = 6.74 KL pairwise distances Similar Different
  7. 13 RUNNING TRIANGLE TESTS PROTOTYPICAL CASE a2 a3 a4 a5

    KL distribution of base article a1 KL Which article is more similar to a1 ? a2 or a5 Sample of 12 journalists
  8. 50 topic model Average agreement: 71% % of answers aligned

    with algorithm per user 15 ALIGNMENT CASE STUDY Random chance: 0.516 30 topic model Average agreement: 54% 70 topic model Average agreement: 62%
  9. • Content similarity recommenders: Use LDA for automatic topic scoring

    pipeline • Potential in capturing alignment between human and machine perception • Tests could be scaled to a much larger population to more formally assess a similarity model 16 CONCLUSIONS AND FUTURE WORK
  10. THANKS! LET’S HAVE SOME QUESTIONS! CLARA HIGUERA CABAÑES, MICHEL SCHAMMEL,

    SHIRLEY KA KEI YU, BEN FIELDS 25 JULY 2019 NEWSIR WORKSHOP @SIGIR2019 HUMAN-CENTRIC EVALUATION OF SIMILARITY SPACES OF NEWS ARTICLES HTTP://CEUR-WS.ORG/VOL-2411/PAPER9.PDF SLIDES: bit.ly/newsirbbc