Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PDCA to improve the LINE NEWS recommendation en...

PDCA to improve the LINE NEWS recommendation engine

LINE DevDay 2020

November 27, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. Agenda › Introduction › About Me › What is LINE

    NEWS › Case Study › AB Testing in LINE NEWS › Post AB Test Analysis
  2. Personalized Contents FOR YOU "FOR YOU" delivers a personalized selection

    of articles to users, based on their interests and browsing histories.
  3. Personalized Contents 20190101 20190201 20190301 20190401 20190501 20190601 20190701 20190801

    20190901 20191001 20191101 20191201 20200101 20200201 20200301 20200401 20200501 20200601 20200701 Monthly Clicks on FOR YOU
  4. - What is the purpose of the AB test? -

    How should the result be interpreted?
  5. Overview of AB Test AB test for comparing Recommendation Engines

    - Control Group: Recommend Engine A (existing) - Treatment Group: Recommend Engine B (new)
  6. What is the Purpose of the AB Test? FOR YOU’s

    Goal - We want users to read more articles - Improve and maximize FOR YOU clicks, impressions, and CTR LINE NEWS’s Goal - We want to increase LINE NEWS' pageviews & ad sales - Potential side effects: - The increase in impressions and clicks on FOR YOU do not exceed the overall decrease in impressions and clicks for non-FOR YOU components - FOR YOU clicks increase, but overall pageviews and/or ad sales decrease
  7. How to Measure the Effect of the AB test? Defining

    KPI 1. FOR YOU Clicks, Impressions and CTR 2. Overall Pageviews per Session 3. Overall News Article Impressions 4. Overall Ad Sales FOR YOU’s KPI LINE NEWS KPI LINE NEWS KPI LINE NEWS KPI
  8. Monitoring the AB test      

                                           - Visualize data with Tableau - Plot both actual values and lift rates
  9. AB Test Results 13% Lift 3% Lift 0.7% Lift 2%

    Lift Lift = (treatment group’s values / control group’s values) - 1 1. FOR YOU Clicks, Impressions and CTR 2. Overall Pageviews per Session 3. Overall News Article Impressions 4. Overall Ad Sales FOR YOU’s KPI LINE NEWS KPI LINE NEWS KPI LINE NEWS KPI
  10. Qualitative Reaction to the AB test Sensational articles are recommended

    through the new recommendation engine… Compared to before, there are more articles that are not suited to my interests
  11. Additional Analysis Hypothesis: Some features are too strong in the

    new recommendation engine. There may be an unintended bias in the recommended articles. Additional analysis: Is there a difference amongst types of article categories exposed in the two groups?
  12. Additional Analysis Result - Impression by article category by gender

    - Impressions of articles in the entertainment category have increased significantly for male Excerpts from article category data
  13. Additional Analysis Result Hypothesis: Some features are too strong in

    the new recommendation engine. There may be an unintended bias in the recommended articles. Conclusion: There is most likely an unintended bias, especially towards men. The exposure frequency of entertainment category articles are clearly higher in the Treatment group.
  14. Next Action Improve the recommendation engine and do the AB

    testing again Examples of further AB tests conducted based on this result include: - Adding features to bring out more categories - Adding features to reflect long-term interests in recommendation results
  15. Session Summary Designing the AB Test - Verbalize and visualize

    the purpose - Aim for overall optimization, not partial optimization After the AB Test - Often does not go as designed - Can be caused by qualitative and ambiguous things - Quantify qualitative feedback - Verbalize with quantitative indicators - Prepare many dimensions for analysis