Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Too many LINE Official Account messages? Call d...

Too many LINE Official Account messages? Call data science!

Yoshiaki Nishite
LINE Data Science Team4 Senior Data Scientist
Takahiro Yoshinaga
LINE Machine Learning Solution Team Senior Data Scientist / Machine Learning Engineer
https://linedevday.linecorp.com/2020/ja/sessions/0816
https://linedevday.linecorp.com/2020/en/sessions/0816

LINE DevDay 2020

November 27, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. Agenda › Introduction : DataLabs › Background & Direction ›

    Building & Deploying the Model › Results & Future Prospects
  2. Organization Data Science Teams Data Platform Data Science And Engineering

    Center Data Management Engineering Infrastructure LPML Machine Learning Teams Data Labs
  3. Data Science Teams Data Science Team 1 Data Science Teams

    Data Science Team 2 Data Science Team 3 Data Science Team 4
  4. Machine Learning Teams etc.. Machine Leaning Teams Machine Learning Infrastructure

    Team Machine Learning Development Team Machine Learning Solution Team Machine Leaning Planing Team ɾɾɾ
  5. Broadcast : One of Core Features Broadcast function enable to

    deliver marketing/branding message to all users who followed the account There are some filtering function, like Estimated demographic, Elapsed days from follow, Managed Audience, etc
  6. “User Voice”: Questionnaire User feel receiving many messages from LINE

    Official Account Is the amount of message from LINE Official Account appropriate ? Is the message from LINE Official Account useful ?
  7. Data Analysis Result : User’s Viewpoint More Receiving, Less Opening

    Users tend to open messages less when receiving them more Horizontal axis : ɹ# of receiving messages per user Vertical axis : ɹupper figure : # of users who received messages ɹlower figure : open rate ( = open / receive messages)
  8. Data Analysis Result : Platform’s Viewpoint On the chat list

    on LINE app, LINE Official Accounts are competing user’s message open Even If OA reduces sending message, Frequently Message Received user’s open rate is still low Horizontal axis : ɹfrequency of OA’s message sending Vertical axis : ɹopen rate Color : ɹfrequency of User’s message receiving
  9. Solution Change rules? Relocating Resources ? Raising a price when

    oversending messages ? Increase the manpower of LINE Official Account Operation ?
  10. Solution Change rules? Relocating Resources ? Raising a price when

    oversending messages ? Predicting “Open Rate” and control the volume of message delivery Increase the manpower of LINE Official Account Operation ?
  11. Overview ML system, which is assigned a high score to

    users who are likely to open based on past message delivery history OA User Score Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling Open Score Generator
  12. Traffic / day 80M+ # MAU (JP) 1B+ # LINE

    Official Accounts (JP) 0.24M # Account-User pairs (JP) MAU: As of Jun. 2020 Accounts: As of Sep. 2020 pairs; As of Oct. 2020
  13. Prediction: source features and target Features Past message delivery history

    and open/click history, estimated demographic, account category, etc Target Whether the message was “opened” on the message receipt date talk room ͷsample ͕΄͍͠ 100% displayed
  14. Data Science utilized Open Score Domain knowledge by Data Scientist

    Platform viewpoint feature: # days received messages in last 1week ɹɹɹɹ(≒ Frequency) User viewpoint feature: # past requests (each / all OA) Open Score Past Requests (Each OA) vs Score past requests past requests Past Requests (All OA) vs Score Open Score Percentage # days received messages in last 1week High Score User (Score >= X) Low Score User (Score < X)
  15. Overview OA Dev/B-part/DataLabs Hadoop Cluster Open Score Generator DataLabs Collect

    Event DataHub Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data
  16. Flow Hadoop Cluster Open Score Generator DataLabs Collect Event DataHub

    Messaging API B-part OA CMS OA Dev Request Import Score Send to Service-side Broadcast Collect Event Data CMS → Messaging API with Open Score → Targeting broadcast
  17. Spec Targeting Option in OA CMS Target User Feature Auto

    Targeting • High + Low (Sampling) Score User • Easy • Detailed operation is not possible Manual Targeting • High Score User • Low Score User • Hard • Detailed operation is possible
  18. Spec Business logic Spec How to handle in targeting New

    Follower Users who follow the account for less than 7days are not scored High Score User Rare broadcast Users who have not received messages from a account for more than 7days are not scored High Score User Secrecy of Communication Not agreed accounts and users are not scored Not agreed account and users are dealt as high score user
  19. Pre-simulation How much the volume of messages will be decreased

    ? Top 5 Internal Official Account - 43% All Internal Official Account - 18% JP - 24%
  20. Whyʁ › API multicast & Manage audience function doesn’t filter

    user by OST › So, they used API & Manage audience more than before › For the short-term, they can’t achieve the business goal by OST
  21. Future Prospects › Open Rate & CTR prediction › Contents

    based Prediction › Expanding the scope of OST
  22. OA Score Open Rate & CTR prediction OA User Score

    Send 1 A 0.7 Yes 1 B 0.3 No 2 B 0.1 No 2 C 0.9 Yes 3 C 0.2 No … … … … Scoring Modeling × ×… Open Score Model Click Score Model
  23. Contents based OA Score Contents based Prediction Contents A Contents

    B Contents C OA CMS Messaging API Register Request Response OA Score Generator Contents A 0.7 Contents B 0.4 Ranking Broadcast Contents C 0.6 Account1 Account2 Account3 Account1 Account3 Account2 A user