The internal of LINE Official Account's advanced broadcast

Haruki Okada
LINE OA Dev Team1 Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/B2-2

Transcript

1. 2019 DevDay The Internal of LINE Official Account's Advanced Broadcast > Haruki Okada > LINE OA Dev Team1 Software Engineer
2. LINE Official Account
3. > Businesses can create their own accounts on LINE > Communication with their customers LINE Official Account
4. LINE Official Account Accounts Globally 11M Estimated
 Peak Event Traffic 500K / Sec
5. LINE Official Account Manager > Create and send various types of message > Show statistics of sent messages > 1 : 1 Chat > And so on
6. Broadcasting
7. • Campaign promotion • Release information of a new product • Send a coupon • And so on Broadcasting > Most widely used feature of LINE Official Account
8. Message Insights
9. Renewal Needed > We started to provide new pricing plan as part of “Redesign” project • For user: Get more valuable information • For businesses: Reach appropriate users > Needed to update message insights • Update timing: daily => realtime • Verbosity: only basic metrics => more detailed metrics
10. Renewed in Sep 2019
11. Message Insights Renewal
12. > These metrics are updated in realtime > Detailed User Interaction Metrics • Number of people who viewed the message (unique impression) • Number of people who clicked the link in the message (unique click) • Total impression count of the message • And so on Message Insights Renewal
13. > ✅ User reactions for broadcasted messages from LINE Official Account Target Events of Message Insights > ❌ User reactions for ordinary (≠ LINE Official Account) LINE messaging • e.g. ordinary “Mark as Read”
14. A/B Testing > Businesses can send multiple message variants at once > Analyze user responses in realtime
15. New Event Tracking System
 Needed
16. > Update metrics in realtime > Maintain both of realtime-ness and persistency > Track events without impacting UX Challenges > Large traffic of message events
17. Architecture Overview 1. Send Messages 2. Track Events 3. Stream Processing 4. Batch Processing
18. Agenda > The Internal of 
 Message Event Tracking System > Future
19. Broadcasting - Send Messages
20. Send Messages LINE Official Account Manager MySQL Message Delivery Server Messaging API Schedule Poll
21. Send Messages Messaging API Metadata MySQL Talk Server LINE Official Account Friends …
22. > Possible architecture • 1. Embedding metadata in message itself, then send back when track an user reaction event • 2. Separate metadata from message itself. Streaming join later Send Messages > Message metadata contains information necessary for aggregation
23. > Why ? • To minimize network usage so that never affecting UX Send Messages > We adopt 2. Separate metadata from message itself > Cons • The architecture becomes complicated a bit
24. Track Events > Event Tracking SDK collects detailed user events • Impression of balloon 1,2,… • Click of url X in balloon 1,… • And so on Balloon 1 Balloon 2 Balloon 3 1 Broadcast
25. Track Events LINE Official Account Friends Event Tracker Kafka Spark Streaming HDFS LINE Ads Platform Data Pipeline
26. Stream Processing Kafka Event Processor Metadata MySQL Redis
 Cluster HBase Consume Join Realtime Data Raw Event Log
27. Joining Metadata Event Processor Metadata MySQL Buffer (LinkedBlockingQueue) Event Event … Lookup Multiple Metadata
28. Store Realtime Data To Redis Redis
 Cluster Event Processor Fetch Impression Count per Balloon Impression UU Per Broadcast …
29. Counting Unique User > “Count-Distinct” problem > Generally, calculating count-distinct requires O(N) space > The space may not fit on memory for massive dataset
30. HyperLogLog > A probabilistic algorithm solving “Count-Distinct” problem > Redis provides HyperLogLog via PFxxx command family • PFADD: Add an element to HyperLogLog data structure • PFCOUNT: Estimate distinct count • PFMERGE: Merge two HyperLogLog data structures into one
31. Lua Scripting INCR {Balloon1_Impression} PFADD {Broadcast1_Impression} EXPIRE {Balloon1_Impression} {TTL} EXPIRE {Broadcast1_Impression} {TTL} …. Redis
 Cluster EVALSHA 1 Impression Event
32. Store Raw Event Log to HBase Timestamp EventType UserId AccountId …. HBase PUT Raw Event Log Batch Processing Retargeting Realtime Data
 Recovery
33. Retry Queueing Event Processor Kafka HBase RegionTooBusyException
 FAIL Queueing Reprocess After Backoff
34. Batch Processing HDFS Spark Batch Raw Event Log Table Unique User Metrics Event Count Metrics HBase
 MapReduce Scan Put Import HBase
35. HBase MapReduce Row Column Column impression_broadcastId1 userId1 … impression_broadcastId2 userId42 userId47 impression_broadcastId3 userId55301 … … Raw Event Log Mapper
 Reducer
 MultiTableOutputFormat
 Count Columns Sum Up
36. Gather Realtime Data and Batch Data Redis
 Cluster HBase Official Account Manager Realtime Batch
37. Gather Realtime Data and Batch Data 2019-11-15 (Message Sent Date) 2019-11-16 2019-11-17 2019-11-18 2019-11-19 (Last Batch Date) HBase Redis
 Cluster 2019-11-20 (Realtime) Sum Up
38. Architecture Overview 1. Send Messages 2. Track Events 3. Stream Processing 4. Batch Processing
39. > Provide message retargeting Future Plans > Integration with LINE Ads Platform > Look alike targeting
40. Message Retargeting HBase Kafka LINE DMP User Segment Storage Segment Master SegmentA:
 Users Who Opened Message 1
 SegmentB:
 ….
41. Integration With LINE Ads Platform LINE Ads Platform Message Event Tracking LINE NEWS, Timeline, Smart Channel,…
42. Look Alike LINE Ads Platform Message Event Tracking LINE DMP Look Alike Engine
43. > More new features are planned based on our system > We have built scalable, robust message event tracking system Conclusion
44. Thank You