RecSys Challenge 2020 Workshop: A Stacking Ensemble Model for Prediction of Multi-type Tweet Engagements

Team Wantedly's 3rd place solution

Shuhei Goda, Naomichi Agata, Yuya Matsumura

Transcript

1. ©2020 Wantedly, Inc. Team Wantedly's 3rd place solution RecSys Challenge 2020 Workshop 26.Sep.2020 - Shuhei Goda, Naomichi Agata, Yuya Matsumura for Prediction of Multi-type Tweet Engagements A Stacking Ensemble Model
2. ©2020 Wantedly, Inc. A Task of predicting different types of engagement on Twitter • Multi-label binary classification that predicts each engagement per (tweet ID, engaging user ID). • Four types of engagements: Like, Reply, RT and RT with comment Evaluation Metrics • PR-AUC • RCE (Relative Cross Entropy) • CHALLENGE TASK Indicates the improvement of prediction relative to the naive prediction.
3. ©2020 Wantedly, Inc. The information provided for the challenge dataset • Tweet info. : tweet ID, timestamp, text token, etc. • Engaging User info. : user ID, following count, follower count, etc. • Engaged with User info. : user ID, following count, follower count, etc. • Engagement info. : timestamps of the engagements Train / Test split for evaluating DATASET DESCRIPTION Training Data ( ~ 120 millions samples ) Testing Data Validation Data 1 week 1 week
4. ©2020 Wantedly, Inc. Label Imbalance • The number of positive samples: RT with Comment < Reply < RT < Like • The positive ratio of Like is 43%, while that of RT with Comment is only 0.7% . DATASET CHARACTERISTICS (1)
5. ©2020 Wantedly, Inc. High correlation between engagement types • Users sometimes make multiple types of engagements for one tweet. • High co-occurrences are observed in some pairs. • e.g. RT and Like , RT and RT with comment DATASET CHARACTERISTICS (2)
6. ©2020 Wantedly, Inc. OVERVIEW OF OUR SOLUTION Model Architecture • Stacking LightGBMs Features • Categorical Features • Network Features • Text Features Training Process • Bagging with negative under sampling • Stratified K-Folds over Retweet with Comment
7. ©2020 Wantedly, Inc. MODEL ARCHITECTURE The First Stage Models The Second Stage Models Like Models Reply Models RT Models RT with Comment Models Target Independent Features Target Dependent Features Like Models Reply Models RT Models RT with Comment Models Like Predictions Reply Predictions RT Predictions RT with Comment Predictions Meta Features
8. ©2020 Wantedly, Inc. The First Stage Models The Second Stage Models Like Models Reply Models RT Models RT with Comment Models Target Independent Features Target Dependent Features Like Models Reply Models RT Models RT with Comment Models Like Predictions Reply Predictions RT Predictions RT with Comment Predictions Meta Features Train LightGBMs for each Engagement type 1st Stage MODEL ARCHITECTURE
9. ©2020 Wantedly, Inc. The First Stage Models The Second Stage Models Like Models Reply Models RT Models RT with Comment Models Target Independent Features Target Dependent Features Like Models Reply Models RT Models RT with Comment Models Like Predictions Reply Predictions RT Predictions RT with Comment Predictions Meta Features 2nd Stage Train LightGBM with 1st stage models predictions MODEL ARCHITECTURE
10. ©2020 Wantedly, Inc. Applying different encoding methods to each categorical variable • Low-cardinality categories: Label Encoding • e.g. language, tweet type • High-cardinality categories: Frequency Encoding & Target Encoding • e.g. tweet ID, user ID Considering the combination of categorical variables • Capture more complex relationships among categorical variables • e.g. Hashtag engaging user ID × Categorical Features FEATURES
11. ©2020 Wantedly, Inc. Graph Features FEATURES Social Follow Graph: considering relationships between users and their social influence • flags that represent whether there are first or second degree connections • PageRank Like Graph: considering user similarities in terms of their preferences • each node represents a user and each edge represents Like engagement • Random Walk with Restarts: the number of visits to engaged with users from engaging users
12. ©2020 Wantedly, Inc. A text-based estimation of Engaging User's preferences • Considering two types of preferences • Preferences for the contents of Tweets • Preferences for the Engaged with Users • Express preferences as the similarity by inner products of the vectors • Tweet: The outputs of pretrained multi-lingual BERT • Engaging User: The averaged vectors of the Tweets users are engaging • Engaged with User: The averaged vectors of the users' Tweets Text Features FEATURES
13. ©2020 Wantedly, Inc. • Count: The number of hashtags, media • Following/Follower: Following count, Follower count, F/F Ratio • Account Age: The time elapsed since user accounts were created • User Activity: Relative active time for each user • Main Language: The main language and its ratio in each user’s Home timeline FEATURES Other Features
14. ©2020 Wantedly, Inc. Use the predictions of the 1st Stage Models of each engagement as features of the 2nd Stage Models. • Since every engagement is highly co-occurring, the information on other engagements is important to predict one engagement. • Take the aggregation of the predictions by categories such as user ID and tweet ID. • Express the tendency of the engagements in each category. Meta Features FEATURES
15. ©2020 Wantedly, Inc. Bagging with Negative Under-Sampling • Use Bagging to create high performance models efficiently with small training dataset for each model. • Sampling process is as below: 1. Apply Negative Under-Sampling to reduce the data size and make the number of positive and negative samples is equal. 2. Apply Random Sampling to make the data size even smaller. TRAINING PROCESS Sampling Process
16. ©2020 Wantedly, Inc. Re-Calibration • The predicted probabilities are based on the downsampling space because of Negative Under-Sampling in training. • Predicting the engagement probability is required since RCE is one of the metrics. • Apply re-calibration below as a post-processing. TRAINING PROCESS Re-Calibration where is the prediction in downsampling space and the negative downsampling ratio. p w p p + 1 − p w
17. ©2020 Wantedly, Inc. Use Stratified K-Folds so that the ratio of positive samples of RT with comment in each fold is equal. • We use the same splits for trainings to predict every engagement targets. • if we use different splits, the negative influence of the leakage due to meta features and target encoding gets bigger. • This is because each engagement is not actually independent. • Considering the calculation time, we set the number of folds to 3. TRAINING PROCESS Validation Strategy
18. ©2020 Wantedly, Inc. EXPERIMENTS Environment All our experiments were conducted on resources as follows: • Google BigQuery • Google Dataflow • Google Compute Engine • vCPUs: 64 • Memory: 600GB Our code is available at • https://github.com/wantedly/recsys2020-challenge
19. ©2020 Wantedly, Inc. EXPERIMENTS Final Results The score of the 2nd stage models is considerably better than the 1st stage models. • The 2nd stage models outperform the 1st stage models on both metrics. • This result supports the effectiveness of our stacking architecture.
20. ©2020 Wantedly, Inc. EXPERIMENTS Final Results The difference between the training and validation score of the 2nd stage models is larger than the 1st stage models • In the case of RCE, the difference in the 1st and 2nd stages is as follows. • 4.251 (1st stage models) < 6.048 (2nd stage models) • We finally considered that it is not a problem as both the training and validation scores improved.
21. ©2020 Wantedly, Inc. EXPERIMENTS Training Data Size In the 2nd stage models, the larger the number of training data, the worse the validation score. • This is due to the use of meta features and target encoding. • Change the number of training data in the 2nd stage models depending on the target. • 100,000 for Like, 1,000,000 for other targets
22. ©2020 Wantedly, Inc. CONCLUSION • We described Team Wantedly’s solution for RecSys Challenge 2020, which won the 3rd place. • We train two stage stacking models to capture the characteristics of high co-occurrence between engagements effectively and efficiently.