Management Data Platform Data Labs Engineering Infrastructure Data Governance Data Strategy Inquiry Management Business Consulting Data Product Management Data ETL Data Engineering IU Dev Data Solutions Cloudera PS/PSE Data Science 1-4 Machine Learning 1-2 DSP ML OCR Voice Speech NLP Speech & Voice Planning SET Delivery Infra Observability Infra
POSIX filesystem YARN Container Docker Container Distributed system Execution engine Read data Write data External data source Export to Collect data Business intelligence
First-stage Recommendation Recommendation for User A Service B News articles Sticker Fortune Service C CRS Engine Second-stage Cross-Domain Recommendation targeting scoring filtering Only a subset of items passes User A 35-39 male Feedback
z-features (user features) Contextual Bandits Algorithm to maximize the rewards. As contexts change, the model should adapt its bandit choice. Cross-domain User / Item Embedding Learn node embeddings in an online manner. 35-39 male fond of music User Manga News Sticker Music User News Sticker
z-features (user features) Contextual Bandits Algorithm to maximize the rewards. As contexts change, the model should adapt its bandit choice. Cross-domain User / Item Embedding Learn node embeddings in an online manner. 35-39 male fond of music User Manga News Sticker Music User News Sticker
z-features (user features) Contextual Bandits Algorithm to maximize the rewards. As contexts change, the model should adapt its bandit choice. Cross-domain User / Item Embedding Learn node embeddings in an online manner. 35-39 male fond of music User Manga News Sticker Music User News Sticker
z-features (user features) Contextual Bandits Algorithm to maximize the rewards. As contexts change, the model should adapt its bandit choice. Cross-domain User / Item Embedding Learn node embeddings in an online manner. 35-39 male fond of music User Manga News Sticker Music User News Sticker
User A Service B News articles Sticker Fortune Service C CRS Engine Second-stage Cross-Domain Recommendation targeting scoring filtering Only a subset of items passes User A 35-39 male Feedback Service D Upload Content First-stage recommendation is not mandatory
in the order of priority › Estimate effects of new UI bias › Open Score for OA › Users tend to open messages less when receiving them more › Predicting `open rate’ and control the volume of message delivery
Fast and stable › asyncio with aiozmq library › Transfer Manager › Manage push/pull sockets lifecycle › MPI › State Synchronization › Distributed Training (e.g. Horovod) Kubernetes mpi run CPU Pod Process Process push push CPU Pod Process Process push push mpi run GPU Pod Process pull Process pull GPU Pod Process pull Process pull Transfer Manager