MLOps in Mercari Group’s Trust and Safety ML Team

In the Mercari Group’s Trust and Safety ML Team, we provide solutions to ensure the safety of the users. Examples of the solutions we provide include anti-money laundering countermeasures, credit card fraud detection, and many others. Some of these solutions are powered by machine learning models. In order to be as reactive as possible to emerging frauds, it is important to streamline the model improvement and deployment processes. In this talk, we will explain our platform and automation, and how each element helps us rapidly deploy new countermeasures. We will cover all MLOps steps: experimentation, training/deployment, evaluation, and metric monitoring. We hope our talk benefits those integrating DevOps into their ML solutions or building ML platforms, especially with GCP’s Vertex AI.

Transcript

1. 1 MLOps in Mercari Group's Trust and Safety ML Team PyCon JP 2024 Calvin Janitra Halim
2. 2 Team Trust and Safety ML Position ML Engineer Work MLOps, Data Analysis, etc. Hobbies Music Production, Jamming Career 2021-04 ~ 2023-12 Rakuten 2024-01 ~ Mercari Email c-halim@mercari.com corronade@gmail.com GitHub, Qiita, Spotify, Soundcloud: CJHJ LinkedIn, Medium, X: My name Calvin Janitra Halim
3. 3 Agenda 1. Introduction a. What’s Trust and Safety? b. ML Team’s Responsibilities 2. System Architecture a. Marketplace Service Architecture b. Fintech Service Architecture 3. Automation Steps a. MLOps Overview b. Experimentation c. Feature store d. Training e. Deployment f. Monitoring 4. Struggles and Learnings 5. Closing
4. 4 Introduction
5. 5 What’s Trust and Safety? Japan Region Fintech Marketplace
6. 6 What’s Trust and Safety? Trust and Safety To ensure a safe and secure marketplace Fraud detection Monitoring Prevent policy violations
7. 7 ML Team’s Responsibilities • Item Moderation • Account Moderation • Transaction Moderation Fintech Marketplace
8. 8 System Architectures
9. 9 Marketplace Service Architecture (Realtime)
10. 10 Fintech Service Architecture (Batch)
11. 11 Service Architecture - Feature Update
12. 12 FT Service Architecture - Inference
13. 13 FT Service Architecture - Sync Results
14. 14 MLOps Overview Adapted from this GCP doc with modification Automation
15. 15 MLOps Overview Adapted from this GCP doc with modification
16. 16 Automation Steps
17. 17 Experimentation Model comparison Data loading EDA Evaluation Error analysis
18. 18 Package as a template notebook Experimentation Model comparison To facilitate the collection of results from models based on different configurations, we use the Vertex AI's experiment feature. Data loading We usually use Feast to make it easier to perform point-in-time joins in historical features. EDA E.g. univariate analysis helps us determine which features will contribute to the model’s performance. Evaluation PR-AUC and Feature importances are what we usually check to evaluate models. And also some other business metrics. Error analysis A deep dive into false positives and false negatives is really helpful for identifying the types of patterns the model could miss, as well as any bugs in the data we use. write from scratch
19. 19 Creating and Using Features
20. 20 Creating and Using Features Define feature tables as dbt models Table dependencies, SQL parameterized values, validation
21. 21 Creating and Using Features Sync between offline and online feature stores Less discrepancy
22. 22 Model Training Automate training through GitOps Sync config between repo and trained model
23. 23 Model Deployment
24. 24 Model Deployment Track trained models through GitOps and model registry
25. 25 Model Deployment Flexible Deployment Types!
26. 26 Deploying Pipelines Automate pipeline compl. and config through GitOps
27. 27 Model Monitoring
28. 28 Struggles and Learnings
29. 29 Risks and Costs Risks Automations Optimizations Reviews & Approvals Cost Monitor & Reduce
30. 30 Struggle with Tools Ended up with Vertex AI Feature Store dbt Dataflow BigQuery BigTable Offline features Online features Vertex AI Endpoint or Ended up with Vertex AI Model Registry GKE Pod
31. 31 Closing https://github.com/CJHJ