當機器學習遇上敏捷

在近年來，機器學習及資料導向的技術已是許多產業裡不可或缺的一環。
而在這類的工作中，「研究」時常佔了很重的比例，在如此需要專精的工作模式中，如何導入敏捷是個值得探討的課題。

在這場演說中主要分享我在公司中實際推動敏捷之經驗。背景為一家以資料導向為主的公司，佐以活生生血淋淋的案例，將不同面向的機器學習任務做分類，歸納為經驗與大家分享。
希望能透過我的踩雷實錄與心路歷程，能事先知道要避開哪些雷、吸取成功的方法，讓大家在未來實際操作上更有啟發。

Transcript

1. 當機器學習遇上敏捷 When Machine Learning Meets Agile Agile Tour Kaohsiung 2018
2. Agile Principle: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Valuable software: Product This talk focuses on product, not research. So let us talk about the product that I’m working on first.
3. Bridgewell DSP • A Taiwanese digital marketing company • We are one of the major ad inventory buyer from Google, Facebook, Yahoo, etc… in Taiwan • Automated real time bidding • Trading desk • Ad serving About the product I’m working on
4. What is real time bidding (RTB)? • The product I build is the Demand-Side Platform (DSP) for the Buy-Side
5. 1) We earn money when a user: a) Clicks on our banner ads, or b) Finishes viewing our video ads, or c) Does something (advertiser defined) after viewing our ad 2) We pay money every time we show an ad Profit = 1) - 2) So why do we need ML?
6. Using the a) cost per click (CPC) for example: Profit = 1) - 2) = CPC * CTR - Market price Strategy for making money: Buy ad impression when CPC * CTR > Market price So why do we need ML? (pt. 2)
7. Strategy: CPC * CTR > Market price • We can’t influence market price, yet ;) • We use machine learning: • To predict CTR for each ad impression (when an ad is shown) • To adjust CPC per ad impression but still maintaining the average CPC So why do we need ML? (pt. 3)
8. Our ML environment • A lot of new data everyday • Need real time predictions • Huge amount of predictions per second • Predictions needs to be working 24 hours a day, 365 days an year • Data is very, very, VERY messy
9. Product Value vs Investment
10. ML Performance vs Investment
11. A Real Example A lot of effort, tiny gain. Most of the performance is achieved right at the beginning of the challenge.
12. Story At Bridgewell we had ML researchers, and each one of them are assigned a metric to optimize. • Moral are low • Turnover rate is high • No real break through in years Our solution was to give them more time and freedom: • Everything got worse
13. So what should we do? • Hire a superstar ML expert, or • Hire a ML expert that have product visions Good luck doing that. Real solution: • Bring ML people into planning meetings, or business discussions • People need constant feedback to do stuff good.
14. Pitfalls • No long and pointless meetings • Don’t discuss about stuff that are too vague • Focus on concrete problems
15. Story We start bring ML people into product plannings. • They are able to give solutions to non-ML problems • They can identify what product problems are solvable using data • They can work on stuff with the most value We start to bring ML people into one team • They can work in teams and have solutions with better quality
16. Story A friend of mine made a model that is 3% better than the current one running online. • His model was a bit complex • No one in the engineering team understands ML • He couldn’t find help to make it into production Solution was that he worked on it for three years to make it production ready: • Still did not went online • He learned a lot during the three years • But at a huge expense of the company
17. Story Years ago we let one of the ML researcher write production code. • Hard to understand • A lot of experimental stuff • People are too scared to remove it Our solution was to wrap it with more code. • Stuff are even harder to understand • Development velocity went down vastly
18. ML code: Magical Logic code if (user.name.length >= 20) { return new Error(Error.NameLengthError); } vs for (i = 0; i < x.length; i += 1) { w[i] += w[i] + a[i] * (y - p) * x[i]; } ML code is super hard to understand!
19. So what should we do? • Hire a ML expert that is able to do production level coding Good luck doing that. Real solution: • Bring ML people into development teams • Individuals and interactions over processes and tools
20. As a PO • Clearly state your problems, avoid XY problems. • Try to let ML people come up with their own metrics.
21. How To Write ML Related Tasks • Achieve 90% accuracy in XXX problem. (Bad) • Try to achieve the best accuracy so that we can solve XXX. (Good)
22. As a SM ML people have a strong research tendency. • Motivate ML people to achieve better • Try to bring the whole team together • Try to build a feature team • No need to let everyone know everything • Try giving ML people 10 to 20% of their time to think about/do deeper stuff
23. As a Member If you know ML. • Try to blend into the whole team, it really helps • Need a quick solution? Use Kaggle! If you don’t know ML. • You don’t really need to know a lot to do a lot • Learn tools like • Tensorflow • Scikit Learn • Online resources • https://www.youtube.com/user/hsuantien
24. Tools Matters!
25. DevOps is important for agile • Source code version control • CI/CD • Test Automation • Containerization • Monitoring • Configuration
26. • Data pipeline • Data collection • Data storage • Algorithms • Modeling • Training • Model distribution • Modeling: Prediction • Monitoring DataOps is important for agile
27. Our Data Pipeline • Web Servers • Kafka • Jenkins • Hadoop • Cassandra • MySQL • Spark
28. Our Monitoring • OpenTSDB • Prometheus + • Grafana
29. 特別贊助 協辦單位 台灣敏捷協會