Make Machine Learning Boring Again: Best Practices for Using Machine Learning in Businesses - LA Data Science Meetup - Playa Vista, August 2019

Transcript

1. Make Machine Learning Boring Again: Best Practices for Using Machine Learning in Businesses Szilard Pafka, PhD Chief Scientist, Epoch LA Data Science Meetup Aug 2019
3. Disclaimer: I am not representing my employer (Epoch) in this talk I cannot confirm nor deny if Epoch is using any of the methods, tools, results etc. mentioned in this talk
9. y = f (x1, x2, ... , xn) Source: Hastie etal, ESL 2ed
10. y = f (x1, x2, ... , xn)
15. #1 Use the Right Algo
16. Source: Andrew Ng
34. *
35. #2 Use Open Source
41. in 2006 - cost was not a factor! - data.frame - [800] packages
49. #3 Simple > Complex
51. 10x
60. #4 Incorporate Domain Knowledge Do Feature Engineering (Still) Explore Your Data Clean Your Data
72. #5 Do Proper Validation Avoid: Overfitting, Data Leakage
87. #6 Batch or Real-Time Scoring?
89. https://medium.com/@HarlanH/patterns-for-connecting-predictive-models-to-software-products-f9b6e923f02d
90. https://medium.com/@dvelsner/deploying-a-simple-machine-learning-model-in-a-modern-web-application-flask-angular-docker-a657db075280 your app
93. R/Python: - Slow(er) - Encoding of categ. variables
94. #7 Do Online Validation as Well
96. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation
97. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation
98. https://www.oreilly.com/ideas/evaluating-machine-learning-models/page/2/orientation https://www.slideshare.net/FaisalZakariaSiddiqi/netflix-recommendations-feature-engineering-with-time-travel
99. #8 Monitor Your Models
101. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/
102. https://www.retentionscience.com/blog/automating-machine-learning-monitoring-rs-labs/
104. 20% 80% (my guess)
105. 20% 80% (my guess)
106. #9 Business Value Seek / Measure / Sell
112. #10 Make it Reproducible
122. Cloud (servers)
123. ML training: lots of CPU cores lots of RAM limited time
124. ML training: lots of CPU cores lots of RAM limited time ML scoring: separated servers
125. ML (cloud) services (MLaaS)
127. “people that know what they’re doing just use open source [...] the same open source tools that the MLaaS services offer” - Bradford Cross
128. Kaggle
130. already pre-processed data less domain knowledge (or deliberately hidden) AUC 0.0001 increases "relevant" no business metric no actual deployment models too complex no online evaluation no monitoring data leakage
131. Tuning and Auto ML
132. Ben Recht, Kevin Jamieson: http://www.argmin.net/2016/06/20/hypertuning/
133. GPUs
134. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s]
135. Aggregation 100M rows 1M groups Join 100M rows x 1M rows time [s] time [s] “Motherfucka!”
137. API and GUIs
140. AI?
144. How to Start?