Rachael Tatman - Put down the deep learning: When not to use neural networks and what to do instead

The deep learning hype is real, and the Python ecosystem makes it easier than ever to neural networks to everything from speech recognition to generating memes. But when picking a model architecture to apply to your work, you should consider more than just state of the art results from NeurIPS. The amount of time, money and data available to you are equally, if not more, important. This talk will cover some alternatives to deep learning, including regression, tree-based methods and distance based methods. More importantly, it will include a frank discussion of the pros and cons of different methods and when it makes sense to use each in practice.

https://us.pycon.org/2019/schedule/presentation/200/

Transcript

1. @rctatman PUT DOWN THE DEEP LEARNING When not to use neural networks (and what to do instead) Dr. Rachael Tatman Data Scientist Advocate @ Kaggle
2. @rctatman
3. @rctatman Potterjk [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)]
4. @rctatman Potterjk [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)] Additionally, for BERT LARGE we found that fine-tuning was sometimes unstable on small data sets (i.e., some runs would produce degenerate results), so we ran several random restarts and selected the model that performed best on the Dev set. (Devlin et al 2019)
5. @rctatman GPT-2 model from OpenAI
6. @rctatman I would personally use deep learning if... • A human can do the same task extremely quickly (<1 second) • I have high tolerance for weird errors • I don’t need to explain myself • I have a large quantity of labelled data (>5,000 items per class) • I’ve got a lot of time (for training) and money (for annotation and compute)
7. @rctatman Method Time Money Data Deep Learning A lot A lot A lot
8. @rctatman Method Time Money Data Deep Learning A lot A lot A lot Regression Trees Distance Based
9. @rctatman Regression
10. @rctatman The OG ML technique • In regression, you pick the family of the function you’ll use to model your data • Many existing kinds of regression models ✓ Fast to fit ✓ Works well with small data ✓ Easy to interpret ✘ More data preparation ✘ Models require validation
11. @rctatman My go-to? Mixed effects regression
12. @rctatman # imports for mixed effect libraries import statsmodels.api as sm import statsmodels.formula.api as smf # model that predicts chance of admission based on # GRE & TOEFL score,with university rating as a random effect md = smf.mixedlm("chance_of_admit ~ gre_score + toefl_score", train, # training data groups=train["university_rating"]) # fit model fitted_model = md.fit()
13. @rctatman Mixed Linear Model Regression Results ============================================================= Model: MixedLM Dependent Variable: chance_of_admit No. Observations: 300 Method: REML No. Groups: 5 Scale: 0.0055 Min. group size: 21 Likelihood: 332.7188 Max. group size: 99 Converged: Yes Mean group size: 60.0 -------------------------------------------------------------- Coef. Std.Err. z P>|z| [0.025 0.975] -------------------------------------------------------------- Intercept -1.703 0.169 -10.097 0.000 -2.033 -1.372 gre_score 0.005 0.001 7.797 0.000 0.004 0.007 toefl_score 0.007 0.001 4.810 0.000 0.004 0.009 Group Var 0.002 0.020
14. @rctatman Method Time Money Data Deep Learning A lot A lot A lot Regression Some A little A little Trees Distance Based
15. @rctatman Trees
16. @rctatman Tree based methods
17. @rctatman Random Forests • An ensemble model that combines many trees into a single model • Very popular, especially with Kaggle competitors ◦ 63% of Kaggle Winners (2010-2016) used random forests, only 43% deep learning • Tend to have better performance than logistic regression ◦ “Random forest versus logistic regression: a large-scale benchmark experiment”, Couronné et al 2018 Venkata Jagannath [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)]
18. @rctatman Benefits & Drawbacks ✓ Require less data cleaning & model validation ✓ Many easy to use packages ◦ XGBoost, LightGBM, CatBoost, new one in next scikit-learn release candidate ✖ Can overfit ✖ Generally more sensitive to differences between datasets ✖ Less interpretable than regression ✖ Especially for ensembles, can require more compute/training time
19. @rctatman import xgboost as xgb # split training data into inputs & outputs X = train.drop(["chance_of_admit"], axis=1) Y = train["chance_of_admit"] # specify model (xgboost defaults are generally fine) model = xgb.XGBRegressor() # fit our model model.fit(y=Y, X=X)
20. @rctatman Method Time Money Data Deep Learning A lot A lot A lot Regression Some A little A little Trees Some (esp for big ensembles) A little Some Distance Based
21. @rctatman Distance
22. @rctatman Distance based methods • Basic idea: points closer together to each other in feature space are more likely to be in the same group • Some examples: ◦ K-nearest neighbors ◦ Gaussian Mixture Models ◦ Support Vector Machines Junkie.dolphin [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)] Antti Ajanki AnAj [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0/)]
23. @rctatman Benefits & Drawbacks ✓ Work well with small datasets ✓ Tend to be very fast to train ✖ Overall accuracy is fine, other methods usually better ✖ Good at classification, generally crummy/slow at estimation • These days, tend to show up mostly in ensembles • Can be a good fast first pass at a problem
24. @rctatman from sklearn.svm import SVR # split training data into inputs & outputs X = train.drop(["chance_of_admit"], axis=1) Y = train["chance_of_admit"] # specify hyperparameters for regression model model = SVR(gamma='scale', C=1.0, epsilon=0.2) # fit our model model.fit(y=Y, X=X)
25. @rctatman Method Time Money Data Deep Learning A lot A lot A lot Regression Some A little A little Trees Some (esp for big ensembles) A little Some Distance Based Very little Very little Very little
26. @rctatman So what method should you use?
27. @rctatman Method Time Money Data Deep Learning A lot A lot A lot Regression Some A little A little Trees Some (esp for big ensembles) A little Some Distance Based Very little Very little Very little
28. @rctatman Method Time Money Data Performance (Ideal case) Deep Learning A lot A lot A lot Very high Regression Some A little A little Medium Trees Some A little Some High Distance Based Very little Very little Very little So-so
29. @rctatman Method Time Money Data Performance (Ideal case) Deep Learning A lot A lot A lot Very high Regression Some A little A little Medium Trees Some A little Some High Distance Based Very little Very little Very little So-so User Friendliest Most Lightweight Most Interpretable Most Powerful
30. @rctatman Data Science != Deep Learning • Deep learning is extremely powerful but it’s not for everything • Don’t be a person with a hammer • Deep learning isn’t the core skill in professional data science ◦ “I always find it interesting how little demand there is for DL skills... Out of >400 postings so far, there are 5 containing either PyTorch, TensorFlow, Deep Learning or Keras” -- Dan Becker
31. @rctatman Thanks! Questions? Code & Slides: https://www.kaggle.com/rtatman/non-deep-learning-approaches http://www.rctatman.com/talks/
32. @rctatman Honorable mention: Plain ol’ rules
33. @rctatman Sometimes ✋ Hand-Built ✋ Rules are Best Some examples of proposed deep learning projects from the Kaggle forums that should probably be rule-based systems: • Convert Roman numerals (IX, VII) to Hindu-Arabic numerals (9, 7) • Automate clicking the same three buttons in a GUI in the same order • Given a graph, figure out if a list of nodes is a valid path through it • Correctly parse dates from text (e.g. “tomorrow”, “today”) Remember: If it’s stupid but it works, it’s not stupid.
34. @rctatman (I actually made this figure in R )