Automating Machine Learning

Transcript

1. Andreas Mueller (NYU Center for Data Science, scikit-learn) Automatic Machine Learning?
2. Why?
3. Issues with current tools (scikit-learn)
4. Flow chart / selecting model
5. Selecting Hyper-Parameters
6. Scikit-learn: Explicit is better than implicit make_pipeline( OneHotEncoder(), Imputer(), StandardScaler(), SVC())
7. What? from automl import AutoClassifier clf = AutoClassifier().fit(X_train, y_train) > Current Accuracy: 70% (AUC .65) LinearSVC(C=1), 10sec > Current Accuracy: 76% (AUC .71) RandomForest(n_estimators=20) 30sec > Current Accuracy: 80% (AUC .74) RandomForest(n_estimators=500) 30sec
8. Step 1: Automate Parameter Selection
9. Step 2: Automate Model Selection
10. Step 3: Automate Pipeline Selection
11. How?
12. Formalizing the Search Space Discrete and Continuous Parameters Conditional Parameters Fixed pipeline vs flexible pipeline
13. Formalizing the Search Space Discrete and Continuous Parameters Conditional Parameters Fixed pipeline vs flexible pipeline
14. Search Methods
15. Exhaustive Search (Grid Search)
16. Randomized Search
17. Bayesian Optimization (SMBO)
21. Gaussian Processes
22. Random Forest Based (SMAC)
23. Non-parametric (TPE)
26. Warm-starting and Meta-learning
27. Meta-Learning optimization Algorithm + Parameters Dataset 1
28. Meta-Learning optimization Algorithm + Parameters Dataset 3 optimization Algorithm + Parameters Dataset 2 optimization Algorithm + Parameters Dataset 1
29. Meta-Learning Meta-Features 1 optimization Algorithm + Parameters Dataset 3 optimization Algorithm + Parameters Dataset 2 optimization Algorithm + Parameters Dataset 1 Meta-Features 2 Meta-Features 3 ML model
30. Meta-Learning Meta-Features 1 optimization Algorithm + Parameters Dataset 3 optimization Algorithm + Parameters Dataset 2 optimization Algorithm + Parameters Dataset 1 Meta-Features 2 Meta-Features 3 ML model New Dataset ML model Algorithm + Parameters
31. Meta-Features
32. Existing Approaches
33. auto-sklearn (Hutter, Feurer, Eggensperger) http://automl.github.io/auto-sklearn/stable/
34. Autoweka
35. Hyperopt-sklearn
36. TPot
37. Spearmint https://github.com/HIPS/Spearmint
38. Scikit-optimize
39. Within Scikit-learn • GridSearchCV • RandomizedSearchCV • BayesianSearchCV (coming) • Searching over Pipelines (coming) • Built-in parameter ranges (coming)
40. TODO Clean separation of: • Model Search Space • Pipeline Search Space • Optimization Method • Meta-Learning • Exploit prior knowledge better! • Usability • Runtime consideration
41. TODO Clean separation of: • Model Search Space • Pipeline Search Space • Optimization Method • Meta-Learning • Exploit prior knowledge better! • Usability • Runtime consideration • Data subsampling
42. Criticism
43. Randomized Search works well
44. Do we need 100 Classifiers? Do we need Complex pipelines?
45. I don’t want a black-box!
46. 46 http://oreilly.com/pub/get/scipy
47. 47 Material • Random Search for Hyper-Parameter Optimization (Bergstra, Bengio) • Efficient and Robust Automated Machine Learning (Feurer et al) [autosklearn] • http://automl.github.io/auto-sklearn/stable/ • Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits (Lie et. al) [hyperband] https://arxiv.org/abs/1603.06560 • Scalable Bayesian Optimization Using Deep Neural Networks [Snoek et al]
48. 48 @amuellerml @amueller amueller@nyu.edu http://amueller.io Thank you.