P8105: Cross Validation

Transcript

1. 1 CROSS VALIDATION Jeff Goldsmith, PhD Department of Biostatistics
2. 2 • When you have lots of possible variables, you have you choose which ones will go in your model • In the best case, you have a clear hypothesis you want to test in the context of known confounders • (Always keep in mind that no model is “true”) Model selection
3. 3 • Lots of times you’re not in the best case, but still have to do something • This isn’t an easy thing to do • For nested models, you have tests – You have to be worried about multiple comparisons and “fishing” • For non-nested models, you don’t have tests – AIC / BIC / etc are traditional tools – Balance goodness of fit with “complexity” Model selection is hard
4. 4 • These are basically the same question: – Is my model not complex enough? Too complex? – Am I underfitting? Overfitting? – Do I have high bias? High variance? • Another way to think of this is out-of-sample goodness of fit: – Will my model generalize to future datasets? Questioning fit
5. 5 Flexibility vs fit
6. 6 • Ideally, you could – Build your model given a dataset – Go out and get new data – Confirm that your model “works” for the new data • That doesn’t really happen • So maybe just act like it does? Prediction accuracy
7. 7 • Cross validation
8. 8 Cross validation Split Full data Training Testing Apply model Build model RMSE
9. 9 • Individual training / testing splits are subject to randomness • Repeating the process – Illustrates variability in prediction accuracy – Can indicate whether differences in models are consistent across splits • I usually repeat the training / testing split • Folding (5-fold, 10-fold, k-fold, LOOCV) partitions data into equally-sized subsets – One fold is used as testing, with remaining folds as training – Repeated for each fold as testing • I don’t do this as often Refinements and variations
10. 10 • Can use to compare candidate models that are all “traditional” • Comes up a lot in “modern” methods – Automated variable selection (e.g. lasso) – Additive models – Regression trees Cross validation is general
11. 11 • In the best case, you have a clear hypothesis you want to test in the context of known confounders – I know I already said this, but it’s important • Prediction accuracy matters as well – Different goal than statistical significance – Models that make poor predictions probably don’t adequately describe the data generating mechanism, and that’s bad Prediction as a goal
12. 12 • Lots of helpful functions in modelr – add_predictions() and add_residuals() – rmse() – crossv_mc() • Since repeating the process can help, list columns and map come in handy a lot too :-) Tools for CV