Managing many models

Code at https://gist.github.com/hadley/056cf4074acedc164161d6abb751cb35

1. Hadley Wickham 
 @hadleywickham
 Chief Scientist, RStudio Managing 
 many models February 2017
2. You’ve never seen data presented like this. With the drama and urgency of a sportscaster, statistics guru Hans Rosling debunks myths about the so- called “developing world.” https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen
4. 40 60 80 1950 1960 1970 1980 1990 2000 year lifeExp 142 countries
5. • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • 0.0 0.2 0.4 0.6 0.8 0.00 0.25 0.50 0.75 1.00 R2 Estimated yearly increase in life expectancy continent • • • • • Africa Americas Asia Europe Oceania
6. But... Arbitrarily complicated models Three simple underlying ideas Scales to big data
7. Each idea is partnered with a package 1. Nested data (tidyr) 2. Functional programming (purrr) 3. Models → tidy data (broom)
8. Nested data
9. 40 60 80 1950 1960 1970 1980 1990 2000 year lifeExp 142 countries Want to summarise each with a linear model
10. Currently our data has one row per observation Country Year LifeEx p Afghanistan 1952 28.9 Afghanistan 1957 30.3 Afghanistan ... ... Albania 1952 55.2 Albania 1957 59.3 Albania ... ... Algeria ... ... ... ... ...
11. More convenient to one row per group Country Data Afghanistan <df> Albania <df> Algeria <df> ... ... Year LifeExp 1952 28.9 1957 30.3 ... ... Year LifeExp 1952 55.2 1957 59.3 ... ... I call this a nested data frame
12. library(dplyr) library(tidyr) by_country <- gapminder %>% group_by(continent, country) %>% nest() In R:
13. x %>% f(y) # is the same as: f(x, y) gapminder %>% group_by(continent, country) %>% nest() # same as: nest(group_by(gapminder, continent, country)) Haven’t seen pipes?
14. Each country will have an associated model Country Data Afghanistan <df> Albania <df> Algeria <df> ... ... lm(lifeExp ~ year1950, data = afghanistan) lm(lifeExp1950 ~ year, data = albania)
15. Why not store that in a column too? Country Data Model Afghanistan <df> <lm> Albania <df> <lm> Algeria <df> <lm> ... ... ...
16. List-columns keep related things together Anything can go in a list & a list can go in a data frame
17. library(dplyr) library(purrr) country_model <- function(df) { lm(lifeExp ~ year1950, data = df) } models <- by_country %>% mutate( mod = map(data, country_model) ) In R:
18. Functional programming Motivated by baking cupcakes Or, why for loops are “bad”
19. 1 cup flour a scant ¾ cup sugar 1 ½ t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Vanilla cupcakes The hummingbird bakery cookbook
20. ¾ cup + 2T flour 2 ½ T cocoa powder a scant ¾ cup sugar 1 ½ t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, cocoa, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Chocolate cupcakes The hummingbird bakery cookbook
21. ¾ cup + 2T flour 2 ½ T cocoa powder a scant ¾ cup sugar 1 ½ t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, cocoa, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Chocolate cupcakes The hummingbird bakery cookbook
22. 1 cup flour a scant ¾ cup sugar 1 ½ t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Vanilla cupcakes The hummingbird bakery cookbook
23. 120g flour 140g sugar 1.5 t baking powder 40g unsalted butter 120ml milk 1 egg 0.25 t pure vanilla extract Preheat oven to 170°C. Put the flour, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Vanilla cupcakes 1. Convert units The hummingbird bakery cookbook
24. 120g flour 140g sugar 1.5 t baking powder 40g butter 120ml milk 1 egg 0.25 t vanilla Beat flour, sugar, baking powder, salt, and butter until sandy. Whisk milk, egg, and vanilla. Mix half into flour mixture until smooth (use high speed). Beat in remaining half. Mix until smooth. Bake 20-25 min at 170°C. Vanilla cupcakes 2. Rely on domain knowledge The hummingbird bakery cookbook
25. Beat dry ingredients + butter until sandy. Whisk together wet ingredients. Mix half into dry until smooth (use high speed). Beat in remaining half. Mix until smooth. Bake 20-25 min at 170°C. Vanilla cupcakes 3. Use variables 120g flour 140g sugar 1.5 t baking powder 40g butter 120ml milk 1 egg 0.25 t vanilla The hummingbird bakery cookbook
26. 120g flour 140g sugar 1.5t baking powder 40g butter 120ml milk 1 egg 0.25 t vanilla Beat dry ingredients + butter until sandy. Whisk together wet ingredients. Mix half into dry until smooth (use high speed). Beat in remaining half. Mix until smooth. Bake 20-25 min at 170°C. Cupcakes 4. Extract out common code 100g flour 20g cocoa 140g sugar 1.5t baking powder 40g butter 120ml milk 1 egg 0.25 t vanilla Vanilla Chocolate
27. Cupcakes Vanilla 120 1.5 140 40 1 0.25t vanilla Chocolate 100 1.5 140 40 1 20g cocoa • 0.25t vanilla Lemon 120 1.5 140 40 1 2T lemon zest Red velvet 150 0 150 60 1 10g cocoa • 20ml red colouring • 1.5t vinegar • 0.5 t baking soda Flour Baking powder Sugar Butter Egg Extra 4. Convert to data
28. out1 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out1[[i]] <- mean(mtcars[[i]], na.rm = TRUE) } out2 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out2[[i]] <- median(mtcars[[i]], na.rm = TRUE) } For loops emphasise the objects
29. out1 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out1[[i]] <- mean(mtcars[[i]], na.rm = TRUE) } out2 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out2[[i]] <- median(mtcars[[i]], na.rm = TRUE) } For loops emphasise the objects
30. out1 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out1[[i]] <- mean(mtcars[[i]], na.rm = TRUE) } out2 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out2[[i]] <- median(mtcars[[i]], na.rm = TRUE) } Not the actions
31. library(purrr) means <- mtcars %>% map_dbl(mean) medians <- mtcars %>% map_dbl(median) Functional programming emphasises the actions
32. map_dbl <- function(x, f, ...) { out <- vector("double", length(x)) for (i in seq_along(out)) { out[i] <- f(x[[i]], ...) } out } What does map_dbl() look like? Actual implementation a little different
33. map_int <- function(x, f, ...) { out <- vector("integer", length(x)) for (i in seq_along(out)) { out[i] <- f(x[[i]], ...) } out } There are many variants:
34. map <- function(x, f, ...) { out <- vector("list", length(x)) for (i in seq_along(out)) { out[[i]] <- f(x[[i]], ...) } out } Some vary the output This is the same as lapply()!
35. map2 <- function(x, y, f, ...) { out <- vector("list", length(x)) for (i in seq_along(out)) { out[[i]] <- f(x[[i]], y[[i]]], ...) } out } Others vary the input
36. funs <- list(mean, median, sd) funs %>% map(~ mtcars %>% map_dbl(.x)) We can even think of functions as data
37. Back to gapminder
38. 40 60 80 1950 1960 1970 1980 1990 2000 year lifeExp 142 countries
39. • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • 0.0 0.2 0.4 0.6 0.8 0.00 0.25 0.50 0.75 1.00 R2 Estimated yearly increase in life expectancy continent • • • • • Africa Americas Asia Europe Oceania
40. We nested the data to get a list of data frames Country Data Afghanistan <df> Albania <df> Algeria <df> ... ... Country Year LifeEx p Afghanistan 1952 28.9 Afghanistan 1957 30.3 Afghanistan ... ... Albania 1952 55.2 Albania 1957 59.3 Albania ... ... Algeria ... ... ... ... ... nest()
41. library(dplyr) library(tidyr) library(purrr) country_model <- function(df) { lm(lifeExp ~ year1950, data = df) } gapminder %>% group_by(continent, country) %>% nest() %>% mutate( mod = data %>% map(country_model) ) Then we fitted a model to each country
42. What can we do with a list of models? Country Data Model Afghanistan <data> <lm> Albania <data> <lm> Algeria <data> <lm> ... <data> <lm>
43. Models → tidy data With broom, by David Robinson
44. What data can we extract from a model? year lifeEx p 1952 69.4 1957 70.3 1962 71.2 1967 71.5 ... ... lm(lifeExp ~ year, data = nz) R2=0.95 Intercept -307.7 Slope 0.19 year resid 1952 0.70 1957 0.61 1962 0.63 1967 -0.05 ... ... glance tidy augment New Zealand
45. models <- models %>% mutate( tidy = map(model, broom::tidy), glance = map(model, broom::glance), augment = map(model, broom::augment) ) We need to do that for each model
46. Which gives us: Country Data Model Glance Tidy Augment Afghanistan <df> <lm> <df> <df> <df> Albania <df> <lm> <df> <df> <df> Algeria <df> <lm> <df> <df> <df> ... ... ... ... ... ...
47. Unnest lets us go back to a regular data frame Country Data Afghanistan <df> Albania <df> Algeria <df> ... ... Country Year LifeEx p Afghanistan 1952 28.9 Afghanistan 1957 30.3 Afghanistan ... ... Albania 1952 55.2 Albania 1957 59.3 Albania ... ... Algeria ... ... ... ... ... nest() unnest()
48. Demo
49. Conclusion
50. 1. Store related objects in 
 list-columns. 2. Learn FP so you can focus on verbs, not objects. 3. Use broom to convert models to tidy data.
51. Data frames Lists dplyr purrr tidyr Models broom Workflow replaces many uses of ldply()/dlply() (plyr) and do() + rowwise() (dplyr) http://r4ds.had.co.nz/
52. This work is licensed under the 
 Creative Commons Attribution-Noncommercial 3.0 
 United States License. To view a copy of this license, visit 
 http://creativecommons.org/licenses/by-nc/3.0/us/