Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing many models

Hadley Wickham
February 24, 2017

Managing many models

Hadley Wickham

February 24, 2017
Tweet

More Decks by Hadley Wickham

Other Decks in Science

Transcript

  1. You’ve never seen data presented like this. With the drama

    and urgency of a sportscaster, statistics guru Hans Rosling debunks myths about the so- called “developing world.” https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen
  2. • • • • • • • • • •

    • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • 0.0 0.2 0.4 0.6 0.8 0.00 0.25 0.50 0.75 1.00 R2 Estimated yearly increase in life expectancy continent • • • • • Africa Americas Asia Europe Oceania
  3. Each idea is partnered with a package 1. Nested data

    (tidyr) 2. Functional programming (purrr) 3. Models → tidy data (broom)
  4. 40 60 80 1950 1960 1970 1980 1990 2000 year

    lifeExp 142 countries Want to summarise each with a linear model
  5. Currently our data has one row per observation Country Year

    LifeEx p Afghanistan 1952 28.9 Afghanistan 1957 30.3 Afghanistan ... ... Albania 1952 55.2 Albania 1957 59.3 Albania ... ... Algeria ... ... ... ... ...
  6. More convenient to one row per group Country Data Afghanistan

    <df> Albania <df> Algeria <df> ... ... Year LifeExp 1952 28.9 1957 30.3 ... ... Year LifeExp 1952 55.2 1957 59.3 ... ... I call this a nested data frame
  7. x %>% f(y) # is the same as: f(x, y)

    gapminder %>% group_by(continent, country) %>% nest() # same as: nest(group_by(gapminder, continent, country)) Haven’t seen pipes?
  8. Each country will have an associated model Country Data Afghanistan

    <df> Albania <df> Algeria <df> ... ... lm(lifeExp ~ year1950, data = afghanistan) lm(lifeExp1950 ~ year, data = albania)
  9. Why not store that in a column too? Country Data

    Model Afghanistan <df> <lm> Albania <df> <lm> Algeria <df> <lm> ... ... ...
  10. library(dplyr) library(purrr) country_model <- function(df) { lm(lifeExp ~ year1950, data

    = df) } models <- by_country %>% mutate( mod = map(data, country_model) ) In R:
  11. 1 cup flour a scant ¾ cup sugar 1 ½

    t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Vanilla cupcakes The hummingbird bakery cookbook
  12. ¾ cup + 2T flour 2 ½ T cocoa powder

    a scant ¾ cup sugar 1 ½ t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, cocoa, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Chocolate cupcakes The hummingbird bakery cookbook
  13. ¾ cup + 2T flour 2 ½ T cocoa powder

    a scant ¾ cup sugar 1 ½ t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, cocoa, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Chocolate cupcakes The hummingbird bakery cookbook
  14. 1 cup flour a scant ¾ cup sugar 1 ½

    t baking powder 3 T unsalted butter ½ cup whole milk 1 egg ¼ t pure vanilla extract Preheat oven to 350°F. Put the flour, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Vanilla cupcakes The hummingbird bakery cookbook
  15. 120g flour 140g sugar 1.5 t baking powder 40g unsalted

    butter 120ml milk 1 egg 0.25 t pure vanilla extract Preheat oven to 170°C. Put the flour, sugar, baking powder, salt, and butter in a freestanding electric mixer with a paddle attachment and beat on slow speed until you get a sandy consistency and everything is combined. Whisk the milk, egg, and vanilla together in a pitcher, then slowly pour about half into the flour mixture, beat to combine, and turn the mixer up to high speed to get rid of any lumps. Turn the mixer down to a slower speed and slowly pour in the remaining milk mixture. Continue mixing for a couple of more minutes until the batter is smooth but do not overmix. Spoon the batter into paper cases until 2/3 full and bake in the preheated oven for 20-25 minutes, or until the cake bounces back when touched. Vanilla cupcakes 1. Convert units The hummingbird bakery cookbook
  16. 120g flour 140g sugar 1.5 t baking powder 40g butter

    120ml milk 1 egg 0.25 t vanilla Beat flour, sugar, baking powder, salt, and butter until sandy. Whisk milk, egg, and vanilla. Mix half into flour mixture until smooth (use high speed). Beat in remaining half. Mix until smooth. Bake 20-25 min at 170°C. Vanilla cupcakes 2. Rely on domain knowledge The hummingbird bakery cookbook
  17. Beat dry ingredients + butter until sandy. Whisk together wet

    ingredients. Mix half into dry until smooth (use high speed). Beat in remaining half. Mix until smooth. Bake 20-25 min at 170°C. Vanilla cupcakes 3. Use variables 120g flour 140g sugar 1.5 t baking powder 40g butter 120ml milk 1 egg 0.25 t vanilla The hummingbird bakery cookbook
  18. 120g flour 140g sugar 1.5t baking powder 40g butter 120ml

    milk 1 egg 0.25 t vanilla Beat dry ingredients + butter until sandy. Whisk together wet ingredients. Mix half into dry until smooth (use high speed). Beat in remaining half. Mix until smooth. Bake 20-25 min at 170°C. Cupcakes 4. Extract out common code 100g flour 20g cocoa 140g sugar 1.5t baking powder 40g butter 120ml milk 1 egg 0.25 t vanilla Vanilla Chocolate
  19. Cupcakes Vanilla 120 1.5 140 40 1 0.25t vanilla Chocolate

    100 1.5 140 40 1 20g cocoa • 0.25t vanilla Lemon 120 1.5 140 40 1 2T lemon zest Red velvet 150 0 150 60 1 10g cocoa • 20ml red colouring • 1.5t vinegar • 0.5 t baking soda Flour Baking powder Sugar Butter Egg Extra 4. Convert to data
  20. out1 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out1[[i]] <-

    mean(mtcars[[i]], na.rm = TRUE) } out2 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out2[[i]] <- median(mtcars[[i]], na.rm = TRUE) } For loops emphasise the objects
  21. out1 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out1[[i]] <-

    mean(mtcars[[i]], na.rm = TRUE) } out2 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out2[[i]] <- median(mtcars[[i]], na.rm = TRUE) } For loops emphasise the objects
  22. out1 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out1[[i]] <-

    mean(mtcars[[i]], na.rm = TRUE) } out2 <- vector("double", ncol(mtcars)) for(i in seq_along(mtcars)) { out2[[i]] <- median(mtcars[[i]], na.rm = TRUE) } Not the actions
  23. library(purrr) means <- mtcars %>% map_dbl(mean) medians <- mtcars %>%

    map_dbl(median) Functional programming emphasises the actions
  24. map_dbl <- function(x, f, ...) { out <- vector("double", length(x))

    for (i in seq_along(out)) { out[i] <- f(x[[i]], ...) } out } What does map_dbl() look like? Actual implementation a little different
  25. map_int <- function(x, f, ...) { out <- vector("integer", length(x))

    for (i in seq_along(out)) { out[i] <- f(x[[i]], ...) } out } There are many variants:
  26. map <- function(x, f, ...) { out <- vector("list", length(x))

    for (i in seq_along(out)) { out[[i]] <- f(x[[i]], ...) } out } Some vary the output This is the same as lapply()!
  27. map2 <- function(x, y, f, ...) { out <- vector("list",

    length(x)) for (i in seq_along(out)) { out[[i]] <- f(x[[i]], y[[i]]], ...) } out } Others vary the input
  28. funs <- list(mean, median, sd) funs %>% map(~ mtcars %>%

    map_dbl(.x)) We can even think of functions as data
  29. • • • • • • • • • •

    • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • 0.0 0.2 0.4 0.6 0.8 0.00 0.25 0.50 0.75 1.00 R2 Estimated yearly increase in life expectancy continent • • • • • Africa Americas Asia Europe Oceania
  30. We nested the data to get a list of data

    frames Country Data Afghanistan <df> Albania <df> Algeria <df> ... ... Country Year LifeEx p Afghanistan 1952 28.9 Afghanistan 1957 30.3 Afghanistan ... ... Albania 1952 55.2 Albania 1957 59.3 Albania ... ... Algeria ... ... ... ... ... nest()
  31. library(dplyr) library(tidyr) library(purrr) country_model <- function(df) { lm(lifeExp ~ year1950,

    data = df) } gapminder %>% group_by(continent, country) %>% nest() %>% mutate( mod = data %>% map(country_model) ) Then we fitted a model to each country
  32. What can we do with a list of models? Country

    Data Model Afghanistan <data> <lm> Albania <data> <lm> Algeria <data> <lm> ... <data> <lm>
  33. What data can we extract from a model? year lifeEx

    p 1952 69.4 1957 70.3 1962 71.2 1967 71.5 ... ... lm(lifeExp ~ year, data = nz) R2=0.95 Intercept -307.7 Slope 0.19 year resid 1952 0.70 1957 0.61 1962 0.63 1967 -0.05 ... ... glance tidy augment New Zealand
  34. models <- models %>% mutate( tidy = map(model, broom::tidy), glance

    = map(model, broom::glance), augment = map(model, broom::augment) ) We need to do that for each model
  35. Which gives us: Country Data Model Glance Tidy Augment Afghanistan

    <df> <lm> <df> <df> <df> Albania <df> <lm> <df> <df> <df> Algeria <df> <lm> <df> <df> <df> ... ... ... ... ... ...
  36. Unnest lets us go back to a regular data frame

    Country Data Afghanistan <df> Albania <df> Algeria <df> ... ... Country Year LifeEx p Afghanistan 1952 28.9 Afghanistan 1957 30.3 Afghanistan ... ... Albania 1952 55.2 Albania 1957 59.3 Albania ... ... Algeria ... ... ... ... ... nest() unnest()
  37. 1. Store related objects in 
 list-columns. 2. Learn FP

    so you can focus on verbs, not objects. 3. Use broom to convert models to tidy data.
  38. Data frames Lists dplyr purrr tidyr Models broom Workflow replaces

    many uses of ldply()/dlply() (plyr) and do() + rowwise() (dplyr) http://r4ds.had.co.nz/
  39. This work is licensed under the 
 Creative Commons Attribution-Noncommercial

    3.0 
 United States License. To view a copy of this license, visit 
 http://creativecommons.org/licenses/by-nc/3.0/us/