Tidy evaluation: programming with ggplot2 and dplyr

Learn how to program with tidyverse functions that "automatically quote" their input

1. Hadley Wickham 
 @hadleywickham
 Chief Scientist, RStudio Tidy evaluation: Programming with ggplot2 and dplyr March 2018
2. Writing functions
3. (df$a - min(df$a)) / (max(df$a) - min(df$a)) (df$b - min(df$b)) / (max(df$b) - min(df$b)) (df$c - min(df$c)) / (max(df$c) - min(df$c)) (df$d - min(df$d)) / (max(df$d) - min(df$d)) Rule of three: make a function if you’ve copy-pasted threes times
4. (df$a - min(df$a)) / (max(df$a) - min(df$a)) (df$b - min(df$b)) / (max(df$b) - min(df$b)) (df$c - min(df$c)) / (max(df$c) - min(df$c)) (df$d - min(df$d)) / (max(df$d) - min(df$d)) First, identify the parts that might change
5. (df$a - min(df$a)) / (max(df$a) - min(df$a)) (df$b - min(df$b)) / (max(df$b) - min(df$b)) (df$c - min(df$c)) / (max(df$c) - min(df$c)) (df$d - min(df$d)) / (max(df$d) - min(df$d)) Then give them names x x x x
6. rescale01 <- function(x) { } Make the function template
7. rescale01 <- function(x) { (df$a - min(df$a)) / (max(df$a) - min(df$a)) } Then copy in one example
8. rescale01 <- function(x) { (x - min(x)) / (max(x) - min(x)) } And use the variable
9. rescale01 <- function(x) { rng <- range(x) (x - rng[1]) / (rng[2] - rng[1])) } And maybe refactor a little
10. rescale01 <- function(x) { rng <- range(x, na.rm = TRUE, finite = TRUE) (x - rng[1]) / (rng[2] - rng[1])) } And handle more cases
11. Motivation
12. df %>% group_by(x1) %>% summarise(mean = mean(y1)) df %>% group_by(x2) %>% summarise(mean = mean(y2)) df %>% group_by(x3) %>% summarise(mean = mean(y3)) df %>% group_by(x4) %>% summarise(mean = mean(y4)) Let’s try with some dplyr code
13. df %>% group_by(x1) %>% summarise(mean = mean(y1)) df %>% group_by(x2) %>% summarise(mean = mean(y2)) df %>% group_by(x3) %>% summarise(mean = mean(y3)) df %>% group_by(x4) %>% summarise(mean = mean(y4)) First identify the parts that change
14. df %>% group_by(x1) %>% summarise(mean = mean(y1)) df %>% group_by(x2) %>% summarise(mean = mean(y2)) df %>% group_by(x3) %>% summarise(mean = mean(y3)) df %>% group_by(x4) %>% summarise(mean = mean(y4)) Then give them names summary_var group_var df
15. grouped_mean <- function(df, group_var, summary_var) { df %>% group_by(group_var) %>% summarise(mean = mean(summary_var)) } Now make a function
16. grouped_mean <- function(df, group_var, summary_var) { df %>% group_by(group_var) %>% summarise(mean = mean(summary_var)) } grouped_mean(mtcars, cyl, mpg) #> Error: Column `group_var` is unknown It doesn’t work
17. Vocabulary
18. (x - min(x)) / (max(x) - min(x)) mtcars %>% group_by(cyl) %>% summarise(mean = mean(mpg)) We need some new vocabulary Evaluated using usual R rules Automatically quoted and evaluated in a “non-standard” way
19. df <- data.frame( y = 1, var = 2 ) df$y var <- "y" df$var You’re already familiar with this idea Predict the output!
20. df <- data.frame( y = 1, var = 2 ) df$y #> [1] 1 var <- "y" df$var #> [1] 2 $ automatically quotes the variable name
21. df <- data.frame( y = 1, var = 2 ) var <- "y" df[[var]] #> [1] 1 If you want refer indirectly, must use [[ instead
22. Quoted Evaluated Direct df$y ??? Indirect ??? var <- "y"
 df[[var]]
23. Quoted Evaluated Direct df$y df[["y"]] Indirect ??? var <- "y"
 df[[var]]
24. Quoted Evaluated Direct df$y df[["y"]] Indirect var <- "y"
 df[[var]]
25. library(MASS) mtcars2 <- subset(mtcars, cyl == 4) with(mtcars2, sum(vs)) sum(mtcars2$am) rm(mtcars2) Identify which arguments are auto-quoted
26. library(MASS) #> Works MASS #> Error: object 'MASS' not found # -> The 1st argument of library() is quoted Can’t tell? Try running the code
27. subset(mtcars, cyl == 4) #> Works cyl == 4 #> Error: object 'cyl' not found # -> The 2nd argument of subset() is quoted Can’t tell? Try running the code
28. library(MASS) mtcars2 <- subset(mtcars, cyl == 4) with(mtcars2, sum(vs)) sum(mtcars2$am) rm(mtcars2) You can now identify the quoted arguments
29. Base R has 3 primary ways to “unquote” Quoted/Direct Evaluated/Indirect df$y x <- "y"
 df[[x]] library(MASS) x <- "MASS"
 library(x, character.only = TRUE) rm(mtcars) x <- "mtcars"
 rm(list = x)
30. library(tidyverse) mtcars %>% pull(am) by_cyl <- mtcars %>% group_by(cyl) %>% summarise(mean = mean(mpg)) ggplot(by_cyl, aes(cyl, mpg)) + geom_point() Identify which arguments are auto-quoted
31. library(tidyverse) mtcars %>% pull(am) by_cyl <- mtcars %>% group_by(cyl) %>% summarise(mean = mean(mpg)) ggplot(by_cyl, aes(cyl, mpg)) + geom_point() Identify which arguments are auto-quoted
32. Quoted Evaluated Tidy Direct df$y df[["y"]] pull(df, y) Indirect var <- "y"
 df[[var]] ???
33. Quoted Evaluated Tidy Direct df$y df[["y"]] pull(df, y) Indirect var <- "y"
 df[[var]] var <- quo(y)
 pull(df, !!var)
34. x_var <- quo(cyl) y_var <- quo(mpg) by_cyl <- mtcars %>% group_by(!!x_var) %>% summarise(mean = mean(!!y_var)) ggplot(by_cyl, aes(!!x_var, !!y_var)) + geom_point() Everywhere in the tidyverse uses !! to unquote Pronounced bang-bang
35. Wrapping quoting functions
36. df %>% group_by(x1) %>% summarise(mean = mean(y1)) df %>% group_by(x2) %>% summarise(mean = mean(y2)) df %>% group_by(x3) %>% summarise(mean = mean(y3)) df %>% group_by(x4) %>% summarise(mean = mean(y4)) New: Identify quoted vs. evaluated arguments
37. df %>% group_by(x1) %>% summarise(mean = mean(y1)) df %>% group_by(x2) %>% summarise(mean = mean(y2)) df %>% group_by(x3) %>% summarise(mean = mean(y3)) df %>% group_by(x4) %>% summarise(mean = mean(y4)) New: Identify quoted vs. evaluated arguments
38. df %>% group_by(x1) %>% summarise(mean = mean(y1)) df %>% group_by(x2) %>% summarise(mean = mean(y2)) df %>% group_by(x3) %>% summarise(mean = mean(y3)) df %>% group_by(x4) %>% summarise(mean = mean(y4)) Then identify the parts that could change
39. df %>% group_by(x1) %>% summarise(mean = mean(y1)) df %>% group_by(x2) %>% summarise(mean = mean(y2)) df %>% group_by(x3) %>% summarise(mean = mean(y3)) df %>% group_by(x4) %>% summarise(mean = mean(y4)) These become the function arguments summary_var group_var df
40. grouped_mean <- function(df, group_var, summary_var) { data %>% group_by(group_var) %>% summarise(mean = mean(summary_var)) } Next write the function template & identify quoted arguments
41. grouped_mean <- function(df, group_var, summary_var) { group_var <- enquo(group_var) summary_var <- enquo(summary_var) data %>% group_by(group_var) %>% summarise(mean = mean(summary_var)) } New: Wrap every quoted argument in enquo()
42. grouped_mean <- function(df, group_var, summary_var) { group_var <- enquo(group_var) summary_var <- enquo(summary_var) data %>% group_by(!!group_var) %>% summarise(mean = mean(!!summary_var)) } New: And then unquote with !!
43. Is it worth it?
44. filter(diamonds, x > 0 & y > 0 & z > 0) # vs diamonds[ diamonds$x > 0 & diamonds$y > 0 & diamonds$z > 0, ] It saves a lot of typing
45. filter(diamonds, x > 0 & y > 0 & z > 0) # vs diamonds[ diamonds[["x"]] > 0 & diamonds[["y"]] > 0 & diamonds[["z"]] > 0, ] It saves a lot of typing
46. mtcars_db %>% filter(cyl > 2) %>% select(mpg:hp) %>% head(10) %>% show_query() #> SELECT `mpg`, `cyl`, `disp`, `hp` #> FROM `mtcars` #> WHERE (`cyl` > 2.0) #> LIMIT 10 And makes it possible to translate to other languages
47. 1. R code is a tree 2. Unquoting builds trees 3. Environments map 
 names to values Now for some theory
48. R code is a tree
49. f x "y" 1 f(x, "y", 1)
50. f x "y" 1 A function call First child = function Other children = arguments
51. More complex calls have multiple levels f "y" 1 f(g(x), "y", 1) x g
52. Every expression has a tree y <- x * 10 <- y 10 * x
53. Because every expression can be rewritten `<-`(y, `*`(x, 10)) <- y 10 * x
54. > lobstr::ast(if(x > 5) y + 1) █#`if` $#█#`>` % $#x % &#5 &#█#`+` $#y &#1 You can see this yourself with lobstr::ast()
55. Unquoting builds trees
56. library(rlang) expr(y + 1) #> y + 1 expr() captures your expression
57. x1 <- expr(a + b) expr(f(!!x1, z)) #> f(a + b, z) # !! is called the unquoting operator # And is pronounced bang-bang Unquoting allows you to build your own trees
58. + a b x1 <- expr(a + b) f z expr(f(!!x1, z)) x1
59. + a b f z expr(f(!!x1, z))
60. + a b f z expr(f(!!x1, z))
61. ex1 <- expr(x + y) ex2 <- expr(!!ex1 + z) ex3 <- expr(1 / !!ex1) Predict what this code will return
62. ex1 <- expr(x + y) # x + y ex2 <- expr(!!ex1 + z) ex3 <- expr(1 / !!ex1) Predict what this code will return
63. ex1 <- expr(x + y) # x + y ex2 <- expr(!!ex1 + z) # x + y + z ex3 <- expr(1 / !!ex1) Predict what this code will return
64. ex1 <- expr(x + y) # x + y ex2 <- expr(!!ex1 + z) # x + y + z ex3 <- expr(1 / !!ex1) # 1 / (x + y) # Not 1 / x + y Predict what this code will return
65. # expr() quotes your expression f1 <- function(z) expr(z) f1(a + b) #> z # enexpr() quotes user’s expression f2 <- function(z) enexpr(z) f2(x + y) #> x + y enexpr() lets you capture user expressions
66. Environments map 
 names to values
67. my_mutate <- function(df, var) { n <- 10 var <- enexpr(var) mutate(df, y = !!var) } df <- tibble(x = 1) n <- 100 my_mutate(df, x + n) #> x y #> 1 1.00 11 Capturing just expression isn’t enough
68. my_mutate <- function(df, var) { n <- 10 var <- enexpr(var) mutate(df, y = !!var) } df <- tibble(x = 1) n <- 100 my_mutate(df, x + n) #> x y #> 1 1.00 11
69. # quo() quotes your expression f1 <- function(z) quo(z) f1(a + b) #> <quosure> #> expr: ^z #> env: 0x10d3b9308 # enquo() quotes user’s expression f2 <- function(z) enquo(z) f2(x + y) #> <quosure> #> expr: ^x + y #> env: 0x10d3b9309 quo() captures expression and environment
70. Your code User’s code Expression expr(x) enenxpr(x) Expression + environment quo(x) enquo(x) Think enrich
71. my_mutate <- function(df, var) { n <- 10 var <- enquo(var) mutate(df, y = !!var) } df <- tibble(x = 1) n <- 100 my_mutate(df, x + n) #> x y #> 1 1.00 101
72. my_mutate <- function(df, var) { n <- 10 var <- enquo(var) mutate(df, y = !!var) } df <- tibble(x = 1) n <- 100 my_mutate(df, x + n) #> x y #> 1 1.00 101
73. df <- data.frame(x = 1:5, y = 5:1) filter(df, abs(x) > 1e-3) filter(df, abs(y) > 1e-3) filter(df, abs(z) > 1e-3) my_filter <- function(df, var) { var <- enquo(var) filter(df, abs(!!var) > 1e-3) } my_filter(df, x) Key pattern is to quote and unquote Quote Unquote
74. Conclusion
75. In development Tidy evaluation = principled NSE
76. df1 %>% group_by(g1) %>% summarise(mean = mean(a)) df2 %>% group_by(g2) %>% summarise(mean = mean(b)) df3 %>% group_by(g3) %>% summarise(mean = mean(c)) df4 %>% group_by(g4) %>% summarise(mean = mean(d)) Tidy eval lets you reduce duplication df1 %>% grouped_mean(g1, a) df2 %>% grouped_mean(g2, b) df3 %>% grouped_mean(g3, c) df4 %>% grouped_mean(g4, d)
77. Code is a tree f y !!x `-` 1 Build trees with unquoting Quote to capture code + env enquo() Learn more https://adv-r.hadley.nz/expressions.html https://adv-r.hadley.nz/quasiquotation.html https://adv-r.hadley.nz/evaluation.html WIP 2nd ed
79. This work is licensed as Creative Commons
 Attribution-ShareAlike 4.0 
 International To view a copy of this license, visit 
 https://creativecommons.org/licenses/by-sa/4.0/