Text Mining, the Tidy Way

January 2017 talk at rstudio::conf

Transcript

1. THE TIDY WAY Julia Silge @juliasilge http://juliasilge.com/ Text Mining
2. TIDY DATA PRINCIPLES CAN MAKE TEXT MINING EASIER AND MORE EFFECTIVE
3. https://github.com/juliasilge/tidytext
4. HOW IS TEXT TYPICALLY STORED? • Raw string • Corpus • Document-term matrix
5. What do we mean by tidy text?
6. > text <- c("Because I could not stop for Death -", "He kindly stopped for me -", "The Carriage held but just Ourselves -", "and Immortality") > > text ## [1] "Because I could not stop for Death -" "He kindly stopped for me -" ## [3] "The Carriage held but just Ourselves -" "and Immortality" What do we mean by tidy text?
7. > library(dplyr) > text_df <- data_frame(line = 1:4, text = text) > > text_df ## # A tibble: 4 × 2 ## line text ## <int> <chr> ## 1 1 Because I could not stop for Death - ## 2 2 He kindly stopped for me - ## 3 3 The Carriage held but just Ourselves - ## 4 4 and Immortality What do we mean by tidy text?
8. > library(tidytext) > text_df %>% unnest_tokens(word, text) ## # A tibble: 20 × 2 ## line word ## <int> <chr> ## 1 1 because ## 2 1 i ## 3 1 could ## 4 1 not ## 5 1 stop ## 6 1 for ## 7 1 death ## 8 2 he ## 9 2 kindly ## 10 2 stopped ## # ... with 10 more rows What do we mean by tidy text?
9. > library(tidytext) > text_df %>% unnest_tokens(word, text) ## # A tibble: 20 × 2 ## line word ## <int> <chr> ## 1 1 because ## 2 1 i ## 3 1 could ## 4 1 not ## 5 1 stop ## 6 1 for ## 7 1 death ## 8 2 he ## 9 2 kindly ## 10 2 stopped ## # ... with 10 more rows • Other columns have been retained • Punctuation has been stripped • Words have been converted to lowercase What do we mean by tidy text?
11. Tidying the works of Jane Austen > library(janeaustenr) > library(dplyr) > library(stringr) > > original_books <- austen_books() %>% group_by(book) %>% mutate(linenumber = row_number(), chapter = cumsum(str_detect(text, regex("^chapter [\\divxlc]", ignore_case = TRUE)))) %>% ungroup()
12. Tidying the works of Jane Austen > original_books # A tibble: 73,422 × 4 text book linenumber chapter <chr> <fctr> <int> <int> 1 SENSE AND SENSIBILITY Sense & Sensibility 1 0 2 Sense & Sensibility 2 0 3 by Jane Austen Sense & Sensibility 3 0 4 Sense & Sensibility 4 0 5 (1811) Sense & Sensibility 5 0 6 Sense & Sensibility 6 0 7 Sense & Sensibility 7 0 8 Sense & Sensibility 8 0 9 Sense & Sensibility 9 0 10 CHAPTER 1 Sense & Sensibility 10 1 # ... with 73,412 more rows
13. Tidying the works of Jane Austen > tidy_books <- original_books %>% unnest_tokens(word, text) > > tidy_books # A tibble: 725,054 × 4 book linenumber chapter word <fctr> <int> <int> <chr> 1 Sense & Sensibility 1 0 sense 2 Sense & Sensibility 1 0 and 3 Sense & Sensibility 1 0 sensibility 4 Sense & Sensibility 3 0 by 5 Sense & Sensibility 3 0 jane 6 Sense & Sensibility 3 0 austen 7 Sense & Sensibility 5 0 1811 8 Sense & Sensibility 10 1 chapter 9 Sense & Sensibility 10 1 1 10 Sense & Sensibility 13 1 the # ... with 725,044 more rows
14. WHAT NEXT? OUR TEXT IS TIDY NOW
15. REMOVING STOP WORDS > data(stop_words) > > tidy_books <- tidy_books %>% anti_join(stop_words) > > tidy_books %>% count(word, sort = TRUE)
17. Sentiment analysis > get_sentiments("afinn") # A tibble: 2,476 × 2 word score <chr> <int> 1 abandon -2 2 abandoned -2 3 abandons -2 4 abducted -2 5 abduction -2 6 abductions -2 7 abhor -3 8 abhorred -3 9 abhorrent -3 10 abhors -3 # ... with 2,466 more rows > get_sentiments("bing") # A tibble: 6,788 × 2 word sentiment <chr> <chr> 1 2-faced negative 2 2-faces negative 3 a+ positive 4 abnormal negative 5 abolish negative 6 abominable negative 7 abominably negative 8 abominate negative 9 abomination negative 10 abort negative # ... with 6,778 more rows > get_sentiments("nrc") # A tibble: 13,901 × 2 word sentiment <chr> <chr> 1 abacus trust 2 abandon fear 3 abandon negative 4 abandon sadness 5 abandoned anger 6 abandoned fear 7 abandoned negative 8 abandoned sadness 9 abandonment anger 10 abandonment fear # ... with 13,891 more rows
18. > library(tidyr) > > janeaustensentiment <- tidy_books %>% inner_join(get_sentiments("bing")) %>% count(book, index = linenumber %/% 100, sentiment) %>% spread(sentiment, n, fill = 0) %>% mutate(sentiment = positive - negative) Sentiment analysis
20. TERM FREQUENCY INVERSE DOCUMENT FREQUENCY What is a document about?
21. TF-IDF > book_words <- austen_books() %>% unnest_tokens(word, text) %>% count(book, word, sort = TRUE) %>% ungroup() > > total_words <- book_words %>% group_by(book) %>% summarize(total = sum(n)) > > book_words <- left_join(book_words, total_words)
22. > book_words # A tibble: 40,379 × 4 book word n total <fctr> <chr> <int> <int> 1 Mansfield Park the 6206 160460 2 Mansfield Park to 5475 160460 3 Mansfield Park and 5438 160460 4 Emma to 5239 160996 5 Emma the 5201 160996 6 Emma and 4896 160996 7 Mansfield Park of 4778 160460 8 Pride & Prejudice the 4331 122204 9 Emma of 4291 160996 10 Pride & Prejudice to 4162 122204 # ... with 40,369 more rows TF-IDF
24. TF-IDF > book_words <- book_words %>% bind_tf_idf(word, book, n) > book_words # A tibble: 40,379 × 7 book word n total tf idf tf_idf <fctr> <chr> <int> <int> <dbl> <dbl> <dbl> 1 Mansfield Park the 6206 160460 0.03867631 0 0 2 Mansfield Park to 5475 160460 0.03412065 0 0 3 Mansfield Park and 5438 160460 0.03389007 0 0 4 Emma to 5239 160996 0.03254118 0 0 5 Emma the 5201 160996 0.03230515 0 0 6 Emma and 4896 160996 0.03041069 0 0 7 Mansfield Park of 4778 160460 0.02977689 0 0 8 Pride & Prejudice the 4331 122204 0.03544074 0 0 9 Emma of 4291 160996 0.02665284 0 0 10 Pride & Prejudice to 4162 122204 0.03405780 0 0 # ... with 40,369 more rows
25. > book_words %>% + select(-total) %>% + arrange(desc(tf_idf)) # A tibble: 40,379 × 6 book word n tf idf tf_idf <fctr> <chr> <int> <dbl> <dbl> <dbl> 1 Sense & Sensibility elinor 623 0.005193528 1.791759 0.009305552 2 Sense & Sensibility marianne 492 0.004101470 1.791759 0.007348847 3 Mansfield Park crawford 493 0.003072417 1.791759 0.005505032 4 Pride & Prejudice darcy 373 0.003052273 1.791759 0.005468939 5 Persuasion elliot 254 0.003036207 1.791759 0.005440153 6 Emma emma 786 0.004882109 1.098612 0.005363545 7 Northanger Abbey tilney 196 0.002519928 1.791759 0.004515105 8 Emma weston 389 0.002416209 1.791759 0.004329266 9 Pride & Prejudice bennet 294 0.002405813 1.791759 0.004310639 10 Persuasion wentworth 191 0.002283132 1.791759 0.004090824 # ... with 40,369 more rows TF-IDF
27. • As part of the NASA Datanauts program, I am working on a project to understand NASA datasets • Metadata includes title, description, keywords, etc
29. TAKING TIDY TEXT TO THE NEXT LEVEL n-grams, networks, & negation
30. TAKING TIDY TEXT TO THE NEXT LEVEL tidying & casting
31. http://tidytextmining.com/
32. Julia Silge @juliasilge http://juliasilge.com/ Thank You