Keyword Extraction with Ruby

Transcript

1. Keyword Extraction ... or why NLP matters!
2. Konstantin Tennhard Ruby Developer at flinc Hi, I‘m…
3. Konstantin Tennhard Ruby Developer at flinc Hi, I‘m… Ruby enthusiast Bartender Computer Science student Photographer Mountain bike addict Computer Linguist
4. Problem What are we talking about?
5. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for.
6. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for.
7. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for.
8. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for.
9. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for.
10. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for.
11. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for.
12. The Occupy Wall Street movement began in Zuccotti Park on a glorious mid-September Saturday and, so far, many of its larger marches have taken place in the warmth of New York's Indian summer. But winter has been looming, and on Saturday, just a couple days before Halloween, the protesters got a preview of what they're in for. Named Entity Adjective Noun
13. Cooccurrence Don't be anxious ... ... it's not that bad
14. ... protesters got a preview ... Word window Cooccurrence
15. What do we want?
16. What do we want? Well, how about extracting nouns and adjectives that cooccur in word windows of a certain size? Wouldn't that be something.
17. But ... ... let's talk about something else first.
18. Natural Language Processing ... and the sad state of Ruby libraries
19. NLP What is it?
20. NLP Language analysis Compute intensive tasks Large data sets Machine learning State-of-the-art Science
21. NLP tasks What is NLP good for?
22. NLP tasks Keyword extraction Opinion mining Text summarization Text classification Machine Translation Named Entity Recognition
23. Sounds fancy, doesn't it?
24. But ... ... before you can do the fancy stuff, you need to do a couple of other things first!
25. NLP Tasks POS Tagging Stemming Chunking Segmentation Lemmatization Tokenization The cranky ones.
26. NLP Pipeline Sentence splitting Tokenization POS tagging Stemming / Lemmatization Clean up Fancy stuff
27. Ruby Libraries Well, there isn't really much available!
28. Keyword Extraction Right,that's what we are actually talking about here.
29. Solution How can we solve this problem?
30. Three Steps Generate Cooccurrence Graph Part-of-Speech Tagging Stemming Apply Weighted PageRank Extract Nodes with Highest Rank 1 2 3
31. The algorithm is known as TextRank and has been published by Rada Mihalcea and Paul Tarau in 2004 in their Paper “TextRank: Bringing Order into Texts” http:/ /acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf
32. ... protesters got a preview ... Word window Cooccurrence
33. ... protesters/NN got/VB a/DET preview/NN ... POS Tags
34. Street Wall movement Zuccotti Park gloriou mid-Septemb Saturdai larger mani march place warmth New york Indian summer winter coupl dai Halloween protest preview Cooccurrence Graph
35. Weighted PageRank WS(Vi) = (1 d) + d · X Vj 2In(Vi ) wji P Vk 2Out(Vj ) wjk WS(Vj) We will use Lexicographer's Pointwise Mutual Information as weighting function.
36. Code Well, how do we really solve this? Show me some code!
37. Code •Simple interface •Poor performance - this is just an academic example! text = "..." # your text count = 5 # number of words to extract KeywordExtractor. extract_most_important_words(text, count)
38. Demo Yeah, it actually works!
39. Questions?
40. github.com/t6d/keyword_extractor Purely developed for academic purposes. This library is by far not production ready.
41. Thanks! W e‘re hiring! @t6d github.com/t6d me@t6d.de