Frotiers of Natural Language Processing

Recruit Technologies Open Lab #01 （テーマ: 自然言語処理）で話したときに使ったスライドです。

https://atnd.org/events/64383

Transcript

1. ࣗવݴޠॲཧͷ৽ల։ 2015೥4݄21೔ ट౎େֶ౦ژ γεςϜσβΠϯֶ෦ খொक
2. ࣗݾ঺հ: খொकʢ͜·ͪ·΋Δʣ 2 ß 2005.03 ౦ژେֶڭཆֶ෦جૅՊֶՊ Պֶ࢙ɾՊֶ఩ֶ෼Պଔۀ ß 2010.03 ಸྑઌ୺େɾത࢜ޙظ՝ఔमྃ ത࢜ʢ޻ֶʣ ઐ໳: ࣗવݴޠॲཧ ß 2010.04ʙ2013.03 ಸྑઌ୺େ ॿڭʢদຊ༟࣏ݚڀࣨʣ ß 2013.04〜 ट౎େֶ౦ژ ।ڭतʢࣗવݴޠॲཧݚڀࣨʣ
3. ຊ೔ͷ໨࣍ ß ਂ૚ֶश͕ࣗવݴޠॲཧʹ༩͑ΔΠϯύ Ϋτ ß ࣗવݴޠॲཧͷ৽ͨͳൃల 3
4. ਂ૚ֶशʢdeep learningʣ ß ෳ਺ϨΠϠʔͷχϡʔϥϧωοτϫʔΫ ʹΑͬͯෳࡶͳϞσϧΛֶश͢Δ࢓૊Έ ß ༷ʑͳύλʔϯೝࣝλεΫͰେ෯ͳੑೳ ޲্Λୡ੒͠ɺGoogle, Facebook, Microsoft, Baidu ͳͲ͞·͟·ͳاۀ͕͜ ͧͬͯݚڀ 4
5. Lee et al., ICML 2009. 5
6. ਂ૚ֶशͷ௕ॴ ß ૉੑ޻ֶʢfeature engineeringʣ͕ෆཁɻ ϥϕϧͳ͠σʔλ͔Βࣗಈతʹ༗ޮͳૉ ੑͷ૊Έ߹Θֶ͕ͤशՄೳɻ →ϋΠύʔύϥϝʔλ͸ଘࡏ ß σʔλ͔ΒେҬతͳදݱֶशʢdistributed representationʣ͕Մೳ →ΫϥελϦϯά͸ہॴతͳදݱֶश 6
7. χϡʔϥϧωοτϫʔΫ ͷϒϨΠΫεϧʔ ß Hinton et al., A Fast Learning Algorithm for Deep Belief Nets, Neural Computing, 2006. ß χϡʔϥϧωοτϫʔΫ͸1950೥୅͔Β ͕͋ͬͨɺදݱೳྗ͕ߴ͗ͯ͢ʢσʔλ ྔʹରͯ͠ʣաֶशʹͳΓ΍͔ͬͨ͢ɻ →૚͝ͱʹֶशΛߦ͍ɺෳ਺૚ΛॏͶΔ ͜ͱͰաֶशͷ໰୊͕ղܾͰ͖ͨʂ 7
8. ࠶ؼతχϡʔϥϧωοτϫʔΫ Λ༻͍ͨը૾ೝࣝͱߏจղੳ 8 • Parsing Natural Scenes and Natural Language with Recursive Neural Networks, Socher et al., ICML 2011. • ྡ઀͢Δը૾ྖҬɾ୯ ޠ͔Β࠶ؼతʹߏ଄Λ ೝࣝ͢Δ →Staford Parser ʹ౷ ߹ (ACL 2013)
9. ࠶ؼతχϡʔϥϧωοτϫʔΫͰ ϑϨʔζͷײ৘ۃੑ෼ྨ΋࣮ݱ 9 • Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, Socher et al., EMNLP 2013.
10. Socher et al. (NIPS 2011): ୯ޠϕΫ τϧ͔ΒจͷҙຯΛ࠶ؼతʹܭࢉ 10
11. ϦΧϨϯτχϡʔϥϧωοτ ϫʔΫͰແݶ௕ͷจ຺ΛߟྀՄೳ 11 • Recurrent Neural Network based Language Model, Mikolov et al., InterSpeech 2010. →աڈͷཤྺΛߟྀͯ͠ݱࡏͷ୯ޠΛ༧ଌ͢ΔϞσϧ
12. ػց຋༁΋ܥྻ͔ΒܥྻΛੜ੒͢ ΔϞσϧͱͯ͠ਂ૚ֶशͰѻ͑Δ ß Sequence to Sequence Learning with Neural Networks, Sutskever et al., NIPS 2014. →LSTM (Long-Short Term Memory) Λ2ͭ༻ ͍ɺೖྗܥྻΛݻఆ௕ͷϕΫτϧʹม׵ ͠ɺͦͷϕΫτϧ͔Βग़ྗܥྻΛੜ੒ 12
13. จࣈ͚͔ͩΒਂ૚ֶशͰςΩετ ෼ྨ΍ϓϩάϥϜ͕Ͱ͖ͯ͠·͏ ß Text Understanding from Scratch, Zhang and LeCun, arXiv 2015. →จࣈ͚͔ͩΒதӳͷςΩετ෼ྨثΛֶश ß Learning to Execute, Zaremba and Sutskever, arXiv 2015. →RNNͱLTSM͚͔ͩΒPythonϓϩάϥϜΛ ʮֶशʯ࣮ͯ͠ߦ 13
14. ਂ૚ֶशΛ࢖ͬͯϚϧνϞʔμϧ ͳೖग़ྗΛࣗવʹ౷߹ ß ը૾͚͔ͩΒΩϟϓγϣϯΛੜ੒ http://deeplearning.cs.toronto.edu/i2t http://googleresearch.blogspot.jp/2014/11/a-picture-is- worth-thousand-coherent.html 14
15. ຊ೔ͷ໨࣍ ß ਂ૚ֶश͕ࣗવݴޠॲཧʹ༩͑ΔΠϯύ Ϋτ ß ࣗવݴޠॲཧͷ৽ͨͳൃల 15
16. ࣗવݴޠॲཧͷ੒ޭ ß ࣝผϞσϧ Þ λά͖ͭίʔύεΛ༻ҙͯ͠ڭࢣ͋Γֶश Þ ܗଶૉղੳɺݻ༗දݱೝࣝɺߏจղੳɺetc ß ࠷దԽ໰୊ Þ ϥϯΩϯά΍૊Έ߹Θͤ࠷దԽ໰୊ʹఆࣜԽ Þ ΢Σϒݕࡧɺػց຋༁ɺจॻཁ໿ɺetc 16
17. ੈքΛڍ͛ͨଟݴޠॲཧͷͨΊͷ ཁૉٕज़ͷݚڀ։ൃ ß CoNLL: Conference on Natural Language Learning ͷڞ௨λεΫʢຖ೥։࠵ʣ Þ 2012: ଟݴޠஊ࿩ղੳ Þ 2009: ଟݴޠߏจɾҙຯղੳ Þ 2006, 2007: ଟݴޠߏจղੳ ß ಉ͡ΞϧΰϦζϜΛෳ਺ͷݴޠʹద༻͠ɺ ݴޠʹΑΒͳ͍ղੳख๏Λ୳ٻ 17
18. Java ʹΑΔଟݴޠॲཧπʔϧ ʢ঎༻ͷϞσϧϥΠηϯε͸ཁަবʣ ß Stanford CoreNLP (Java) Þ ӳޠɺεϖΠϯޠɺதࠃޠͷܗଶૉղੳɾݻ ༗දݱೝࣝɾߏจղੳɾஊ࿩ղੳπʔϧ ß Apache OpenNLP (Java) Þ σϯϚʔΫޠɺυΠπޠɺӳޠɺεϖΠϯޠɺ ΦϥϯμޠɺϙϧτΨϧޠɺε΢Σʔσϯޠ Λαϙʔτ ß LingPipe (Java) Þ ӳޠʢ඼ࢺ෇༩ɾݻ༗දݱநग़ʣɾதࠃޠ ʢ୯ޠ෼ׂʣͷϞσϧ 18
19. ଟݴޠܗଶૉղੳͷͨΊͷ λά࢓༷ͱίʔύε ß A Universal Part-of-Speech Tagset, Petrov et al., LREC 2012. Þ 22ݴޠ: ӳޠɺதࠃޠɺ೔ຊޠɺؖࠃޠɺetc Þ ଟݴޠɾݴޠΛ·͍ͨͩߏจղੳͷݚڀ։ൃ ͷͨΊʹɺ·ͣ඼ࢺΛҰ؏͚͍ͯͭͨ͠ Þ ೔ຊޠ͸೔ຊޠॻ͖ݴ༿ۉߧίʔύε ʢBCCWJʣͷ୹୯Ґʹ४ڌͨ͠୯ޠ෼ׂ 19
20. ଟݴޠ܎Γड͚ղੳͷͨΊͷ λά࢓༷ͱίʔύε ß Universal Dependency Annotation for Multilingual Parsing, McDonald et al., ACL 2013. Þ υΠπޠɾӳޠɾε΢ΣʔσϯޠɾεϖΠϯޠɾ ϑϥϯεޠɾؖࠃޠɾetc Þ ೔ຊޠ Universal Dependencies ͷࢼҊ, ۚࢁΒ, ݴ ޠॲཧֶձ೥࣍େձ 2015. 20
21. ࣗવݴޠॲཧͷཁૉٕज़͸੒ख़ظ ཁૉٕज़ ਫ਼౓ ܗଶૉղੳʢ෼͔ͪॻ͖ʣ 99% ߏจղੳʢ܎Γड͚ʣ 90% ҙຯղੳʢड़ޠ߲ߏ଄ʣ 60% ஊ࿩ղੳʢจΛ௒͑ͨؔ܎ʣ 30% 21 ղ ੳ ͷ ྲྀ Ε จਖ਼ղ཰ʹ͢Δͱ5ׂ ཁૉٕज़୯ମͰͷਫ਼౓޲্͸಄ଧͪ ᶃΞϓϦέʔγϣϯʹଈͨ͠ੑೳධՁͷඞཁ ᶄਫ਼౓Ҏ֎ͷ໘ͰͷΞϐʔϧ
22. ӳޠͷݴޠղੳ΋৽ฉهࣄ͔Β ΢ΣϒςΩετ΁ ß Workshop on Syntactic Analysis on Non- Canonical Language (SANCL 2012) ß Google English Web Treebank (2012) Þ ΢ΣϒςΩετʢϒϩάɺχϡʔεάϧʔϓɺ ϝʔϧɺϦϏϡʔɺQA ʣʹܗଶૉɾߏจʢ܎ Γड͚ʣ৘ใΛλά͚ͮ 22
23. ΢ΣϒςΩετ΋ɺΑΓ೉͍͠ Ϣʔβੜ੒ܕͷςΩετղੳ΁ ß Tweet NLPʢӳޠͷΈʣ http://www.ark.cs.cmu.edu/TweetNLP/ Þ Twokenizer: ܗଶૉղੳ Þ Tweeboparser: ܎Γड͚ղੳ Þ Tweebank: Twitter ίʔύε Þ Twitter Word Clusters: ୯ޠΫϥελ 23
24. ฼ޠ࿩ऀ͕ॻ͍ͨจ๏తʹਖ਼͍͠ςΩ ετ͔ΒɺݴޠֶशऀͷςΩετ΁ ß 2011೥લޙ͔Βຖ೥ͷΑ͏ʹӳޠֶशऀ ͷ࡞จͷจ๏ޡΓగਖ਼ڞ௨λεΫ͕։࠵ Þ Helping Our Own (HOO) 2011, 2012 Þ CoNLL 2013, 2014 ß ӳޠֶशऀίʔύε΋ଟ਺ϦϦʔε Þ NUS Corpus of Learner English Þ Lang-8 Learner Corpora 24
25. ݻ༗දݱೝࣝɾޠٛᐆດੑղফ ͔Β entity linking ΁ ß ݻ༗දݱೝࣝ Þ ݻ༗දݱͷՕॴΛಉఆ ß entity linking Þ ݻ༗දݱ͕ԿΛࢦ͔͢ᐆດੑղফ Þ Wikify (Wikification) 25 ҆ഒट૬͕ࣄ࣮ޡೝΛೝΊɺҨ״Λද໌ͨ͠ɻ
26. ຊ೔ͷ·ͱΊ ß ਂ૚ֶश͕ݴޠॲཧʹ༩͑ΔΠϯύΫτ Þ ߏจղੳ͔Βҙຯղੳ·Ͱ end-to-end Þ ϚϧνϞʔμϧʢը૾ɾԻ੠ɾݴޠʣॲཧ Þ ςΩετੜ੒͕ࠓޙരൃతʹීٴͦ͠͏ ß ࣗવݴޠॲཧͷ৽ͨͳൃల Þ ݴޠඇґଘͳख๏ͷݕ౼ͱ໰୊ͷ෼ੳ Þ ؤ݈ͳղੳख๏ͷ໛ࡧ Þ ΢Σϒͷొ৔ʹΑΔݹͯ͘৽͍͠໰୊ઃఆ 26