LSTMを用いた自然言語処理について

第3回TokyoCL勉強会 でのLSTMについての発表資料です

Transcript

1. LSTMΛ༻͍ͨ ࣗવݴޠॲཧʹ͍ͭͯ ಙӬ୓೭ (@tkng) 
   5PLZP$-ษڧձ
   ୈճ!άʔάϧגࣜձࣾ
2. ࣗݾ঺հ • Twitter: @tkng • ΧϨʔ͕޷͖Ͱ͢
3. ຊ೔ͷ࿩୊ • Recurrent Neural Network & LSTM • LSTMΛ࢖ͬͨ࠷ۙͷݚڀࣄྫ • LSTMΛ࢖ͬͨಠ࣮ࣗݧ
4. Recurrent Neural Network • ࣌ܥྻʹฒͿཁૉΛ1ͭͣͭड͚औͬͯɺঢ়ଶ Λߋ৽͍ͯ͘͠ωοτϫʔΫͷ૯শ • LSTM΋RNNͷҰछͱݴ͑Δ IUUQDPMBIHJUIVCJPQPTUT6OEFSTUBOEJOH-45.T
5. Seq2seqϞσϧͱ͸ʁ • Մม௕ͷೖྗσʔλΛɺRecurrent Neural NetworkΛ࢖ͬͯݻఆ௕ͷϕΫτϧʹΤϯίʔ υͯ͠ɺ͔ͦ͜Β຋༁ޙͷσʔλΛσίʔυ ͢Δ
6. Seq2seqϞσϧͰͷ຋༁ 5IJT JT B QFO &04 ͜Ε ͸ ϖϯ Ͱ͢ &04 ͜Ε ͸ ϖϯ Ͱ͢
7. Seq2seqϞσϧͰͷ຋༁ 5IJT JT B QFO &04 ͜Ε ͸ ϖϯ Ͱ͢ &04 ͜Ε ͸ ϖϯ Ͱ͢ 5IJT
   JT
   B
   QFOΛݻఆ௕ʹ
   Τϯίʔυ͍ͯ͠Δʂ
8. Effective Approaches to Attention- based Neural Machine Translation (Bahdanau+, 2015) • Seq2seqͰػց຋༁ • Local Attentionͱ͍͏৽͍͠ख๏ΛఏҊ • ͍͔ͭ͘ͷݴޠϖΞͰɺstate of the artΛୡ੒
9. A Neural Conversational Model (Vinyals+, 2015) • LSTMΛ࢖ͬͯର࿩γεςϜΛ࡞ͬͨΒͦΕͬ Ά͘ಈ͍ͨ
10. Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (Kumar+, 2015) • Dynamic Memory NetworksΛఏҊͨ͠ • Recurrent Neural NetworkΛ૊Έ߹ΘͤͨΑ ͏ͳϞσϧʹͳ͍ͬͯΔ • ࣭໰Ԡ౴ɺ඼ࢺλά෇͚ɺڞࢀরղੳɺධ൑ ෼ੳͰstate of the art • ΄΅ಉ͡ϞσϧͰ৭ʑղ͚Δͷ͕͍͢͝ͱ ͍͏ओு
11. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (Xu+, 2015) • ը૾ʹର͢Δղઆจͷੜ੒ • CNN + LSTM + Attention IUUQLFMWJOYVHJUIVCJPQSPKFDUTDBQHFOIUNM
12. Semi-supervised Sequence Learning (Dai+, 2015) • LSTMΛ࢖ͬͯ sentiment analysis ͳͲ෼ྨܥ ͷλεΫΛ࣮ݧ • Language ModelͱSequence Autoencoderͷ2 ͭΛpretrainingͷख๏ͱͯ͠༻͍ɺ্هͷ࣮ݧ Ͱstate of the artΛߋ৽ • γϯϓϧͳख๏Ͱɺ஍ຯʹ͍͢͝
13. An Empirical Exploration of Recurrent Network Architectures ( Jozefowicz+, 2015) • Googleͷܭࢉػύϫʔʹ෺ΛݴΘͤͯɺ LSTM΍GRUͷ೿ੜϞσϧΛͨ͘͞ΜධՁͨ͠ • LSTMΛॳظԽ͢Δࡍɺforget gateͷbiasΛ1 ʹ͢Δ͜ͱΛڧ͘קΊ͍ͯΔ
14. ͭ·ΓɺࠓɺLSTM͕φ΢͍ʂ • ෼ྨλεΫͰ΋state of the art͕ग़ͤΔ • Seq2seqͰจੜ੒͕Ͱ͖Δ
 Ͳ͏͍͏ཧ۶ͳͷ͔Α͘Θ͔Βͳ͍…
15. ͔͜͜Βઌ͸ಠ࣮ࣗݧͷ࿩
16. ͳʹΛ࣮ݧ͔ͨ͠ʁ • LSTMʹΑΔ Sentence Auto-encoder • ࣮ݧ͸͍Ζ͍ΖࡶͰ͢ ͜Ε ͸ ϖϯ Ͱ͢ &04 ͜Ε ͸ ϖϯ Ͱ͢ &04 ͜Ε ͸ ϖϯ Ͱ͢
17. ͳΜͰ࣮ݧ͔ͨ͠ʁ • Seq2seq͸͓΋͠Ζ͍ٕज़͕ͩɺத਎͕Α͘ Θ͔Βͳ͍
18. ஌Γ͍ͨ͜ͱ • Ͳ͏͍͏จΛؒҧ͑΍͍͢ͷ͔ʁ • ͳΜͰ։͍ׅͨހΛด͡ΒΕΔͷ͔ʁ • ͳΜͰೖྗΛٯʹͨ͠ํ͕͍͍ͷ͔ʁ • ࣅͨจ͕࡞Γग़͢ঢ়ଶ͸ࣅ͍ͯΔͷ͔ʁ • ͳΜͰݻఆ௕ͷσʔλ͔ΒՄม௕ͷग़ྗ͕Ͱ ͖Δͷ͔ʁ
19. σʔλ • ݸਓతʹूΊ͍ͯͨ೔ຊޠͷϒϩάσʔλ • ܇࿅ɿ ໿60ສจ (33MB) • ςετɿ ໿3ສ5ઍจ (2MB)
20. ࣮ݧઃఆʢ1ʣ • ޠኮʢor จࣈʣ਺͸80000 or 10000 • ೖྗ͸100࣍ݩͷembeddingʹม׵ • LSTM͸1૚ɺঢ়ଶ਺͸200࣍ݩ • ࠷దԽʹ͸AdamΛ࢖༻ • ίʔυ͸ https://github.com/odashi/ chainer_examples Λར༻ͨ͠
21. ࣮ݧઃఆʢ2ʣ • ೖྗΛ୯ޠ୯Ґʹ͢Δ͔ɺจࣈ୯Ґʹ͢Δ͔ • ೖྗΛٯॱʹ͢Δ͔ɺͦͷ··ೖΕΔ͔ • ߹ܭ4ύλʔϯΛ࣮ݧͨ͠ • ୯ޠ୯Ґ͸50epoch, ޠኮ਺80000ޠ • จࣈ୯Ґ͸100epoch, ޠኮ਺10000จࣈ
22. ͲΜͳײ͡ͰֶशͰ͖Δ͔ʁ • trg = ͠Ό͘ ͠Ό͘ ͱ φγ ͷ Α͏ ͳ ৯ ײ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = Χϥʔ ໾ ͱ ग़਎ ͷ Α͏ ͳ ࣄଶ ʹ ʹ ඇৗ ͳ ࢪઃ ɻ • hyp = ਎ମ ౳ ͱ ࠃՈ ͷ Α͏ ͳ ৯ ײ ʹ ߦ͘ ͳ ୴ಹ ɻ • hyp = ౳ ౳ ͱ φγ ͷ Α͏ ͳ ৯ ײ ʹ ্඼ ͳ ୴ಹ ɻ • hyp = ೖࡳ ౳ ͱ φγ ͷ Α͏ ͳ ৯ ԰ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = ࣇۄ Ԃ ͱ φγ ͷ Α͏ ͳ ৯ ײ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = PA ͠Ό͘ ͱ ٳܜ ͷ Α͏ ͳ ৯ ͗͢ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = ͠Ό͘ ͠Ό͘ ͱ φγ ͷ Α͏ ͳ ৯ ײ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = ͠Ό͘ ͠Ό͘ ͱ ↑ ͷ Α͏ ͳ ৯ ײ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = ͠Ό͘ ͠Ό͘ ͱ ↑ ͷ Α͏ ͳ ৯ ײ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = ͠Ό͘ ͠Ό͘ ͱ φγ ͷ ͳ ͳ ৯ ײ ʹ ্඼ ͳ ؁ຯ ɻ • hyp = ͠Ό͘ ͠Ό͘ ͱ φγ ͷ Α͏ ͳ ৯ ײ ʹ ্඼ ͳ ؁ຯ ɻ
23. ֶशͷਐΉ༷ࢠʢจਖ਼ղ཰ʣ
24. ֶशͷਐΉ༷ࢠʢจਖ਼ղ཰ʣ
25. จ௕ʹΑΔਖ਼ղ཰ͷมԽ
26. ਖ਼ॱͱٯॱͱͷൺֱ
27. ग़ྗΛٯॱʹͯ͠ΈΔͱ…ʁ
28. ͜͜·Ͱͷߟ࡯ • ௕จΛؒҧ͑΍͍͢܏޲͸໌Β͔Ͱ͋Δ • ٯॱͰೖྗͨ͠΄͏͕ੑೳ͕Α͍ • ୅ΘΓʹग़ྗΛٯॱʹͯ͠΋Α͍ • ֶश཰Λ్தͰԼ͛Δͷ͸ޮՌతͰ͋Δ
29. ඍົʹೖྗΛม͑ͯΈΔͱʁ ʮձࣾͰΠϯϑϧΤϯβ͕େྲྀߦ͍ͯ͠Δɻʯ ೖྗɿձࣾͰϯϑϧΤϯβ͕େྲྀߦ͍ͯ͠Δɻ ग़ྗɿձࣾͰϯϑϧΤϯβ͕େྲྀͦ૗ʹ͍Δɻ ೖྗɿձࣾͰΠϑϧΤϯβ͕େྲྀߦ͍ͯ͠Δɻ ग़ྗɿձࣾͰΠϑϧΤϯβ͕େྲྀͦ૗͍͠Δɻ ೖྗɿΠϯϑϧΤϯβ ग़ྗɿΠϯϑϧΤϯτ
30. ผͷྫͰ͸Ṗͷ่յ΋ ೖྗɿձࣾͰΠϯϑϧϯΤβ͕େྲྀߦ͍ͯ͠Δɻ ग़ྗɿձࣾͰΠϯϑϧϯΤϨʢ໙Θ։ྲྀͩ૿ خɻ
31. ೲಘͰ͖Δؒҧ͍ྫ ೖྗɿࢲ͸ʰिؒগ೥αϯσʔʱΛຖिߪಡ͠ ͍ͯ·͢ɻ ग़ྗɿࢲ͸ʰिؒগ೥αϯσʔΛΛຖ೔ങಡ͠ ͍ͯ·͢ɻ
32. ೖྗΛม͑ͯΈͯͷߟ࡯ • ୹͍จͰ΋ࣦഊ͢Δ͜ͱ͕͋Δ • 1จࣈ่͚ͩΕΔ৔߹΋͋Δ͕ɺ్த͔Βେ่ յΛ࢝ΊΔ৔߹΋͋Δ • िͱ೔Λؒҧ͑ͨΓɺ༻๏͕ࣅͨจࣈΛؒҧ ͍͑ͯΔྫ΋ݟΒΕΔ
33. ࠓޙͷ՝୊ • ΋ͬͱͪΌΜͱ࣮ͨ͠ݧ • ΑΓ޿ൣͳύϥϝʔλʔ୳ࡧ • 2૚ɺ3૚ͷLSTMͷར༻ • dropoutͷར༻ • ঢ়ଶͷՄࢹԽ
34. ·ͱΊ • LSTMʹจΛ෮ݩͤͯ͞Έͨ • ࣮ࡍɺͦΕͳΓʹ෮ݩͰ͖Δ • ೖྗΛٯॱʹ͢Δ୅ΘΓʹɺग़ྗΛٯॱʹ͠ ͯ΋Α͍͜ͱ͕Θ͔ͬͨ
35. ࠷ۙͷؔ࿈ݚڀ • http://arxiv.org/abs/1506.02078 • LSTMͷՄࢹԽ • http://citeseerx.ist.psu.edu/viewdoc/download? doi=10.1.1.1.3412&rep=rep1&type=pdf • RNNͰจ຺ࣗ༝จ๏ΛύʔεͰ͖Δ • http://www.aclweb.org/anthology/P/P15/ P15-1107.pdf • Sentence Auto encoderͷ࣮ݧ