Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ABCNN: Attention-Based Convolutional Neural Net...

Mamoru Komachi
September 12, 2016

ABCNN: Attention-Based Convolutional Neural Network for Medeling Sentence Pairs

第8回最先端NLP勉強会で小町が発表した、以下の研究に関するスライドです。
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs

Mamoru Komachi

September 12, 2016
Tweet

More Decks by Mamoru Komachi

Other Decks in Research

Transcript

  1. ABCNN: Attention-Based Convolutional Neural Network for Medeling Sentence Pairs Wenpeng

    Yin, Hinrich Schütze, Bin Xiang, Bowen Zhou TACL 2016 ※εϥΠυதͷਤද͸શͯ࿦จ͔ΒҾ༻͞Εͨ΋ͷ খொक <[email protected]> ୈ8ճ࠷ઌ୺NLPษڧձ@౦େ 2016/09/12
  2. ABCNN: Attention Based Convolutional Neural Network | ݸผͷλεΫʹಛԽͯ͠γεςϜΛ࠷దԽ →จରϞσϦϯά͕ඞཁͳ༷ʑͳλεΫʹద༻Մ |

    ϖΞΛߟྀͤͣʹจදݱΛݸผʹϞσϧԽ →จϖΞΛߟྀ | ਓखͰઃܭͨ͠λεΫʹಛԽͨ͠ૉੑ →ૉੑઃܭͳ͠ʹɺԠ౴બ୒ɾݴ͍׵͑ಉఆɾςΩετ ؚҙؔ܎ೝࣝλεΫͰ state-of-the-art https://github.com/yinwenpeng/Answer_Selection 4
  3. ࣗવݴޠॲཧʹ͓͚Δ CNN | จʢରʣ෼ྨλεΫͰΑ͘༻͍ΒΕ͍ͯΔ (Kalchbrenner et al., 2014; Kim, 2014;

    Socher et al., 2011; Yin and Schütze 2015) | ؤ݈ʹೖྗͷந৅తͳಛ௃Λଊ͑Δ͜ͱ͕Ͱ͖ Δʢͱߟ͑ΒΕ͍ͯΔʣ 5
  4. ͳͥ LSTM Ͱ͸ͳ͘ CNN? | LSTM ͸จ຺શମΛΤϯίʔυ; CNN ͸ϩʔΧ ϧͳϑϨʔζΛʢϑΟϧλʔʹΑͬͯʣݕग़

    →attention ͷ࡞༻ͷ࢓ํ͕ҧ͍ɺCNN ͷํ͕ attention ͕༗ޮʹʢہॴΛݟΔ CNN ͱશମΛݟ Δ attention ͕૬ิతʹʣಇ͘ | ৞ΈࠐΈ૚ΛॏͶͯந৅౓Λ্͛Δ͜ͱ͕Մೳ →ʢจର͔Βɺಉٛͱ͍͏ΑΓΉ͠ΖಉٛͰͳ͍෦ ෼Λݕग़͢΂͖ʣԠ౴બ୒λεΫɺςΩετؚҙؔ ܎ೝࣝλεΫͰ͸ಛʹ༗ޮ 6
  5. ରԠ͢ΔจͷͲ͜ΛݟΔ ͔ΛΞςϯγϣϯͰߟྀ ͢Δ 8 F 0,a = W 0 ⋅AT

    F 1,a = W 1 ⋅A A i, j = match-score(F 0,r [:,i],F 1,r [:, j]) Ξςϯγϣϯಛ௃ྔϚοϓ →W0 ͱW1 ͸ֶश͢Δ match-score ͸ϢʔΫϦου ڑ཭Λ༻͍ͯ 1 / (1 + | x – y |)
  6. ABCNN-2:ΞςϯγϣϯΛ ೖྗͰ͸ͳ͘ग़ྗʹ͔͚ Δ 9 Fp i,r [:, j]= a i,k

    k=j:j+w ∑ Fc i,r [:,k] ৞ΈࠐΈͷग़ྗʹΞςϯγϣϯॏΈ ʢߦ·ͨ͸ྻͰ࿨ΛऔΔ͜ͱʹΑΓɺ Ϣχοτ͝ͱͷॏΈʹू໿͢Δʣ
  7. σʔληοτ | WikiQA { ΦʔϓϯυϝΠϯͷQAσʔληοτ { ܇࿅ : ։ൃ :

    ςετ = 20,360 : 1,130 : 2,352 | MSRP (Microsoft Research Paraphrase) { ܇࿅ = 2,753 T : 1,323 F, ςετ = 1,147 T : 578 F; ։ൃσʔλ͸܇࿅͔ΒϥϯμϜʹ400ࣄྫநग़ | SICK { ؚҙɺໃ६ɺதཱͷ3Ϋϥε෼ྨ { ܇࿅ : ։ൃ : ςετ = 4,439 : 495 : 4,906 12