Share
Embed
NLPコロキウム:Unsupervised Abstractive Summarization Based on Tree-Structured Topic Guidance and Rate-Distortion Theory
NLPコロキウムで使用したスライド
https://nlp-colloquium-jp.github.io/schedule/2022-03-23_masaru-isonuma/
Masaru Isonuma
March 24, 2022
Transcript
22 slides
-
ࣗݾհ 2 • үপ େ/Masaru Isonuma • ܦྺ – 20153݄
౦ژେֶֶ෦ ଔۀʢେɾṦํݚڀࣨʣ – 20173݄ ౦ژେֶେֶӃֶܥݚڀՊम࢜՝ఔ मྃʢࡔాɾݚڀࣨʣ ʢίϯαϧςΟϯάاۀۈΛܦͯʣ – 20219݄ ౦ژେֶେֶӃֶܥݚڀՊത࢜՝ఔ मྃʢࡔాɾݚڀࣨʣ – ݱࡏ ౦ژେֶେֶӃֶܥݚڀՊ ࡔాɾݚڀࣨ ಛݚڀһ ܦྺ ݚڀ༰ • ڭࢣͳ͠/ऑڭࢣ͖ཁ – จॻྨͱͷϚϧνλεΫֶशʹΑΔॏཁจநग़ (EMNLP’17) – ஊߏΛଊ͑ͨڭࢣͳ͠ϔουϥΠϯੜ (ACL’19) – τϐοΫߏʹجͮ͘ڭࢣͳ͠ཁੜ (TACL’21) • τϐοΫϞσϧ – ߏχϡʔϥϧτϐοΫϞσϧ (ACL’20) ίϯαϧ͔Βڭࢣͳ͠ཁݚڀʹࢸͬͨܦҢʢʁʣ ͕هࡌ͞Ε͍ͯΔnoteهࣄ https://note.com/jst_kisokenkyu/n/n90d06ec74985 -
• τϐοΫߏʹجͮ͘ڭࢣͳ͠ཁੜ w/ J. Mori, D. Bollegala, I. Sakata (TACL’21)
ຊͷྲྀΕ 3 લɿ͜Ε·Ͱͷݚڀ ޙɿݱࡏͷݚڀ༰ • ϨʔτΈཧʹجͮ͘ڭࢣͳ͠ཁੜ overall service food atmo- sphere location place taste price The food here is fantastic, easily the best sub sandwiches in the Arizona area. The shop is local and family run, so I definitely choose it over a lot of the large national chains that are all around town. The staff are extremely friendly and will always go above and beyond in creating a delicious sandwich for you. You will not be let down by the great food that they make here! ҰൠԽ Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ Ռ͕͖ ڕ͕͖ จষ ཁ 1bit 2bit -
ࡢࠓɺࣗಈจॻཁඈ༂తʹਐԽ͍ͯ͠Δ matthew kenney, 34, said he smoked flakka before he
went streaking . was arrested on saturday after run through fort lauderdale, florida . drug is made from same version of stimulant used to produce bath salts . it causes euphoria, hallucinations, psychosis and superhuman strength . kenney has prior arrests and was hospitalized for a psychiatric evaluation . ৽ฉهࣄ matthew kenney, 34, told police he smoked flakka before he streaked through traffic in fort lauderdale while only wearing a pair of sneakers . he said he was escaping imaginary killers who he believed stole his clothes and wanted to murder him . kenney has previous arrests for disorderly conduct, making a riot and possession of a controlled substance . ࣗಈཁ (PEGASUS; Zhang et al., 2020) ਓ͕࡞ͨ͠ࢀরཁ • ສ݅୯ҐͷࢀরཁͷύλʔϯΛֶश͢Δڭࢣ͋Γख๏Ͱඈ༂తʹੑೳ্͕ • ۙͰ৽ฉهࣄͳͲͷσʔληοτͰਓͱಉͷੑೳΛୡ (PEGASUS; Zhang et al., 2020) ≈ 2022/4/14 4 ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ………………………………… -
͔͠͠ࢀরཁ͕গͳ͍จॻʹద༻͕ࠔ The food here is fantastic, easily the best sub
sandwiches in the Arizona area. The shop is local and family run, so I definitely choose it over a lot of the large national chains that are all around town. The staff are extremely friendly and will always go above and beyond in creating a delicious sandwich for you. You will not be let down by the great food that they make here! ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ……………………………………………… ………………………………… McGurkee's is a family run restaurant that has been serving up Italian sandwiches for over 30 years . • ྫ͑Web্ͷϨϏϡʔSNSߘͳͲͷҙݟจॻτϐοΫ͕ଟ༷Ͱɺࢀরཁͷ༻ҙ͕ࠔ • ͜͏ͨ͠จॻʹɺڭࢣ͋Γख๏͏·͘ػೳ͠ͳ͍ ≈ 2022/4/14 5 YelpϨϏϡʔ ࣗಈཁ (PEGASUS; Zhang et al., 2020) ਓ͕࡞ͨ͠ࢀরཁ -
τϐοΫߏΛख͕͔Γʹ͢Δ͜ͱͰࢀরཁΛෆཁʹ͢Δ 6 The food here is fantastic, easily the best
sub sandwiches in the Arizona area. The shop is local and family run, so I definitely choose it over a lot of the large national chains that are all around town. The staff are extremely friendly and will always go above and beyond in creating a delicious sandwich for you. You will not be let down by the great food that they make here! The sesame bread is amazing. The sandwich was huge but I ended up eating the whole thing. Very very good! Delicious. Cheap. Friendly. Quick (if you want it to be!). What more could you ask for? The atmosphere at McGurkee's is reminiscent of an old town go-to with the friendliest staff in the valley. The sandwiches are inexpensive and are, in my opinion, the best Italian subs in AZ. Get the Sicilian if you want to know what a real sub tastes like. Awesome sandwiches best bread, had the Pastrami Sandwich and it was a great experience with the homemade bread, pastrami meat and the condiments included, would not change a thing. Did I mention the bread? It is toasted and has a great taste and firm consistency. The restaurant is an older quaint building which is nice and fun. better One of the best sandwiches in town. There is nothing not to like about this place, great sandwiches and a friendly staff of people working behind the counter. Forget about the chains and give this place a try you willl be glad you did. It's been a while since I've been in this place, and I figured it was time for a good sammie. I've had a few of their sandwiches in the past, but today I got the Athenian chicken sandwich. The onions and feta cheese adds a nice flavor to it. I would recommend this place if you are in the area or for someone who wants to venture out from their usual sandwich joint. • τϐοΫߏΛਪఆ • τϐοΫຖʹཁจΛੜ YelpϨϏϡʔ ཁ ఏҊख๏ overall service food atmo- sphere location place taste price -
จॻதͷจͷજࡏΛࠞ߹ΨεͰදݱ જࡏʢࠞ߹Ψεʣ ߏτϐοΫϞσϧͰ τϐοΫൺΛਪఆ͠ɺ ରԠؔΛֶशʢVAEʣ 0.1 0.5 0.4 Food Service
Overall The food is delicious and the service is top notch. 2021/7/7 7 จॻதͷจ food service overall -
ࠞ߹ΨεΛղ͠ɺτϐοΫຖͷจͷʹղ food service overall I love this restaurant. The sandwich
is amazing. Wonderful service and staffs. ղ small large small જࡏʢࠞ߹Ψεʣ જࡏʢ୯ๆΨεʣ ͷࢄ͕ࢠΑΓେ͖͘ͳΔ Α͏ʹֶश͢Δ͜ͱͰɺཻͷ ҟͳΔཁจΛੜ 0.1 0.5 0.4 Food Service Overall The food is delicious and the service is top notch. 2021/7/7 8 food service overall จॻதͷจ ཁจ ฏۉΛσίʔυ ߏτϐοΫϞσϧͰ τϐοΫൺΛਪఆ͠ɺ ରԠؔΛֶशʢVAEʣ -
જࡏͷࢄ͕খ͘͞ͳΔͱੜ͞ΕΔจ͕ৄࡉʹͳΔ 2021/7/7 10 1. A great thing to go to
the place. 11. Friendly staff and I appreciated the food. 12. There is a lot of a great meal. 112. Food was great, and the service was great. 111. The staff is very friendly and knowledgable. 121. The meal was great, but the food was fantastic. 122. The <unk> was delicious and the fish tacos were great. τϐοΫจͷજࡏʢPCAʣ ੜ͞ΕͨτϐοΫจ ਂ͘ͳΔ΄Ͳ ࢄখ͍͞ ਂ͘ͳΔ΄Ͳ ੜจৄࡉ -
• ͜Ε·Ͱ༷ʑͳڭࢣͳ͠ཁϞσϧ͕ఏҊ – จϕΫτϧͷΫϥελॏ৺ɾத৺ੑ (Radev et al., 2000; Erkan &
Radev, EMNLP’04) – จॻϕΫτϧͷฏۉ (Chu & Liu, ICML’19; Bražinskas, ACL’20) – ֤τϐοΫͷղ (Isonuma et al., TACL’21) ͳͲͳͲ… • ͲΕͬͱΒ͍͕͠ɺԿ͕ຊʹඞཁͳͷ ͔/ຊ࣭తʹಉ֓͡೦ͳͷ͔Θ͔Βͳ͍ → ʢڭࢣͳ͠ʣจষཁͷഎܠཧ͕΄͍͠ ͜Ε·Ͱͷڭࢣͳ͠ཁݚڀʢࣗͷݚڀؚΉʣͷলʢʁʣ 11 ཧֶతͳ໘ ֶతͳ໘ • ڭࢣͳ͠ཁݚڀͷΒͰɺࣄલֶशϞσϧ ʹΑΔڭࢣ͋Γཁ͕ඈ༂తʹਐԽ – BART (Lewis et al., ACL’20) , – PEGASUS (Zhang et al., 2020), etc. • ҟͳΔΞʔΩςΫνϟͷཁϞσϧʹ ࣄલֶशࡁΈϞσϧΛΈࠐΉͷ݁ߏ͍͠ (Liu et al., EMNLP’19) → ৽͍͠ڭࢣͳ͠ཁϞσϧͰͳ͘ɺ ৽͍͠ڭࢣͳֶ͠शํ๏͕΄͍͠ -
• จষཁΛใཧͷݴ༿Ͱද͢ͱɺ ඇՄٯσʔλѹॖͱΈͳͤΔ • ͕ͨͬͯ͠ඇՄٯσʔλѹॖͷഎܠཧͰ͋Δ ϨʔτΈཧ (Shannon, 1959; Berger, 1971)
ɺ จষཁͷഎܠཧʹͳΔͷͰͳ͍͔ লΛ౿·͑ͨݱࡏͷݚڀ֓ཁ 12 ཧֶతͳ໘ ֶతͳ໘ • ϨʔτΈཧʹجͮ͘จষཁͷ৽͍͠త ؔʢֶशํ๏ʣΛఏҊ • จষཁϞσϧ𝑝(𝑦|𝑥)ԿͰΑ͘ɺ ࣄલֶशϞσϧΛ͏͜ͱՄೳ • ϋΠύʔύϥϝʔλΈؔʢޙड़ʣͰ ཁͷ͞༰Λ੍ޚՄೳ Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ Ռ͕͖ ڕ͕͖ จষ ཁ 1bit 2bit จॻ𝑥 ཁ𝑦 จষཁ Ϟσϧ 𝑝(𝑦|𝑥) 𝑝(𝑦|𝑥) = argmin !(#|%) 𝐿[𝑝 𝑦 𝑥 ] -
• ೖྗʢѹॖ͍ͨ͠ใʣΛ𝑋ɺग़ྗʢ𝑋Λѹॖͨ͠ใʣΛ𝑌ͱ͢Δ 𝑋Ұ༷𝑝(𝑥)ʹج͖ͮɺ𝑌𝑝(𝑦|𝑥)ʹج͖ͮఆ·ΔͱԾఆ͢Δ ϨʔτΈཧͱ 13 0 1 2 3 0
1 2 3 0 1 2 3 0.5 2.5 𝑋 𝑌 𝑋 𝑌 0 1 2 3 0.5 2.5 𝑋 𝑌 𝑝(𝑦|𝑥) (c) (b) (a) -
• ೖྗʢѹॖ͍ͨ͠ใʣΛ𝑋ɺग़ྗʢ𝑋Λѹॖͨ͠ใʣΛ𝑌ͱ͢Δ 𝑋Ұ༷𝑝(𝑥)ʹج͖ͮɺ𝑌𝑝(𝑦|𝑥)ʹج͖ͮఆ·ΔͱԾఆ͢Δ • ϨʔτΈཧͱɺѹॖޙͷใྔʢϨʔτʣΛҰఆʹͨ͠ͱͰɺ Ͳ͜·ͰѹॖલޙͷޡࠩʢΈʣΛখ͘͞Ͱ͖Δ͔ʹ͍ͭͯͨ͡ɺඇՄٯσʔλѹॖͷجૅཧ ϨʔτΈཧͱ 14 0 1
2 3 0 1 2 3 0 1 2 3 0.5 2.5 𝑋 𝑌 𝑋 𝑌 2bit 1bit (c) 0 1 2 3 0.5 2.5 𝑋 𝑌 Ϩʔτ: 1bit (b) (a) 1bit খ ແ = > Ϩʔτ Έ 2bit 1bit େ < > 𝑝(𝑦|𝑥) -
• ϨʔτΈཧͰϨʔτͱΈΛҎԼͰఆࣜԽ – Ϩʔτɿ૬ޓใྔ 𝐼 𝑋; 𝑌 = 𝐻 𝑋
− 𝐻 𝑋 𝑌 ʢ𝑌ΛΔ͜ͱͰݮΔ𝑋ͷෆ͔֬͞ʣ – Έɿޡࠩͷظ Ε[𝑑 𝑥, 𝑦 ] = ∑!,# 𝑝 𝑥 𝑝 𝑦|𝑥 𝑑 𝑥, 𝑦 ʢ𝑑 𝑥, 𝑦 ҙͷΈؔʣ ϨʔτͱΈͷఆࣜԽ 15 𝐼 𝑋; 𝑌 = 1 Ε[𝑑 𝑥, 𝑦 ] = 0.25 Ε[𝑑 𝑥, 𝑦 ] = 0 = > Ϩʔτ Έ ʢe.g., 2ޡࠩʣ 𝐼 𝑋; 𝑌 = 2 𝐼 𝑋; 𝑌 = 1 Ε[𝑑 𝑥, 𝑦 ] = 4.25 < > 0 1 2 3 0 1 2 3 0 1 2 3 0.5 2.5 𝑋 𝑌 𝑋 𝑌 2bit 1bit 0 1 2 3 0.5 2.5 𝑋 𝑌 Ϩʔτ: 1bit 𝑝(𝑦|𝑥) (c) (b) (a) -
• ϨʔτͱΈͷํΛ࠷খԽ͢Δ࠷దͳѹॖϞσϧ𝑝(𝑦|𝑥)ɺԼهࣜΛ࠷খԽ͢Δ͜ͱͰಘΒΕΔ 𝐿 𝑝 𝑦 𝑥 = 𝐼 𝑋; 𝑌
+ 𝛽Ε[𝑑 𝑥, 𝑦 ] ϨʔτΈཧʹجͮ͘σʔλѹॖͷఆࣜԽ 16 0 1 2 3 0 1 2 3 0 1 2 3 0.5 2.5 𝑋 𝑌 𝑋 𝑌 0 1 2 3 0.5 2.5 𝑋 𝑌 𝑝(𝑦|𝑥) 𝛽 = େ 𝛽 = খ ࠷దԽޙ ࠷దԽલ 𝐼 𝑋; 𝑌 = 1 Ε[𝑑 𝑥, 𝑦 ] = 0.25 Ε[𝑑 𝑥, 𝑦 ] = 0 = > Ϩʔτ Έ ʢe.g., 2ޡࠩʣ 𝐼 𝑋; 𝑌 = 2 𝐼 𝑋; 𝑌 = 1 Ε[𝑑 𝑥, 𝑦 ] = 4.25 < > -
• ಉ༷ʹɺจॻΛ𝑥ɺཁΛ𝑦ͱ͠ɺ𝑝(𝑦|𝑥)ΛཁϞσϧʢe.g., Τϯίʔμσίʔμʣͱͨ͠ͱ͖ɺ ཁλεΫ·ͨ𝐿 𝑝 𝑦 𝑥 Λ࠷খԽ͢Δͱͯ͠ଊ͑ΒΕΔͷͰͳ͍͔ 𝐿 𝑝
𝑦 𝑥 = 𝐼 𝑋; 𝑌 + 𝛽Ε[𝑑 𝑥, 𝑦 ] • 𝛽Ͱཁͷ͞୯ޠͷछྨΛίϯτϩʔϧͰ͖ͦ͏ ϨʔτΈཧʹجͮ͘จষཁͷఆࣜԽ 17 Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ 𝑋 𝑌 𝛽 = େ 𝛽 = খ ࠷దԽޙ ࠷దԽલ Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ Ռ͕͖ ڕ͕͖ 𝑋 𝑌 Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ Ռ͕͖ ڕ͕͖ 𝑋 𝑌 ཁ͕͍or୯ޠͷछྨ͕গͳ͍ ཁ͕͍or୯ޠͷछྨ͕ଟ͍ -
• ڭࢣͳ͠ཁɺจ/จॻϕΫτϧͷฏۉத৺ੑΛ༻͍Δͷ͕ଟ͍ (Radev et al., 2000; Erkan & Radev, EMNLP’04;
Chu & Liu, ICML’19; Bražinskas, ACL’20; Isonuma et al., TACL’21, etc.) • ͜ΕΒͷख๏͍ͣΕϨʔτͷ্ݶΛݻఆ͠ͳ͕ΒԿΒ͔ͷΈΛ࠷খԽ͍ͯ͠ΔͱҰൠԽͰ͖Δ طଘͷڭࢣͳ͠ཁख๏ͱͷ 18 ੜܕྫ (Isonuma et al., 2021) நग़ܕྫ (Radev et al., 2000) ϨʔτΈཧతݟํ طଘख๏ GMMͷ֤Ψεͷฏۉ ˔͔ΒཁจΛσίʔυ จϕΫτϧ จ𝑥 ཁจ𝑦 จϕΫτϧ จ𝑥 ཁจ𝑦 จϕΫτϧͷ֤Ϋϥελฏۉ ʹ࠷͍ۙཁจ˔Λநग़ ϋʔυͳ𝑝(𝑦|𝑥)ͷͱͰɺϨʔτ ͷ্ݶΛΫϥελʹݻఆ͠ɺ ΈʢೋޡࠩʣΛ࠷খԽ ιϑτͳ𝑝(𝑦|𝑥)ͷͱͰɺϨʔτ ͷ্ݶΛΨεͷʹݻఆ͠ɺ ΈʢೋޡࠩʣΛ࠷খԽ -
• ڭࢣ͋ΓཁֶशͷաఔͰɺϨʔτ𝐼 𝑋; 𝑌 ͱΈΕ[𝑑 𝑥, 𝑦 ]͕Ͳ͏มԽ͢Δ͔ݟͯΈΔ – ࣄલֶशࡁBART
(Lewis et al., ACL’20) Λ༻͍ͯɺ CNN-DailyMailσʔληοτͰཁੜλεΫΛֶश – ೖྗͷ୯ޠ𝒑! ͱग़ྗͷ୯ޠ𝒑# ؒͷKLใྔΛΈؔͱͯ͠༻ (Peyrard, ACL’19) Ε[𝑑 𝑥, 𝑦 ] = ; !,# 𝑝 𝑥 𝑝 𝑦|𝑥 D$%[𝒑#||𝒑!] • Ծઆɿ – BARTͰೖྗจΛͦͷ··ग़ྗ͢ΔΑ͏ʹࣄલֶशΛߦ͏ͨΊɺཁֶशલࠨͷঢ়ଶʹ͍ۙ – ཁֶश͕ਐΉͱɺग़ྗ͕͘ͳΔor୯ޠͷछྨ͕গͳ͘ͳΔ͜ͱͰɺӈͷঢ়ଶʹۙͮ͘ ॳظతͳ࣮ݧ 19 Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ 𝑋 𝑌 Έ͔Μ͕͖ ΓΜ͕͖͝ ͍ڕ͕͖ ੨͍ڕ͕͖ 𝑋 𝑌 Ռ͕͖ ڕ͕͖ ཁֶशલ Ϩʔτɿେ Έɿখ ཁֶशޙ Ϩʔτɿখ Έɿେ -
• ֓ͶԾઆ௨Γͷ݁Ռ͕ಘΒΕͨ – ʢࠨਤʣֶश͕ਐΉͱϨʔτ͕খ͘͞ͳΓɺΈ͕େ͖͘ͳΔ – ʢӈਤʣϨʔτͷ૿ݮʹԠͯ͡ɺཁ͕૿ݮ͢Δ ࣮ݧ݁Ռ 20 Ϩʔτ Έ
ֶशεςοϓ Ϩʔτʢࠨ࣠ʣ Έʢӈ࣠ʣ Ϩʔτ ཁ Ϩʔτʢࠨ࣠ʣ ཁʢӈ࣠ʣ ֶशεςοϓ ≈ -
• ֶश͕ਐΉʹͭΕɺϨʔτͱΈͷ߹ܭ 𝐿 𝑝 𝑦 𝑥 = 𝐼 𝑋; 𝑌
+ 𝛽Ε[𝑑 𝑥, 𝑦 ]͕খ͘͞ͳΔ • Έؔݕ౼ͷ༨͋Γʢ୯ޠͰͳ͘ɺ୯ޠຒΊࠐΈΛ༻͍ͨͷͳͲʣ ࣮ݧ݁Ռ 21 Ϩʔτ𝐼 𝑋; 𝑌 ΈΕ[𝑑 𝑥, 𝑦 ] 100εςοϓ 200εςοϓ … ࠨԼํ ʹਐΉ -
• ༷ʑͳڭࢣͳ͠ཁख๏͕ɺϨʔτͱΈͷ࠷খԽʹؼண͞ΕΔ͜ͱΛࣔͨ͠ – τϐοΫจੜʹΑΔڭࢣͳ͠ཁͦͷҰͭ • ࣮ࡍʹɺڭࢣ͋ΓཁֶशͷաఔͰϨʔτͱΈͷ߹ܭ͕খ͘͞ͳΔ͜ͱΛ֬ೝͨ͠ – ࠓޙٯʹɺϨʔτͱΈΛ࠷খԽ͢Δ͜ͱͰɺจষཁλεΫΛղ͚Δ͔ݕূ – ߋʹɺϨʔτΈཧΛ֦ுͨ͠ΈͰ͋Δinformation
bottleneck๏ʹΑΓɺ aspect-based summarizationͱ͍ͬͨɺཁ༰Λ੍ޚ͢Δඞཁͷ͋ΔจষཁλεΫΛղ͚ͳ͍͔ݕূ • ߋʹɺΩϟϓγϣϯੜͳͲɺจষҎ֎ͷσʔλΛจষʹʮѹॖʯ͢ΔλεΫΛ ͜ͷΈͰఆࣜԽͰ͖ͳ͍͔ઓ͍ͨ͠ʢໝʣ ·ͱΊͱࠓޙͷల 22