Share
Embed
SNLP2021: Style is NOT a single variable: Case Studies for Cross-Stylistic Language Understanding
Reina Akama
September 06, 2021
Transcript
21 slides
-
Style is NOT a single variable: Case Studies for Cross-Stylistic
Language Understanding Dongyeop Kang and Eduard Hovy ACL-IJCNLP2021 ಡΈख: ؒ ྯಸʢ౦େʣ @ୈ13ճ࠷ઌNLPษڧձ -
ຊݚڀͷ࢟ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 2 • Systemic Functional Linguistics (SFL) Theory
[Halliday 2003] ʹجͮ͘ 選択体系機能⾔語学: ݴޠ͕ࣾձͷதͰͲͷΑ͏ʹΘΕΔ͔ ίϛϡχέʔγϣϯʹ͓͚ΔׂɾཱͪҐஔ Textual Interpersonal Ideational Dongyeop Kang; Linguistically Informed Language Generation: A Multi-faceted Approach (CMU-LTI-20-003) -
• Systemic Functional Linguistics (SFL) Theory [Halliday 2003] ʹجͮ͘ ຊݚڀͷ࢟
2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 3 選択体系機能⾔語学: ݴޠ͕ࣾձͷதͰͲͷΑ͏ʹΘΕΔ͔ Dongyeop Kang; Linguistically Informed Language Generation: A Multi-faceted Approach (CMU-LTI-20-003) ίϛϡχέʔγϣϯʹ͓͚ΔׂɾཱͪҐஔ ֤ػೳΛߏ͢Δཁૉྫ -
ελΠϧͷཁૉΛͲ͏ཧ͢Δ͔ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 4 • ैདྷͷࣾձݴޠֶͷҰྫ • Hovy (1987): pragmatics
constraints e.g., strong or neutral stance about the topic of a text or a different level of formality • Biber (1991): participant roles and characteristics and the relationships among participants • ຊݚڀͷ৽͍͠ྨ • Social participation [Biber 1991] ͱ Content coupledness ʹجͮ͘4ͭͷΧςΰϦʹྨ スタイルが会話の中で 話し⼿と聞き⼿の どちらに関係しているか スタイルが原⽂の内容に どれだけ影響を受けているか -
Cross-style Language Understanding 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 5 • Cross-style Language Understanding
• ςΩετͷޠኮతಛͱײنൣͱͷ૬ؔ [Warriner et al. 2013] • formality, frustration, and politeness ؒͷ૬ؔؔ [Chhaya et al. 2018] • demographic information (e.g., gender, age) ͱ emotion ؒͷ૬ؔؔ [Preotiuc-Pietro and Ungar 2018] • ϝλϑΝʔͱײͷ૬ޓ࡞༻ [Dankers et al. 2019; Mohammad et al. 2016] -
֓ཁ • ͜Ε͔Β包括的・横断的ͳ Cross-style language understanding • ʮελΠϧΦʔέετϥʯ •
ґଘؔɺڞଘՄೳੑΛ͘ௐࠪ • ߩݙ • ^ ΛՄೳʹ͢ΔϕϯνϚʔΫσʔλ X SLUE Λඋ • Total 1.6M sentences on 15 stylesʢطଘσʔλ 14 + ৽ن 1ʣ • Cross-style diagnostic set (500 sentences with 15 styles) • XSLUE Λ༻͍ͨ cross-style ͳ case-study Λհ • ϥϕϧྨɺ૬ؔੳɺ͖݅ੜ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 6 -
• ᶃ Individual style dataset: طଘσʔλΛେنʹूɾܗࣜΛ౷Ұ • ᶄ Cross-style diagnostic
set: ελΠϧԣஅతʹݕূ͍ͨ͠ɺશελΠϧϥϕϧΛਓखͰ༩ X SLUE: A Benchmark for Cross-Style Language Understanding and Evaluation 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 7 SarcasmDetection [Gosh and Veale 2016] PASTEL [Kang et al. 2019] Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male Text: I just imagined you dancing like this Style: Sarcasm:NonSarcastic Text: :) They always had pizza around, too! Style: Formality:Informal, Politeness:Polite, Humor:NonHumorous, Sarcasm:NonSarcasm, …, Gender:NonBinary, Age:18-24, Country:USA, PoliticalView:RightWing, … -
X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 8 包括的
• ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male -
X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 9 包括的
• ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male -
X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 10 包括的
• ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male -
X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 11 包括的
• ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male -
X SLUE ― Cross-style diagnostic set 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 12 診断セット作成⽅法:
1. ೋ௨Γͷํ๏Ͱஅ༻αϯϓϧ 500 sentences Λ֫ಘ • Individual style dataset ʹ༻͍Δ֤σʔλͷςετηοτ͔Βແ࡞ҝʹ 250 sentences • ࣄલֶशࡁྨثʹΑΔελΠϧ༧ଌͰείΞʹେ͖ͳΒ͖͕ͭੜͨ͡πΠʔτ͔Βແ࡞ҝʹ 250 tweets 2. ֤αϯϓϧʹର͠ɺશελΠϧ (e.g., Formality, Gender) ʹ͍ͭͯϥϕϧ (e.g., Informal, Male) ΛਓखͰ༩ • 1αϯϓϧ͋ͨΓ5ਓ͕Ξϊςʔγϣϯɺ֤ελΠϧͷ࠷ऴతͳϥϕϧ5ਓͷ majority vote Ͱܾఆ Text: :) They always had pizza around, too! Style: Formality:Informal, Politeness:Polite, Humor:NonHumorous, Sarcasm:NonSarcasm, …, Gender:NonBinary, Age:18-24, Country:USA, PoliticalView:RightWing, … ✔ 複数スタイル間の依存関係や相互作⽤が検証可能に -
Three Case studies on XSLUE 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 13 • Case
#1: スタイルラベル分類タスク • Style is NOT a single variable! • ݸผʹϞσϧԽ͢ΔΑΓ·ͱΊͯϞσϧԽ͢Δ΄͏্͕ख͍ͣ͘͘ • Case #2: スタイル属性間の依存関係(相関)分析 • ελΠϧଐੑؒͷ݁ͼ͖ͭͷେখΛௐࠪ • Case #3: スタイル属性の組み合わせと⽣成 • ελΠϧଐੑʹڞଘՄೳͳΈ߹Θͤͱɺͦ͏Ͱͳ͍Έ߹Θ͕ͤ͋Δ -
Case #1: ελΠϧϥϕϧྨλεΫ ― ઃఆ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 14 モデル •
Single models: • Majority classifier • biLSTM [Hochreiter and Schmidhuber 1997] • BERT [Devlin et al. 2019] • RoBERTa [Liu et al. 2019] • T5 [Raffel et al., 2019] タスクと評価尺度 • (1) individual-set, (2) cross-set ্ͰͷελΠϧϥϕϧྨ • F1 score (multi-label ͷ߹ϚΫϩฏۉΛ༻͍Δ) • Cross model: (based on pre-trained T5) ⼊⼒: Style + Text → 出⼒: Label スタイル毎にモデル化 vs 複数スタイルをまとめてモデル化 -
スタイル毎にモデル化 Case #1: ελΠϧϥϕϧྨλεΫ ― ݁Ռ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 15 Text:
The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male Text: :) They always had pizza around, too! Style: Formality:Informal, Politeness:Polite, Humor:NonHumorous, Sarcasm:NonSarcasm, …, Gender:NonBinary, Age:18-24, Country:USA, PoliticalView:RightWing, … Cross ͕ single Λେ෯ʹվળ Cross ͰੑೳԼ Metaphor ͷ߹ target metaphor verb ͕ Text ͷจ಄ʹདྷΔͨΊ ଞͱएׯλεΫઃఆ͕ҟͳΔ: Style + Text (=Verb:Text) → Label → ྆ऀΛҰॹʹֶश͢Δͱੑೳ͕Լ͢ΔՄೳੑ 全体としては Cross で性能が改善する傾向 ෳελΠϧΛಉ࣌ʹϞσϧԽ͢Δ͜ͱͷ༗ޮੑΛࣔࠦ ✔ T5 < Cross (T5-based) -
Case #2: ελΠϧؒͷґଘؔੳ ― ઃఆ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 16 χϡʔεهࣄʹൺͯελΠϧ͕ଟ༷ 分析⽅法
1. ৽ͨʹऩूͨ͠πΠʔτ1Mʹର͠ɺ ελΠϧྨث (cross) Λ༻͍ͯશ53छͷελΠϧϥϕϧ (=ଐੑ) (e.g., Informal, Male) ͷ֬Λ༧ଌ 2. ྨث͕ࢉग़ͨ͠༧ଌΑΓɺϐΞιϯͷ૬ؔͱϢʔΫϦουڑΛ༻͍ͯଐੑؒͷ૬ؔߦྻΛಘΔ ਖ਼ͷ૬ؔ ෛͷ૬ؔ スタイル属性間の結びつきの⼤⼩を調査 -
2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 17 Case #2: ελΠϧؒͷґଘؔੳ ― ݁Ռ ✔得られた属性間の依存関係は⼈間の主観と⼀致 (Target
Style, Correlated Style) ϖΞʹ͍ͭͯɺ͜ΕΒͷଐੑ͕૬ޓʹґଘ ͍ͯ͠Δ͜ͱͷଥੑΛධՁऀ2໊͕5ຬͰධՁɻ2ਓͷฏۉΛHɻ ✔Ward法よる可視化で解釈可能なクラスタを確認 ΞδΞܥຽ (SouthAsian, EastAsian), த (35-44, 45-54, 55-74)ɺ ੵۃੑ (happiness, dominance, positive, polite)ɺѱ͍ײ (anger, disgust, sadness, fear) ͳͲɻ スタイル属性間の結びつきの⼤⼩を調査 -
Case #3: ελΠϧͷΈ߹Θͤͱੜ ― ઃఆɾ݁Ռ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 18 条件付き⽂⽣成モデル •
PPLM [Dathathri et al. 2019] ! "|$! , … , $" ∝ !(") * !($! |") ⋯ !($" |") タスクと評価尺度 • ͖݅จੜ ೖྗ: Prompt “Every natural text is”, Target style {Polite, Impolite}º{Positive, Negative} → ग़ྗ: Stylized text • ਓखධՁ Stylistic appropriateness score (1-5), ධՁऀ3໊ͷฏۉ 結果 スタイル属性には共存可能な組み合わせと、そうでない組み合わせがある ݴޠϞσϧ (GPT-2) ελΠϧྨث (cross) ✔⽣成例 (!=1 ͷ߹) ਓखධՁ݁Ռ ૬ؔείΞ (Case #2) ✔属性の組み合わせによりスコアが低下、相関とも関連か PosºImpolite, NegºPolite ͕͍ɻ͜ͷ߹ʹ͍ͭͯଐੑؒͷ૬ؔͱ͕Ұகɻ -
·ͱΊ 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 19 • Cross-style language understanding ݚڀͷͨΊͷϕϯνϚʔΫσʔλ XSLUE
Λ࡞ • Social participation ͱ Content coupledness ʹج͖ͮελΠϧଐੑΛ4ΧςΰϦʹྨ • จϨϕϧͷελΠϧྨσʔληοτΛू͠ܗࣜΛ౷Ұ 15छͷελΠϧΛؚΉ 1M ͷσʔληοτ: Individual style dataset • ಉҰςΩετʹશ15छͷελΠϧϥϕϧΛਓखͰ༩ͨ͠ධՁηοτ: Cross-style diagnostic set • XSLUE Λ༻͍ͨ cross-style ͳ case-study Λհ • ϥϕϧྨɿෳελΠϧΛಉ࣌ʹϞσϧԽ͢Δ͜ͱͷ༗ޮੑΛࣔࠦ • ૬ؔੳɿελΠϧଐੑؒͷ݁ͼ͖ͭͷେখΛௐࠪɺਓؒͷओ؍ͱҰ؏͢Δ͜ͱΛ֬ೝ • ͖݅ੜɿελΠϧଐੑʹΈ߹ΘͤՄೳͳͷͱɺͦ͏Ͱͳ͍ͷ͕͋Δ 感想 • ར༻Մೳͳσʔλ͕͜͜5͘Β͍Ͱ૿͑ͨҹɺྑ͍λΠϛϯάͰͷཧ • ஶऀʮελΠϧͷछྨͬͱ૿͍ͨ͠ɺΑΓแׅతͳϕϯνϚʔΫʹ͍ͨ͠ʯ • Ξϊςʔγϣϯ࡞ۀͷઃఆͷઆ໌͕ͱͯஸೡ (ΞϊςʔγϣϯλεΫ࣮ࢪը໘ɺใुͷܾఆํ๏) -
XSLUE ―Cross-style diagnostic set 2021-09-16 ୈ13ճ࠷ઌNLPษڧձ 21 அηοτ࡞ํ๏ɿ 1. ೋ௨Γͷํ๏Ͱஅ༻αϯϓϧ
500 sentences Λ֫ಘ • Individual style dataset ʹ༻͍Δ֤σʔλͷςετηοτ͔Βແ࡞ҝʹ 250 sentences • ࣄલֶशࡁྨثʹΑΔελΠϧ༧ଌͰείΞʹେ͖ͳΒ͖͕ͭੜͨ͡πΠʔτ͔Βແ࡞ҝʹ 250 tweets 2. ֤αϯϓϧʹର͠ɺશελΠϧ (e.g., Formality, Gender) ʹ͍ͭͯϥϕϧ (e.g., Informal, Male) ΛਓखͰ༩ • 1αϯϓϧ͋ͨΓ5ਓ͕Ξϊςʔγϣϯɺ֤ελΠϧͷ࠷ऴతͳϥϕϧ5ਓͷ majority vote Ͱܾఆ Style IAA ͍͠ ओ؍తͳελΠϧ