Share
Embed
【輪講資料】From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
研究室内の輪講用に作った資料です
Yano
May 03, 2023
Transcript
42 slides
-
From Zero to Hero: On the Limitations of Zero-Shot Language
Transfer with Multilingual Transformers 5݄2 ాɾݚڀࣨɹM2 ઍߛ Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavas 2020 EMNLP -
From Zero to Hero: On the Limitations of Zero-Shot Language
Transfer with Multilingual Transformers ֓ཁ • ଟݴޠϞσϧͷݴޠؒసҠೳྗͷݶքʹ͍ͭͯௐࠪ 1. ݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζݴޠؒసҠೳྗʹͲͷΑ͏ͳӨڹΛ༩͑Δ͔ʁ 2. ԼҐλεΫͱݴޠؒసҠೳྗʹؔ͋Δ͔ʁ 3. ݴޠؒసҠೳྗͷ༧ଌՄೳ͔ʁ 4. few-shotʹίετΛࢧ͏͖͔ʁ • ιʔεݴޠ͔Βԕ͍ݴޠϦιʔεͳݴޠͰzero-shotਪ͍͠ • ϨϕϧλεΫͷసૹੑೳʹߏతͳݴޠྨࣅੑ͕େ͖͘Өڹ • ߴϨϕϧͷLUλεΫͷసૹੑೳʹࣄલֶशͷίʔύεαΠζ͕Өڹ • zero-shot͕͍͠߹΄Ͳfew-shotͰੑೳ͕վળ • few-shotͷඅ༻ରޮՌߴ͍ 2 -
Cross-lingual Transfer ✦ ڭࢣ͋Γֶश (Supervised Learning) • ेͳαΠζͷσʔληοτΛڭࢣ৴߸ͱֶͯ͠श ✖︎ σʔληοτ͕५ͳݴޠҰѲΓ
• λεΫ͝ͱʹҟͳΔΞϊςʔτ͕ඞཁ… ➡ߴϦιʔεݴޠͷϥϕϧ͖σʔλΛར༻ͯ͠ϦιʔεݴޠͰ ͷਪΛߦ͍͍ͨʂ ✦ ݴޠؒసҠֶश (Cross-lingual Transfer Learning) • ݴޠԣஅతʹλεΫΛղ͘ྗΛ֫ಘ͢Δ 4 -
Cross-lingual Transfer ✦ Cross-Lingual Word Embeddings (CLWE) • ݴޠԣஅతͳ୯ޠຒΊࠐΈ ✦
Massively Multilingual Transformer networks (MMTs) • ଟݴޠίʔύεͰࣄલֶश͞ΕͨݴޠϞσϧ • mBERTɺXLM-R… • ݴޠؒసҠೳྗΛ࣋ͭ • FineTuning࣌ʹ༩͍͑ͯͳ͍ݴޠͰੑೳվળ • ྫ) ӳޠͰλεΫAʹରֶͯ͠श -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ 5 ▪ ӳޠ ▪ ΠλϦΞޠ CLWEจΑΓҾ༻ ݴޠԣஅతͳ୯ޠຒΊࠐΈۭؒ -
Massively Multilingual Transformers ✦ multilingual BERT (mBERT) • MLMͱNSPͰࣄલֶश •
Mask Language ModelingɿϥϯμϜʹϚεΫͨ͠จதͷτʔΫϯΛ༧ଌ • Next Sentence Predictionɿ̎ͭͷจ͕ྡΓ߹͍ͬͯΔ͔༧ଌ • 104ݴޠͷWikipediaΛֶशʹར༻ ✦ XLM RoBERTa (XLM-R) • MLMͷΈͰࣄલֶश • 100ݴޠͷCommonCrawlΛֶशʹར༻ • ΣϒςΩετ • mBERTΑΓେ͖͍ 6 XLM-RจΑΓҾ༻ -
Massively Multilingual Transformers ✓ ݴޠؒసҠೳྗΛ࣋ͭ ✦ Zero-shot • FineTuning࣌ʹ༩͍͑ͯͳ͍Ϋϥε (ݴޠ)Ͱਪ
• ྫ) ӳޠͰλεΫAʹରֶͯ͠श -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ ✦ Few-shot • FineTuning࣌ʹগྔͷڭࢣσʔλ (λʔήοτݴޠ)Λ ()ར༻ • ྫ) ӳޠͱগྔͷΠλϦΞޠͰλεΫAʹରֶͯ͠श -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ 7 ଟݴޠࣄલֶशʹΑͬͯ ֤ݴޠͷදݱۭ͕ؒྨࣅ͠ɺ ݴޠؒసૹ͕ՄೳͳͷͰʁ[1] <>&NFSHJOH$SPTTMJOHVBM4USVDUVSFJO1SFUSBJOFE-BOHVBHF.PEFMT -
Q1:ݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζݴޠؒసҠೳྗʹͲͷΑ͏ͳ ɹɹӨڹΛ༩͑Δ͔ʁ • ଟݴޠ୯ޠຒΊࠐΈ (CLWE)ྨࣅͨ͠ݴޠؒेͳίʔύε͕ଘࡏ ͢ΔݴޠͰͷΈੑೳΛൃش • CLWEͰ֤ݴޠͷ୯ޠຒΊࠐΈۭ͕ؒࣅ͍ͯΔ͜ͱ͕લఏͱͯ͠ඞཁ • ྫ)
ӳޠͱࣅ͍ͯΔݴޠ • υΠπޠɺεΧϯδφϏΞޠɺϑϥϯεޠɺεϖΠϯޠ • ಉ͡ޠ (ΠϯυɾϤʔϩούޠ)ʹଐ͢Δ ➡MMTsͷݴޠؒసҠೳྗͲ͏͔ʁ ✓ ෳݴޠؒͰͷݴޠؒసҠੑೳΛௐࠪ 9 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷੳ -
Q2:ԼҐλεΫͱݴޠؒసҠೳྗʹؔ͋Δ͔ʁ • POS-tagging (ࢺλά͚) • Dependency Parsing (ґଘߏղੳ) • Named
Entity Recognition (ݻ༗දݱநग़) • Natural Language Inference (ࣗવݴޠਪ) • Question Answering (࣭Ԡ) 10 ߴϨϕϧ (ݴޠཧղ)λεΫ ϨϕϧλεΫ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷੳ ✓ ෳͷλεΫͰθϩγϣοτݴޠؒసҠೳྗΛଌΔ -
Q2:ԼҐλεΫͱݴޠؒసҠೳྗʹؔ͋Δ͔ʁ ✦ POS-tagging (ࢺλά͚) • ςΩετதͷ୯ޠʹର͠ࢺΛϥϕϦϯά͢Δ ✦ Dependency Parsing (ґଘߏղੳ)
• จதͷґଘؔʹج͍ͮͯߏจղੳΛߦ͏ ✦ Named Entity Recognition (ݻ༗දݱநग़) • ςΩετ͔Βݻ༗දݱ (ਓ໊ɺ৫໊ɺ໊ɺ࣌ؒ……)Λநग़͢Δ ✦ Natural Language Inference (ࣗવݴޠਪ) • ̎ͭͷςΩετͷؒʹΓཱͭਪతؔ (ؚҙ/தཱ/ໃ६)Λ༧ଌ͢Δ ✦ Question Answering (࣭Ԡ) • จॻ͔Β࣭ͷճͱͳΔςΩετΛ୳ࡧ͢Δ 11 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷੳ -
Q4: few-shotʹίετΛࢧ͏͖͔ʁ • few-shotͷඅ༻ରޮՌͲͷఔͳͷ͔ʁ ✓ zero-shotͱfew-shotͷੑೳΛൺֱ ✓ few-shotͷΞϊςʔγϣϯʹ͔͔Δඅ༻Λྫࣔ 13 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷੳ
✦ Zero-shot • FineTuning࣌ʹ༩͍͑ͯͳ͍Ϋϥε (ݴޠ)Ͱਪ • ྫ) ӳޠͰλεΫAʹରֶͯ͠श -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ ✦ Few-shot • FineTuning࣌ʹগྔͷڭࢣσʔλ (λʔήοτݴޠ)Λ ()ར༻ • ྫ) ӳޠͱগྔͷΠλϦΞޠͰλεΫAʹରֶͯ͠श -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ + ੑೳ + ίετ -
ຊจͰѻΘΕΔओͳݴޠʹ͍ͭͯ ޠɿಉ͡ىݯΛ࣋ͭݴޠͷάϧʔϓ • ڞ௨ͷޠኮɺจ๏ɺԻӆମܥͳͲΛ࣋ͭ͜ͱ͕͋ΓɺࣅͨಛΛڞ༗͢Δݴޠ܈ • ΠϯυɾϤʔϩούޠ EN (ӳޠ), RU (ϩγΞޠ),
HI (ώϯσΟʔޠ), IT (ΠλϦΞޠ), SV (εΣʔσϯޠ), ES (εϖΠϯޠ), EL (ΪϦγϟޠ),DE (υΠπޠ), FR (ϑϥϯεޠ), BG (ϒϧΨϦΞޠ), UR (ϧυΡʔޠ), … • ΞϑϩɾΞδΞޠ AR (ΞϥϏΞޠ), HE (ϔϒϥΠޠ), … • ଞͷޠ TR (τϧίޠ): Φάζޠ܈, ZH (தࠃޠ): γφɾνϕοτޠ, EU (όεΫޠ): όεΫޠ, FI (ϑΟϯϥϯ υޠ): ϥϧޠ, JA (ຊޠ): ຊޠ, KO (ؖࠃޠ): ؖࠃޠ, VI (ϕτφϜޠ): ΦʔετϩΞδΞޠ , TH (λΠޠ): λΠɾΧμΠޠ, SW (εϫώϦޠ): χδΣʔϧɾίϯΰޠ, … 14 -
θϩγϣοτసҠೳྗͷ࣮ݧͱੳ ✓ ෳͷλεΫΛෳͷݴޠʹର͠θϩγϣοτͰਪ • Q1ͱQ2ʹճ͢Δ • Q1: ੑೳͱݴޠྨࣅੑɺࣄલֶशσʔλαΠζͷؔʹ͍ͭͯ • Q2:
ੑೳͱλεΫͷؔʹ͍ͭͯ • λεΫ: ݴޠ (ISO 639-1ܗࣜ) • POS(ࢺλά͚)ɿZH, TR, RU, AR, HI, EU, FI, HE, IT, JA, KO, SV • DEP (ґଘߏղੳ): ZH, TR, RU, AR, HI, EU, FI, HE, IT, JA, KO, SV • NER (ݻ༗දݱநग़): ZH, TR, RU, AR, HI, EU, FI, HE, IT, JA, KO, SV • NLI (ࣗવݴޠਪ): ZH, TR, RU, AR, HI,VI, TH, ES, EL, DE, ER, BG, SW, UR • QA (࣭Ԡ): ZH, TR, RU, AR, HI,VI, TH, ES, EL, DE 15 -
θϩγϣοτసҠೳྗͷ࣮ݧͱੳ ✓ ෳͷλεΫΛෳͷݴޠʹର͠θϩγϣοτͰਪ • λεΫ: σʔληοτ • POS(ࢺλά͚)ɿUniversal Dependency •
DEP (ґଘߏղੳ): Universal Dependency • NER (ݻ༗දݱநग़): NER WikiANN dataset [2] • NLI (ࣗવݴޠਪ): XNLI [3] • QA (࣭Ԡ): XQuAD [4] 16 <>.BTTJWFMZ.VMUJMJOHVBM5SBOTGFSGPS/&3 <>9/-*&WBMVBUJOH$SPTTMJOHVBM4FOUFODF3FQSFTFOUBUJPOT <>0OUIF$SPTTMJOHVBM5SBOTGFSBCJMJUZPG.POPMJOHVBM3FQSFTFOUBUJPOT -
θϩγϣοτసҠೳྗͷ࣮ݧͱੳ ✓ ෳͷλεΫΛෳͷݴޠʹର͠θϩγϣοτͰਪ • λεΫ: ධՁࢦඪ • POS(ࢺλά͚)ɿAccuracy • DEP
(ґଘߏղੳ): Unlabeled Attachment Scores (Γઌ͕ਖ਼͚͠Εྑ͍) • NER (ݻ༗දݱநग़): Accuracy • NLI (ࣗવݴޠਪ): Accuracy • QA (࣭Ԡ): Exact Match 17 -
θϩγϣοτసҠೳྗͷ࣮ݧͱੳ ✓ ӳޠͰͷ݁ՌͱͷࠩΛදࣔ • ࣄલֶशϞσϧ • B: mBERT base-cased •
X: XLM-R base • 19 -
θϩγϣοτసҠೳྗͷ࣮ݧͱੳ ✓ ݴޠྨࣅੑͷਪఆʹlang2vec͔ΒҎԼͷಛྔΛར༻ • Syntax: ߏจใΛදݱ ओޠͱಈࢺͷલޙؔͳͲ • Phonology:ԻӆతಛΛදݱ ࢠԻͱԻͷൺͳͲ
• Inventory: Իӆʹ͓͚Δࣗવྨ (ಛఆͷൃԻΛߦ͏Ի)ͷ༗ແ • Language families: ޠΛදݱ • Geography: ٿ্ͰͷڑΛදݱ ➡͜ΕΒͷίαΠϯྨࣅΛ͏͜ͱͰݴޠྨࣅੑΛਪఆ ✓ ࣄલֶश࣌ͷίʔύεαΠζΛඪ४Խͯ͠ಛྔSIZEͱͯ͠ར༻ 22 -
Few-shotసҠೳྗͷ࣮ݧͱੳ • Q4: few-shotʹίετΛ͔͚Δ͖͔ʁ ➡few-shotʹΑΔݴޠؒసҠੑೳͷมԽΛௐΔ • ӳޠʹՃ͑ͯkݸͷλʔήοτݴޠͷσʔλΛֶशʹར༻ 29 ✦ Zero-shot
• FineTuning࣌ʹ༩͍͑ͯͳ͍Ϋϥε (ݴޠ)Ͱਪ • ྫ) ӳޠͰλεΫAʹରֶͯ͠श -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ ✦ Few-shot • FineTuning࣌ʹগྔͷڭࢣσʔλ (λʔήοτݴޠ)Λ ()ར༻ • ྫ) ӳޠͱগྔͷΠλϦΞޠͰλεΫAʹରֶͯ͠श -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ + ੑೳ + ίετ -
Few-shotసҠೳྗͷ࣮ݧͱੳ • ιʔεݴޠͱݴޠྨࣅੑ͕͍ݴޠͷํ͕few-shotͷԸܙΛड͚͍͢ • zero-shotͰͷੑೳ͕ߴ͍ݴޠfew-shotͷԸܙΛड͚ʹ͍͘ (SV) • zero-shotͰͷੑೳ͕͍ݴޠfew-shotͰੑೳ͕େ্͖͘ (JA, AR)
32 ◀︎ few-shotࣄྫͱ ੑೳͷਪҠ mBERTΛ༻͍ͨ ґଘߏղੳλεΫ Ͱͷ݁Ռ -
Few-shotసҠೳྗͷ࣮ݧͱੳ Direct Target Language Few-Shot Fine-Tuning ӳޠͰͷλεΫͷֶशΛߦΘͣɺfew-shotͰͷΈֶश • ߴϨϕϧλεΫʹ͓͍ͯӳޠେنσʔλͰͷֶशΛল͘ͱ͍ੑೳ •
ιʔεݴޠͰͷλεΫֶशͱλʔήοτݴޠͰͷֶशɺ྆ํ͕ඞཁ • ϨϕϧλεΫʹ͓͍ͯӳޠͰͷֶशΛলུՄೳ • ΑΓ୯७ͳϨϕϧλεΫʹରͯ͠গͷσʔλͰֶशՄೳ 33 -
Q1:ݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζͲͷΑ͏ͳӨڹΛ༩͑Δ͔ʁ 34 ✓ సҠֶशͷੑೳݴޠྨࣅੑɺࣄલֶश࣌ͷίʔύεαΠζʹґଘ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷੳ • ଟݴޠ୯ޠຒΊࠐΈ (CLWE)ྨࣅͨ͠ݴޠؒेͳίʔύε͕ଘࡏ͢ΔݴޠͰͷΈੑೳΛൃش • CLWEͰ֤ݴޠͷ୯ޠຒΊࠐΈۭ͕ؒࣅ͍ͯΔ͜ͱ͕લఏͱͯ͠ඞཁ
• ྫ) ӳޠͱࣅ͍ͯΔݴޠ • υΠπޠɺεΧϯδφϏΞޠɺϑϥϯεޠɺεϖΠϯޠ • ಉ͡ޠ (ΠϯυɾϤʔϩούޠ)ʹଐ͢Δ ➡ MMTsͷݴޠؒసૹೳྗͲ͏͔ʁ ✓ ෳݴޠؒͰͷݴޠؒసૹੑೳΛௐࠪ -
Q2:ԼҐλεΫͱݴޠؒసҠೳྗʹؔ͋Δ͔ʁ • POS-tagging (ࢺλά͚) • Dependency Parsing (ґଘߏղੳ) • Named
Entity Recognition (ݻ༗දݱநग़) • Natural Language Inference (ࣗવݴޠਪ) • Question Answering (࣭Ԡ) 35 ߴϨϕϧ (ݴޠཧղ)λεΫ ϨϕϧλεΫ ✓ zero-shot, few-shotͷ྆ઃఆͰݴޠؒసҠੑೳͷ͕λεΫͷϨϕϧʹґଘ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷੳ -
Q3:ݴޠؒసҠೳྗͷ༧ଌՄೳ͔ʁ ✓ ಛఆͷλεΫͷసҠ݁Ռ͕ଘࡏ͢Εɺ؆୯ͳճؼΛ༻͍Δ͜ͱͰ৽͍͠ ݴޠͰͷసҠೳྗ͕͓͓·͔ʹ༧ଌՄೳ ✓ λεΫ͝ͱʹґଘ͢Δಛྔʹ͕ଘࡏ • ϨϕϧλεΫɿݴޠྨࣅੑʹґଘ • ߴϨϕϧλεΫɿݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζʹґଘ
36 • ݴޠͷྨࣅੑɺࣄલֶश࣌ͷίʔύεαΠζͳͲ͔ΒಛఆλεΫɺ ݴޠͰͷੑೳΛ༧ଌͰ͖Δ͔ʁ ※ ݴޠͷྨࣅੑ • lang2vecͰݴޠͷಛΛϕΫτϧԽ͠ɺͦͷίαΠϯྨࣅΛར༻ • ݴޠͷಛ͕ऩू͞Εͨσʔλϕʔε͔ΒಛྔΛநग़ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷੳ -
ΑΓେ͖ͳଟݴޠϞσϧͰͷθϩγϣοτݴޠؒసҠੑೳʹ͍ͭͯ ✓ ͜ͷจ2020ʹൃද͞Ε͓ͯΓɺݱͰঢ়گ͕ҟͳΔՄೳੑ͕͋Δ • จͰ࣮ݧʹ༻͍ΒΕ͍ͯΔϞσϧ • mBERT (base): 110M •
XLM-R (base): 270M • ݱࡏൃද͞Ε͍ͯΔେنଟݴޠϞσϧ • mT5-XXL[5]: 13B • Bloomz[6]: 176B 39 <>N5".BTTJWFMZ.VMUJMJOHVBM1SFUSBJOFE5FYUUP5FYU5SBOTGPSNFS <>$SPTTMJOHVBM(FOFSBMJ[BUJPOUISPVHI.VMUJUBTL'JOFUVOJOH -
ΑΓେ͖ͳଟݴޠϞσϧͰͷθϩγϣοτݴޠؒసҠੑೳʹ͍ͭͯ • XNLIλεΫʹ͓͍ͯmT5ͷzero-shotͰͷੑೳmBERTɺXLM-RͰͷ few-shotͰͷੑೳΑΓྑ͍ • ಉ༷ͷ݁ՌΛXQuADʹ͓͍ͯ֬ೝ 40 L L N#&35
9-.3 MBSHF mT5จΑΓҾ༻ ˛ ຊจͰࣔ͞Ε͍ͯͨXNLIͷੑೳ k = few-shot -
ݱʹ͓͚ΔConclusion • ιʔεݴޠ͔Βԕ͍ݴޠϦιʔεͳݴޠͰzero-shotਪ͍͠ • ͕ɺେ͖ͳϞσϧͰԕ͍ݴޠʹରͯ͠zero-shotਪ͕Մೳ • ϨϕϧλεΫͷసҠੑೳʹߏతͳݴޠྨࣅੑ͕େ͖͘Өڹ • ߴϨϕϧͷLUλεΫͷసҠੑೳʹࣄલֶशͷίʔύεαΠζ͕Өڹ •
zero-shot͕͍͠߹΄Ͳfew-shotͰੑೳ͕վળ • zero-shotੑೳ্͕͓ͯ͠Γɺfew-shotʹΑΔӨڹվΊͯݕ౼͍ͨ͠ • few-shotͷඅ༻ରޮՌߴ͍ 42