Share
Embed
Transcript
62 slides
-
ँࣙ • খྛᰜհࢯˏPreferred Networks / ౦େ • ߴᠳࢯˏ౦େ • ླ५ઌੜˏ౦େ
ʹຊߨԋͷςʔϚઃఆεϥΠυͷਪᏏʹ͝ڠྗ͍͖ͨͩ·ͨ͠ɻ ͜ͷΛआΓͯਂ͘ײँ͍ͨ͠·͢ɻ ໊ݹ۠NLPηϛφʔ 2022/06/07 3 -
͓͞Β͍ɿTransformer [Vaswani+2017] • Attention→FeedforwardΛͨ͘͞ΜੵΉ+৭ʑ • ܥྻมϞσϧࣄલ܇࿅ࡁΈݴޠϞσϧͷࠜװ • ࠷ۙԻɾը૾Ͱ׆ൃ • ∴
TransformerࣗମͷվྑΠϯύΫτେ • Change the worldͰ͖ΔՄೳੑ͕͋Δ ໊ݹ۠NLPηϛφʔ 2022/06/07 4 Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + -
ྑ͍Transformerɺੑೳͷߴ͍Transformer • ຊൃදͰओʹ͢͜ͱ • λεΫ൚༻తʹੑೳ্ΛͶΒ͏ํ๏ • ຊൃදͰओʹ͞ͳ͍͜ͱ • ಛఆͷλεΫɾσʔληοτʹணͨ͠ํ๏ •
ܭࢉޮΛ্͛ΔͨΊͷํ๏ • Ex) ύϥϝʔλݮ • Ex) ΞςϯγϣϯͷߴԽ • Ex) ྔࢠԽ ໊ݹ۠NLPηϛφʔ 2022/06/07 7 ͋͘·Ͱຊൃදͷఆٛ Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + -
ຊൃදͷߏɿ֤෦ҐͷऔΓΈΛհ ໊ݹ۠NLPηϛφʔ 2022/06/07 8 Multi-Head Attention Add & Norm Feed
Forward Add & Norm Input Embedding + ᶃ Ґஔදݱ ᶄ Attention ᶅ Feed-forward ᶆ Layer Normalization -
ઈରҐஔදݱ; Absolute Position Embedding (APE) • ֤ҐஔΛઐ༻ͷϕΫτϧΛͬͯදݱ͢Δ • e.g., sin/cosͷΈ߹Θͤ
[Vaswani+2017] • 😀 γϯϓϧɺ͍ɺ࣮͕؆୯ • Ґஔ୯ͳΔߦྻ͔ΒͷࣙॻҾ͖Ͱදݱ • 😰 ະͷ͞ͷܥྻʹରͯ͠൚Խ͠ͳ͍ • i.e., ֎ૠ͕ࠔ ໊ݹ۠NLPηϛφʔ 2022/06/07 10 (a) ASPEC English-to-Japanese (b) WMT2014 English-to-German Figure 3: BLEU scores on test data split by the sentence length (no training data in the gray-colored area). Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + John yelled at Kevin Position 0 1 2 3 4 Figure from [Neishi+2019] ະͷ͞ -
૬ରҐஔදݱ; Relative Position Embedding (RPE) • ֤τʔΫϯؒͷڑΛAttention෦Ͱར༻ • 😀 ະͷ͞ʹରͯ͠ؤ݈
• Ωʔϫʔυɿγϑτෆมੑ • 😰 ઈରҐஔΑΓܭࢉ͕͍࣌ؒ • 😰 AttentionͷѥछͱͷΈ߹Θ͕ͤࠔ • Performer, Linformer, etc… ໊ݹ۠NLPηϛφʔ 2022/06/07 11 Multi-Head Attention Add & Norm Feed Forward Add & Norm Input + -2 -1 0 1 2 Key Value Relative Distance Additional Features for Attention John yelled at Kevin ! !"# {%&',)*+,&} ! ."# {%&',)*+,&} Position 0 1 2 3 -
ઈରҐஔ vs ૬ରҐஔɿ͜͜·Ͱͷ·ͱΊ ໊ݹ۠NLPηϛφʔ 2022/06/07 12 ઈରҐஔ ૬ରҐஔ <৽نͷख๏> ੑೳ
😰 😀 ? 😀 😰 ? ࣮ίετ 😀 😰 ? ·ͣզʑͷ औΓΈΛհ -
SHAPE: Shifted Absolute Position Embedding • w/ Sosuke Kobayashi, Jun
Suzuki, and Kentaro Inui • EMNLP2021ʹ࠾ ໊ݹ۠NLPηϛφʔ 2022/06/07 13 Sosuke Kobayashi~,} Jun Suzuki~, Kentaro Inui~, ~ Tohoku University } Preferred Networks, Inc. [email protected], [email protected], jun.suzuki, inui}@tohoku.ac.jp t rucial for building ns in Transformers. ations suffer from test data with un- utational cost. We osition embedding ues. The basic idea invariance, which uccessful position y shifting absolute Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + John yelled at Kevin Position 0 1 2 3 4 k-1 k k+1 k+2 k+3 Absolute Position Embedding (APE) (a) Shifted APE (SHAPE) (c) Transformer Relative Position Embedding (RPE) (b) Relative Distance John yelled at Kevin ! !"# {%&',)*+,&} ! ."# {%&',)*+,&} -2 -1 0 1 2 Key Value Shifted by random offset k Figure 1: Overview of position representations. (a) APE -
ઈରҐஔʹγϑτෆมੑΛऔΓೖΕΔ • ૬ରҐஔະͷ͞ʹؤ݈→γϑτෆมੑ͕ॏཁͦ͏ ໊ݹ۠NLPηϛφʔ 2022/06/07 14 John yelled at Kevin
! !"# {%&',)*+,&} ! ."# {%&',)*+,&} Position 0 1 2 3 Position 34 35 36 37 John yelled at Kevin ! !"#!$ {&'(,*+,-'} ! !/#!$ {&'(,*+,-'} e.g. ૬ରҐஔͷ߹: ! !"# {%&',)*+,&} ! !"# {%&',)*+,&} ! !"#!$ {&'(,*+,-'} ! !"#!$ {&'(,*+,-'} à ্ۭؒͷҐஔγϑτ͕ؔͷग़ྗʹӨڹ͠ͳ͍…ͱ͍͏ੑ࣭ ͱ͍͠ -
ઈରҐஔ+ϥϯμϜγϑτ=γϑτෆมੑ • Shifted Absolute Position Embedding (SHAPE) • ઈରҐஔΛΦϑηοτ 𝒌~𝓤(𝟎,
𝑲) ͰϥϯμϜʹγϑτ • ϞσϧઈରҐஔΛར༻ෆՄೳ • ΘΓʹ૬ରҐஔΛར༻͢ΔֶशΛڧ੍ ໊ݹ۠NLPηϛφʔ 2022/06/07 15 Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + John yelled at Kevin Position 0 1 2 3 4 k-1 k k+1 k+2 k+3 APE Shifted by random offset k SHAPE -
Ұ୴ɺ͜͜·Ͱͷ·ͱΊ ໊ݹ۠NLPηϛφʔ 2022/06/07 16 ing ers. om un- We ing
dea ich ion ute Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + John yelled at Kevin Position 0 1 2 3 4 k-1 k k+1 k+2 k+3 Absolute Position Embedding (APE) (a) Shifted APE (SHAPE) (c) Transformer Relative Position Embedding (RPE) (b) Relative Distance John yelled at Kevin ! !"# {%&',)*+,&} ! ."# {%&',)*+,&} -2 -1 0 1 2 Key Value Shifted by random offset k -
༧උௐࠪɿSHAPEγϑτෆมੑΛ֫ಘ͢Δ͔ʁ • ֶशࡁΈϞσϧʹ͍ͭͯɺ𝑘 ͷΛมԽͤͯ͞ΈΔ • ֤ 𝑘 ʹ͍ͭͯΤϯίʔμͷίαΠϯྨࣅΛൺֱ • APE:
֤ 𝑘 ҟͳΔӅΕΛੜ • SHAPE: ֤ 𝑘 ͷӅΕͷʹӨڹ͠ͳ͍ ໊ݹ۠NLPηϛφʔ 2022/06/07 17 0 100 250 500 APE EmbeddingLayer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 0 100 250 500 0 100 250 500 SHAPE k 0 100 250 500 0 100 250 500 0 100 250 500 0 100 250 500 0 100 250 500 0 100 250 500 0.8 0.9 1.0 -
࣮ݧઃఆͷ֓ཁ • Ϟσϧ: Transformer + APE, RPE, or SHAPE •
λεΫ:ػց༁ • ܇࿅σʔλ 1. Vanilla • WMT 2016 EnDe [Ott+2018] 2. Extrapolate • ͞50Ҏ্ͷܥྻΛ܇࿅σʔλ͔Βআͯ͠࡞ 3. Interpolate • ྡ͢ΔܥྻΛ݁߹ͯ͠࡞ • ։ൃσʔλ: newstest2010-2013 • ςετσʔλ: newstest2014-2016 • ධՁ: sacreBLEU ໊ݹ۠NLPηϛφʔ 2022/06/07 18 ࠓճলུ ઈରҐஔ ૬ରҐஔ -
࣮ݧઃఆɿσʔληοτ ໊ݹ۠NLPηϛφʔ 2022/06/07 19 ط&ະͷ͞ ͰධՁ͍ͨ͠ ᶃ Vanilla: WMT EnDe
2016 [Ott+2018] English German • ػց༁ͷඪ४తͳϕϯνϚʔΫ • ϕʔεϥΠϯͷੑೳͷ֬ೝ༻ ᶄ Extrapolate: ͞50Ҏ্ͷܥྻΛ܇࿅σʔλ͔Βআ English German • Ϟσϧͷ֎ૠੑೳͷ֬ೝ • ະͷ͞ʹରͯ͠ؤ݈͔ʁ -
࣮ݧ݁ՌɿRPEͱSHAPEಉͷੑೳ • Extrapolateͷ݁Ռ • RPEͱSHAPE͕APEΛ্ճΔੑೳ • SHAPERPEͱಉͷੑೳͰɺRPEΑΓߴ • Vanillaͷ݁Ռ •
શͯͷϞσϧ͕ಉͷੑೳ • SHAPEੑೳѱԽͷڪΕͳͬͯ͘ྑͦ͞͏ ໊ݹ۠NLPηϛφʔ 2022/06/07 20 Figure 2: Cosine similarities of the encoder hidden states with different offsets k 2 {0, 100, 250, 500}. Only the representation of SHAPE is invariant with k. Dataset Model Valid Test Speed VANILLA APE† 23.61 30.46 x1.00 RPE† 23.67 30.54 x0.91 SHAPE† 23.63 30.49 x1.01 EXTRAPOLATE APE 22.18 29.22 x1.00 RPE 22.97 29.86 x0.91 SHAPE 22.96 29.80 x0.99 Table 2: BLEU scores on newstest2010-2016. Valid is Figure 3: BLEU score improveme dation and test sets with respect to length. The gray color means no t -
͞͝ͱͷੑೳΛௐࠪɿ֎ૠੑೳ্͕ ໊ݹ۠NLPηϛφʔ 2022/06/07 21 0-9 10-19 20-29 30-39 40-49 50-59
60-69 70- Source Sequence Length (tokens) 0 2 4 6 8 BLEU improvement APE RPE SHAPE ϕʔεϥΠϯ "1& ͔Βͷੑೳ্෯ ະͷ͞ κʔϯ • SHAPEͱRPEAPEΑΓߴ͍֎ૠੑೳΛൃش • SHAPEͱRPEಉͷੑೳ -
SHAPEͷ·ͱΊ • SHAPE : shifted absolute position embedding • APEʹγϑτෆมੑΛ༩
• APEͱಉͷ • RPEͱಉͷੑೳ • ࣮؆୯ ໊ݹ۠NLPηϛφʔ 2022/06/07 22 ॏཁ SHAPEͷ࣮ྫ ઈରҐஔ ૬ରҐஔ SHAPE ੑೳ 😰 😀 😀 😀 😰 😀 ࣮ίετ 😀 😰 😀 -
ͦͷ΄͔ɺ࠷ۙͷҐஔදݱͨͪ • ALiBi: Attention with Linear Biases [Press+2021] • RoPE:
Rotary Position Embedding [Su+2021] • CAPE: Continuous Augmented Positional Embeddings [Likhomanenko+2021] • TUPE: Transformer with Untied Positional Encoding [Ke+2020] • ࠓճলུ ໊ݹ۠NLPηϛφʔ 2022/06/07 23 Q. શ෦ಉ͡͡Όͳ͍Ͱ͔͢ʁ A. ͕͍ͪ·͢… -
ALiBi: Attention with Linear Biases [Press+2021] • Query-Keyͷੵʹ͍ͭͯڑʹ Ԡͨ͡ϖφϧςΟΛ༩ •
ԕ͍τʔΫϯ΄ͲAttentionͷ είΞ͕Լ͕Δ • ۙΛ༏ઌ͢ΔΑ͏ͳInductive biasΛೖΕ͍ͯΔͱΈͳͤΔ • APEΑΓੑೳྑ͍ • ֎ૠੑೳྑ͍ • BigScienceWSͷϞσϧͰ࠾༻ • [Le Scao+2022] ໊ݹ۠NLPηϛφʔ 2022/06/07 24 Figures from [Press+2021] -
RoPE: Rotary Position Embedding [Su+2021] • తɿQuery-KeyͷੵΛ ୯ޠؒͷ૬ରڑʹґଘͤ͞ ͍ͨ •
ҐஔʹԠͯ͡୯ޠϕΫτϧΛ 𝑚𝜃͚ͩճస • Query-Keyͷ૬ରڑ͕ಉ͡ →ͳ͕֯͢ಉ͡ • ઈରҐஔΛ্ճΔੑೳ • GPT-JGPT-NeoXPaLMͰ ࠾༻ ໊ݹ۠NLPηϛφʔ 2022/06/07 25 Figures from [Su+2021] -
CAPE: Continuous Augmented Position Embeddings • APEΛྑ͍ͨ͘͠ൊ • ܇࿅தʹҎԼͷૢ࡞ΛऔΓೖΕΔͱྑ͍ •
(MPCBM4IJGU ೖྗશମͷҐஔΛϥϯμϜʹͣΒ͢ • Local Shift (ೖྗҰ෦ͷҐஔΛϥϯμϜʹͣΒ͢) • Global Scaling (ॖईΛϥϯμϜʹௐ) • ը૾ɺԻɺݴޠλεΫͰੑೳ্Λใࠂ ໊ݹ۠NLPηϛφʔ 2022/06/07 26 Ͳ͔͜Ͱݟ͕֮͑… [Likhomanenko+2021] Figure from [Likhomanenko+2021] -
͓͞Β͍ɿAttention • Transformerͷ֩৺ʁ • ՄมܥྻͷதͰ͍͍ײ͡ʹใΛࠞͥΔͨΊͷΈ • ڑґଘͷֶश͕͍͢͠ʢͱ͞Ε͍ͯΔʣ • ฒྻԽͰ͖ΔͷͰRNNΑΓߴ •
ͨͩ͠ڑʹͳΔͱܭࢉ͕େม • ܥྻYܥྻ ͷܭࢉ͕ඞཁ Process Output array Q x L V Attention K V Attention Q K V V K Q Attention scores O E ͜Ε͕Qͱಉ͡ͱ͖ Self-Attention Figure from [Jaegle+2022] -
AttentionΛऔΓר͘࠷ۙͷঢ়گ • AttentionΛผͷԿ͔Ͱஔ͖͍͑ͨ • CNN [Wu+2019] • MLP [Sun+2021] •
AttentionΛݮΒ͍ͨ͠ • ΘΓʹFeed-ForwardΛ૿͢ʁ [Zhang+2020] [Irie+2020] • AttentionࣗମΛܰྔԽ͍ͨ͠ • Efficient Transformers: A Survey • AttentionͷಛΛͬͱ׆͔͍ͨ͠ • Perceiver [Jaegle+2021] • Memory Tranformer [Wu+2022] ໊ݹ۠NLPηϛφʔ 2022/06/07 30 -
Dynamic Convolution [Wu+2019] • AttentionΛCNNͰஔ͖͑Α͏ܥͰ࠷༗໊ʁ • ֤࣌ࠁʹ͍ͭͯೖྗ͔ΒCNNͷΧʔωϧΛ ಈతʹੜ • MTɾཁɾݴޠϞσϧͰAttentionΑΓߴੑೳ
• ύϥϝʔλڞ༗ͳͲͷࡉ͔͍ςΫχοΫ͕ॏཁͦ͏ • ࠷େΧʔωϧ෯31→ܥྻશମΛݟΔͷ͕େࣄʁ ໊ݹ۠NLPηϛφʔ 2022/06/07 31 Figure from [Wu+2019] ਤ จհ: Pay Less Attention with Lightweight and Dynamic Convolutions ΑΓҾ༻ -
Pretrain-finetuneઃఆʹ͓͚ΔCNN [Tay+2022] • ٙɿPretrain-finetuneͷԸܙTransformer͚ͩͷͷ͔ʁ • TransformerͱCNN3छྨΛൺֱɿ6/7λεΫͰCNN͕TransformerΛ্ճΔੑೳ • ͨͩ͠ɺCNNଆ3छྨ͋Δͷʹҙ • ͨͩ͠ɺ࣭ԠؚҙؔೝࣝͳͲɺෳจΛ༻͍ΔλεΫͰ
CNN < Transformer • CNNͰจؒͷใͷΓऔΓ͕ࠔ ໊ݹ۠NLPηϛφʔ 2022/06/07 32 Table from [Tay+2022] -
Perceiver [Jaegle+2021] • Perceiver: ݴޠɾԻɾը૾Ͱ൚༻తʹ͑ΔϞσϧΛAttentionͰ࣮ݱ • 😩 ʮ1࣍ݩͳΒLSTMɺ2࣍ݩͳΒCNN…ʯΛղܾ • جຊతʹTransformer
Encoder • ڑܥྻ͕ͭΒ͍ͷͰɺೖྗΛCross-Attentionܦ༝Ͱݻఆʹ • ը૾ྨɺԻ+ಈըղੳɺ܈ྨͳͲͰڧ͍ϕʔεϥΠϯʹඖఢ • ͨͩ͠ɺྨ͔͠ղ͚ͳ͍ߏͳͷʹҙ ໊ݹ۠NLPηϛφʔ 2022/06/07 33 M>>>N ͳͷͰܰྔ ResNetͱ͔ ViTͱ͔ ೖྗ ύϥϝʔλ ߦྻ Figure from [Jaegle+2021] -
Perceiver IO [Jaegle+2022] • Perceiver IOɿPerceiverͷग़ྗଆʹCross AttentionΛ࠾༻ • ೖग़ྗʹରশੑΛ͍࣋ͨͤͯΔͱݟ͑Δ •
͞𝒐ͷܥྻΛग़ྗՄೳàద༻ՄೳλεΫ͕૿Ճ • ϚεΫݴޠϞσϧؚΉ֤छλεΫͰ൚༻తʹಈ࡞ • ग़ྗଆͷQuery ArrayͰະདྷͷใ͕ϦʔΫ͍ͯ͠Δͷʹҙ • ͳͷͰɺͦͷ··Ͱੜʹ͑ͳ͍ ໊ݹ۠NLPηϛφʔ 2022/06/07 34 ৽ن ύϥϝʔλߦྻ ೖྗΛݟΔͷ Ұճ͚ͩ Figure from [Jaegle+2022] -
Perceiver AR [Hawthorne+2022] • ະདྷͷใ͕ϦʔΫ͢ΔͷΛ͍͗ͨ • ग़ྗଆͷAttentionʹϚεΫػߏΛՃ͠ɺ autoregressiveͳੜΛՄೳʹ • ͍ͭͷCausal
Mask • ը૾ੜɺݴޠϞσϧͳͲͰߴੑೳ • Transformer-XLͳͲʹඖఢ • ܥྻ͕͘ͳֶͬͯशߴ • ܥྻมλεΫͰಈ͘ͷ͔Α͔͘Βͳ͍… ໊ݹ۠NLPηϛφʔ 2022/06/07 35 Figures from [Hawthorne+2022] -
Memorizing Transformers [Wu+2022] • Transformer DecoderΛఆ • ઐ༻ͷAttentionΛઃܭ • աڈͷӅΕΛ֎෦ϝϞϦͱ
Έͳͯ͠AttentionΛܭࢉ • AttentionείΞΛݩʹkNNͰ దʹࢬמΓ • ࠷ޙʹී௨ͷAttentionͱࠞͥΔ • ֶه߸ͦͷఆٛΛ ͏·͘ੜͰ͖Δ • ୯ͳΔසޠͷίϐʔʁ • ؔ࿈ɿ[Sun+2021] ໊ݹ۠NLPηϛφʔ 2022/06/07 36 ޯྲྀΕͳ͍ →ߴʹಈ࡞ Figures from [Wu+2022] -
͓͞Β͍ɿFeedForward • શ݁߹+Activation+શ݁߹ • ׳शɿ࣍ݩΛ্͛ͯݩʹ͢ • Transformer-base: 512à2048à5412 • Transformer-big:
1024à4096à1024 • ࣮ࡍʹԿΛ͍ͬͯΔʁ • ֎෦ϝϞϦͷAttentionͱΈͳͤΔ • [Sukhbaatar+2019] • Key-ValueϝϞϦͱͯ͠ಈ࡞ • [Geva+2021] ໊ݹ۠NLPηϛφʔ 2022/06/07 38 Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + Linear Activation Function Linear X ࣍ݩΛ্͛Δ ʙ4ഒఔ ݩʹ͢ -
࠷ۙͷਐలɿ׆ੑԽؔΛม͑ͯΈΔ • Vanilla TransformerͰReLUΛ࠾༻ • GPTBERTͳͲͷݴޠϞσϧɿGeLUΛଟ͘ݟ͔͚Δ • ॳग़GPTͷͣ • GLUܥ͕ޮ͘ͱ͍͏ใࠂ
[Shazeer+2020] • Կނྑ͍ͷ͔͔Βͳ͍͕… • “We offer no explanation as to why these architectures seem to work” ໊ݹ۠NLPηϛφʔ 2022/06/07 39 Figure from [Hendrycks+2016] Figure from [Narang+2021] -
࠷ۙͷਐలɿ࣍ݩΛ্͛Δ • WMT’19 ʹ͓͚ΔFacebookͷใࠂ[Ng+2019] • ৭ʑࢼͨ݁͠ՌɺFFNΛ૿͢ͱྑ͔ͬͨ • We experiment with
increasing network capacity by increasing embed dimension, FFN size, number of heads, and number of layers. • We find that using a larger FFN size (8192) gives a reasonable improvement in performance while maintaining a manageable network size • WMT’20 ࢀՃ࣌ʹࢼɾ࠾༻ [Kiyono+2020] ໊ݹ۠NLPηϛφʔ 2022/06/07 40 Table from [Ng+2019] -
දݱྗͷͦͷઌɿMixture of Experts • ී௨ʹϞσϧΛେ͖͍ͯ͘͘͠ͱ… • 😨 දݱྗUPͷͨΊύϥϝλ૿→Ϟσϧͷܭࢉ࣌ؒ૿ • ҰํɺMixture
of Expertsͷ߹ • 😀 ෳͷHomogenousͳExpertΛ༻ҙ→ύϥϝλ૿ɾදݱྗUP • 😀 ೖྗʹԠͯ͡Ұ෦ΛൃՐ→ܭࢉ࣌ؒݮ • Ұ෦ք۾ʢGoogleͱFacebookʣͰྲྀߦ͍ͬͯΔ༷ࢠ • Switch Transformer [Fedus+2021], Base Layers [Lewis+2021], Sparse all-MLP [Yu+2022] ໊ݹ۠NLPηϛφʔ 2022/06/07 41 Figure from [Fedus+2021] -
͓͞Β͍ɿLayer Normalization (LN) • ֤ӅΕ͔ΒฏۉͱࢄΛܭࢉ →ਖ਼نԽ • ֶशͷ҆ఆԽʹߩݙ • LNͳ͠ͷTransformerֶशࠔ
• AttentionͱFeedForwardϒϩοΫ ʹҰͭͣͭஔ ໊ݹ۠NLPηϛφʔ 2022/06/07 43 Multi-Head Attention Add & Norm Feed Forward Add & Norm Input Embedding + ຒΊࠐΈʹ LayerNorm͢Δ ྲྀ ֶश҆ఆ͢Δ͕ ੑೳ͕ѱԽ͢Δ [Le Scao+2022] LNϙΠϯτᶃ LNϙΠϯτᶄ Figure from [Zhang+2019] RNNͷ߹ -
LNΛͲ͜ʹஔ͔͘ʁɿPost-LNͱPre-LN • Vanilla TransformerPost-LN • ࠷ۙͷTransformerPre-LN͕ଟ͍ • ༨ஊɿPre-LNͷॳग़Tensor2Tensor • ͜ͷ͋ͱจ͕ͨ͘͞Μग़ͨ
໊ݹ۠NLPηϛφʔ 2022/06/07 44 Post-LN ResidualޙʹLN Pre-LN ؔͷલʹLN ͦͷޙPre-LN͕ σϑΥϧτʹ ˠͱͯॏཁ Ռͨͯ͠LNͷҐஔ͕ͦΜͳʹॏཁͳͷ͔ʁ Figure from [Xiong+2020] -
Post-LNͱPre-LNͷ·͍ؔ͠ ໊ݹ۠NLPηϛφʔ 2022/06/07 45 2 st-LN e-LN ual 後に Norm
本研究の貢献 ・Post-LN と Pre-LN の性能差を実験的に⽰す ・多層 Post-LN の学習が難しい原因を⽰す ・⾼い性能を維持しつつ多層化する⼿法を提案 性能 多層化 Post-LN ◦ × Pre-LN × ◦ B2T(提案⼿法) ◦ ◦ Layer Norm Attention FFN Layer Norm Layer Norm Attention FFN Layer Norm Layer Norm Attention × N × N Attention Layer Norm Layer Norm FFN Layer Norm (a) Post-LN er Norm ention Layer Norm Attention Attention Layer Norm Attention Layer Norm Layer Norm Attention Layer Norm Attention (a) Post-LN (b) Pre-LN (c) Post-LN with B2T connection ention er Norm er Norm Attention Layer Norm Layer Norm FFN Attention Layer Norm × N FFN er Norm Layer Norm Attention FFN Layer Norm Layer Norm Attention FFN Layer Norm Layer Norm Attention × N (b) Pre-LN (c) Post-LN with B2T connection 適⽤前に Norm ΈɿPre-LNֶश͕༰қ͕ͩੑೳ͕͍ 1PTU-/ͷ΄͏͕ੑೳ͕ߴ͍ Pre-LN で良いのでは︖ Pre-LN は多層にしたときの学習が安定 – 近年の多層なモデル(例︓GPT)はPre-LN しかし Pre-LN は性能が低い – Post-LN は学習が成功した場合 Pre-LN より⾼性能 • 6層 Transformer エンコーダ・デコーダの性能⽐較 7 Post-LN Post-LN Pre-LN Pre-LN 翻訳(WMT英-独)でのBLEU ⾒出し⽂⽣成でのROUGE-1 ※ߴᠳࢯͷNLP2022ൃදεϥΠυΑΓҾ༻ 1PTU-/ͷଟԽ͍͠ エンコーダ側 デコーダ側 各層の勾配のノルム デコーダ側で勾配消失が発⽣ 101 100 10-1 10-1 100 -
ޯফࣦͷݪҼLN • 勾配消失の原因を探る – デコーダの18層⽬の各位置における勾配のノルムを調べる Layer Norm Attention FFN Layer
Norm Layer Norm Attention (4) → (3),(2) → (1) で勾配が⼤きく減衰 → LN をまたぐと勾配が⼤きく減衰 → LN が勾配を減衰させる → LN が勾配消失の原因 ໊ݹ۠NLPηϛφʔ 2022/06/07 47 ※ߴᠳࢯͷNLP2022ൃදεϥΠυΑΓҾ༻ -
B2T Connection [Takase+2022] Ͱྑ͍ͱ͜औΓΛ࣮ݱ ໊ݹ۠NLPηϛφʔ 2022/06/07 48 Layer Norm Attention
FFN Layer Norm Layer Norm Attention FFN Layer Norm Layer Norm Attention × N × N (c) Post-LN with B2T connection Pre-LNɼ(c) Post-LN ʹఏҊख๏ΛΈ 提案⼿法︓⾚字の Residual Connection を追加 各層の下から上までの経路︓Bottom-to-Top Connection 1. 各層の最後の LN 以外を迂回する → LN による勾配の減衰を抑制 2. 最後の LN は⼊⼒にも適⽤ ※ߴᠳࢯͷNLP2022ൃදεϥΠυΑΓҾ༻ -
ൺֱ࣮ݧɿػց༁λεΫ ໊ݹ۠NLPηϛφʔ 2022/06/07 49 ※ߴᠳࢯͷNLP2022ൃදεϥΠυΑΓҾ༻ B2T connection は 学習に成功 性能も他の⼿法より⾼い
• あ 学習 失敗 6層エンコーダ・デコーダ 18層エンコーダ・デコーダ Post-LN は Pre-LN よりも⾼い性能 B2T connection は Post-LN と同程度の性能 -
LNΛফͯ͠ޯফࣦʹରॲɿT-FixUp [Huang+2020] ໊ݹ۠NLPηϛφʔ 2022/06/07 50 • T-FixUpɿ͔ͳΓݡ͍ॳظԽ • LNΛϞσϧશମ͔ΒຣফͰ͖Δ •
ֶशͷWarmup͕ཁΒͳ͘ͳΔ • ଟʹֶͯ͠श͕҆ఆ͢Δ • Warmupͳͥඞཁ͔ͩͬͨʁ • LNͷޯೖྗͷϊϧϜʹൺྫ • ֶशॳظAdamͷਪఆྔ͕ෆ҆ఆ→ߋ৽ྔ📈 • ݁ՌɺೖྗͷϊϧϜ͕📈 • ޯ͕ফֶ͑ͯशࠔʹ • 👍ϋΠύϥ (Warmup) ͕ফͤΔ • 👎 Կεςοϓ͔ΒֶशΛԼ͛Δ͔ʁ ͱ͍͏ϋΠύϥ͕૿͍͑ͯΔ • ϓϥϚΠθϩͰʁ Figure from [Huang+2020] ޯͷ༷ࢠ -
͋͑ͯLNΛ૿͢ɿNormformer [Shleifer+2021] • LNΛ૿͢ͱऩଋͱੑೳ͕վળ͢Δ ͱ͍͏ݚڀ • େنݴޠϞσϧͰͷ࣮ݧͰൺֱ • Pre-LNʹ͓͚Δͷޯͱ ߴͷޯͷεέʔϧͷෆҰகΛࢹ
• AttentionͱFeedForwardޙʹLNͯ͠ εέʔϧΛҰகͤ͞Δ • εέʔϧͷෆҰக͕ͳͥͳͷ͔ʁ • ΈΜͳͬͯΔσʔλ͕ҧ͏ɿ֤ख๏ͷ ൺֱ͕ࠔ • ࠞಱͱ͍ͯ͠Δ ໊ݹ۠NLPηϛφʔ 2022/06/07 51 Figure from [Shleifer+2021] -
Sandwitch Transformer [Press+2019] ໊ݹ۠NLPηϛφʔ 2022/06/07 53 Multi-Head Attention Add &
Norm Feed Forward Add & Norm Input Embedding + ΞΠσΞ (Attention+FeedForward)ΛNੵΉɺຊʹ࠷ద͔ʁ →֤αϒϨΠϠՄͰ͋Δ͜ͱʹண͠ɺ ॱ൪ΛೖΕସ͑ͯΈΔ -
Sandwitch Transformer [Press+2019] • ॱ൪ΛϥϯμϜʹೖΕସ͑ͯΈ࣮ͯݧ • ϕʔεϥΠϯΑΓྑ͍ߏ͕͋ΔͬΆ͍ • ೖྗଆʹAttentionଟΊɺग़ྗଆʹFFNଟΊ͕ ྑͦ͞͏
• ͭ·Γ • ݴޠϞσϧͰੑೳ্ɺ༁ͰޮՌͳ͠ ໊ݹ۠NLPηϛφʔ 2022/06/07 54 which is comparable to the performance of the base- ine with the same number of parameters. We next generalize this model and the original nterleaved transformer, creating the family of sand- wich transformers. A sandwichn k transformer con- ists of 2n sublayers in total (n of each type), con- orming to the regular expression sk(sf)n k fk. The first k sublayers are purely self-attention (s), while the last k are feedforward sublayers (f). In etween, we use the original interleaving pattern sf) to fill the remaining 2(n k) sublayers. When k = 0, we get the original transformer model, and when k = n 1 (its maximal value) we get the Model Test Baseline (Baevski and Auli, 2019) 18.70 Transformer XL (Dai et al., 2019) 18.30 kNN-LM (Khandelwal et al., 2019) 15.79 Baseline (5 Runs) 18.63 ± 0.26 Sandwich16 6 17.96 Table 3: Performance on the WikiText-103 test set. We compare the best sandwich transformer to the unmod- ified, interleaved transformer baseline (Baevski and Auli, 2019) trained over 5 random seeds and to other previously reported results. than the average baseline transformer. Of those, 6 models outperform the best baseline transformer (k = 5, 6, 8, 9, 10, 11). The best performance of 17.84 perplexity is obtained when k = 6. We com- pare this model to the baseline on WikiText-103’s test set. Table 3 shows that, despite its simple design, the sandwich transformer outperforms the original transformer baseline by roughly double the gap be- tween the baseline (Baevski and Auli, 2019) and Transformer XL (Dai et al., 2019). This improve- ment comes at no extra cost in parameters, data, memory, or computation; we did not even change any of the original hyperparameters, including the number of training epochs. To check whether this advantage is consistent, we train 4 more sandwich16 6 models with different Figure 5: The transformer’s sandwich coefficient (k) and validation perplexity, for k 2 {1, . . . , 15}. The dotted line is the average baseline model’s perplex- ity (trained with different random seeds), whereas the dashed line represents the best baseline model. Figure 6: Performance on the WikiText-103 develop- ment set of the Sandwich16 transformer and the base- Figures from [Press+2019] ଠࣈ͕Baseline ͍΄Ͳྑ͍ੑೳ -
Primer [So+2021] • TransformerͷΞʔΩςΫνϟΛҨతΞϧΰϦζϜͰ୳ࡧ • ୳ࡧۭؒΛAdd/Square/Conv…ͳͲͷϨϕϧԋࢉͰߏ • ൚༻తʹྑ͔֦ͬͨு2ͭ 1. Query,
Key, Valueʹରͯ͠෯3ͷDepthwise Convolution (DConv) 2. ReLUͷग़ྗΛೋ͢Δ • DConvͷؾ࣋ͪʹͳͬͯΈΔ • AttentionͷલʹۙΛݟ͍ͨ • Ͱී௨ͷConv͡ΌͩΊ • ෯Λͯͩ͘͠Ί • αϯϓϧޮ্͕͕Δͱͷ͜ͱ ໊ݹ۠NLPηϛφʔ 2022/06/07 55 νϟωϧؒͰ ใΛࠞͥͳ͍Conv Figure from [So+2021] -
ʮྑ͞ʯ࣮Λ·͍ͨͰ൚Խ͢Δ͔ʁɿNo • Do Transformer Modifications Transfer Across Implementations and Applications?
[Narang+2021] • ͞·͟·ͳTransformerͷ֦ுΛڞ௨ͷίʔυϕʔεʹ࠶࣮ • ݩจͷใࠂ͢Δݟʢi.e., ྑ͞ʣΛ࠶ݱͰ͖Δ͔ࢼ • ࠶ݱͰ͖Δख๏ҎԼͷ͍ͣΕ͔ͱใࠂ • γϯϓϧͳมߋʢ׆ੑԽؔͳͲʣ • ܭࢉྔͷ૿ՃΛ͏ͷʢଟԽͳͲʣ • ύϥϝʔλͷ૿ՃΛ͏ͷʢ.JYUVSFPG&YQFSUTͳͲʣ • ϋΠύϥͷௐΛ͍ͯ͠ͳ͍͜ͱʹҙ • ֦ுख๏ͨͪʹͱͬͯएׯෆརͳઃఆͱΈͳͤΔ • ϋΠύϥͷௐ͕ඞཁͳ࣌ͰμϝͰ͠ΐɺͱ͍͏ڧ͍ओு ໊ݹ۠NLPηϛφʔ 2022/06/07 57 ͩΊ͡ΌΜ… -
ʮྑ͞ʯσʔλʹؔͯ͠൚Խ͢Δ͔ʁɿNo • The Impact of Positional Encodings on Multilingual Compression
• [Ravishankar+2021] • Ґஔදݱख๏ʢઈରҐஔɺ૬ରҐஔʣΛม͑ͯଟݴޠBERTΛ܇࿅ • ଟݴޠλεΫʢ୯ޠΞϥΠϯϝϯτɾจநग़ͳͲʣͰධՁ • ઈରҐஔ͕૬ରҐஔΛ্ճΔੑೳ • ૬ରҐஔ͕ଟݴޠؒͷΞϥΠϯϝϯτͷֶशΛࠔʹ͍ͯ͠Δʁ ໊ݹ۠NLPηϛφʔ 2022/06/07 58 Figure from [Ravishankar+2021] -
ʮྑ͞ʯϞσϧͷεέʔϧʹؔͯ͠൚Խ͢Δ͔ʁɿNo • Transformerͷѥछʹ͍ͭͯϞσϧͷεέʔϧΛม͑ͨͱ͖ͷੑೳΛൺֱ • Vanilla, Universal Transformer, MLP-Mixer, ALBERT, Performer,
etc… • ݁ՌɺVanilla Transformer͕Ұ൪ྑ͔ͬͨ • 😀 ಛఆͷεέʔϧͰTransformerΑΓྑ͍Ϟσϧଘࡏ • 😨 TransformerΑΓҰ؏ͯ͠ྑ͍ͷଘࡏ͠ͳ͍ ໊ݹ۠NLPηϛφʔ 2022/06/07 59 [Anonymous+2021] ARRʹొͯ͠Ҏ߱ Իࠫଡͳ͕ͩ͠… Figure from [Anonymous+2021] -
ؔ࿈ɿ175BݴޠϞσϧʹ͓͚ΔMetaͷݟ • খن࣮ݧ͔ΒಘͨϋΠύϥ͕௨༻͠ͳ͍ • ͪͳΈʹɿ13Bύϥϝλ”খن” • ֶशΛGPT-3ઃఆΑΓେ͖ͯ͘͠Έ͚ͨͲେࣦഊ • طଘͷϞσϧ͕௨༻͠ͳ͍ •
Normformerɿֶश్தͰloss͕ఀ • ֶशͷ҆ఆԽ͕ॏཁ • FP16ͩͱGeLU͕తʹ҆ఆ͠ͳ͍ à ReLUͰଥڠ • EmbeddingʹLN͢Δͱֶश͕҆ఆ͢Δ͚Ͳੑೳ͕ѱ͍ • ؤுͬͯઍUpdate͝ͱʹloss͕ൃࢄ͢Δ à ͦͷrestart͕ඞཁ ໊ݹ۠NLPηϛφʔ 2022/06/07 60 https://github.com/facebookresearch/metaseq/blob/ main/projects/OPT/chronicles/56_percent_update.md LogBookҰಡͷ Ձ͋Γ -
༨ஊɿWMT 2020ͷܦݧ͔Β • γεςϜ։ൃͷ࠷தʹ͋Ε͜Εطଘख๏Λࢼ͢ • େମ͏·͍͔͘ͳ͍ • ྫɿٙࣅσʔλͷϑΟϧλϦϯά • ԯจͷٙࣅσʔλશ෦Ͱֶशͨ͘͠ͳ͍→ྑ͍σʔλΛऔΓग़͍ͨ͠
• ϑΟϧλϦϯάͷ༗ແʹΑΒͣੑೳʹӨڹͳ͠ ໊ݹ۠NLPηϛφʔ 2022/06/07 61 ਗ਼ͷࢀՃͨ͠ ػց༁ͷίϯϖ Setting En!De De!En En!Ja Ja!En ASE (Section 3.1) 42.4 42.0 19.7 21.6 ASE (l = 9)+TAGGED-BT (Section 3.3) 42.7 42.5 22.0 23.9 b) + fine-tuning (Section 3.4) 44.9 42.3 23.1 24.4 c) ⇥ 4 (Section 3.5) 45.5 42.8 23.9 25.4 d) + 4 ⇥ (c)-R2L (Section 3.6) 45.4 43.6 24.2 25.9 e) + reranking (Section 3.7) 45.7 43.8 24.9 26.2 he best system in WMT’19 44.9 42.8 - - of each technique: we use newstest2019 and official validation set for En$De and En$Ja result from WMT’19 is unavailable for En$Ja, because this task has newly appeared this g / ID BLEU chrF able 5) 37.5 0.647 able 5) 43.8 0.690 able 5) 40.1 0.343 ENSEMBLE 25.5 0.536 e on WMT’20 Test Set: refer to newstest Amount of Synthetic Data Used: r (%) 2014 2018 2019 100 33.0 48.0 42.0 50 32.9 48.4 42.3 33 33.1 47.9 42.2 25 32.9 48.5 42.4 Table 7: Effectiveness of corpus filtering on En!De. newstest Table from [Kiyono+2020] -
ΑΓྑ͍TransformerΛͭ͘Δ • Ґஔදݱ • ઈରҐஔΑΓ૬ରҐஔ • ݸਓతʹALiBiͱSHAPEʹՄೳੑΛײ͡Δ • Attention •
͍ܥྻͷॲཧʹ՝ • Perceiver͕࣍ͷݖΛऔΔ͔͠Εͳ͍ • Feed-Forward • ࣍ݩΛ૿͢ɺ׆ੑԽؔɺMixture-of-Experts • Layer Normalization • PreLN͕Ұൠత • ྑ͠ѱࠞ͠ಱͱ͍ͯ͠Δ… • ྑ͞ͷࢼ • ൚༻తʹྑ͍ख๏΄ͱΜͲແ͍ ໊ݹ۠NLPηϛφʔ 2022/06/07 63 -
ΑΓྑ͍TransformerΛͭ͘Δʁ • ಛఆͷઃఆͰ”ྑ͍”Transformerग़͖͍ͯͯΔ • ͔͠͠ɺ൚༻తʹྑ͍ͷΛͭ͘Δͷඇৗʹࠔ • ݴ͍͑ΔͱɺσʔλྔɾλεΫɾεέʔϧɾ࣮ʹΑͬͯ ྑ͍Transformerͷઃఆҧ͏ • …ͱ͍͏ͷ͕͜Ε·Ͱͷݟ
• ݁ՌɿҰൠతʹ࠾༻͞Ε͍ͯΔख๏͔ᷮ͘͝ • Pre-LNɾGeLUʢɾ૬ରҐஔʁʁʣ • ΠϯύΫτͷେ͖͍ݚڀͰ͋Δ͜ͱʹมΘΓͳ͍ • ࠓޙओྲྀͷݚڀͱͯ͠ଓ͖ͦ͏ ໊ݹ۠NLPηϛφʔ 2022/06/07 64