Share
Embed
Transcript
17 slides
-
ಡΉਓ ཧݚ"*1 ౦େֶ סݚڀࣨ ਗ਼ ॢ Reservoir Transformers Sheng Shen†,
Alexei Baevski‡, Ari S. Morcos‡, Kurt Keutzer†, Michael Auli‡, Douwe Kiela‡ †UC Berkeley; ‡Facebook AI Research [email protected], [email protected] Abstract We demonstrate that transformers obtain im- pressive performance even when some of the layers are randomly initialized and never up- is more, we find that freezing layers may actually improve performance. Beyond desirable efficiency gains, random lay- ers are interesting for several additional reasons. ※ऍͷͳ͍ਤදจ͔ΒҾ༻͞ΕͨͷͰ͢ -
ͲΜͳจ͔ʁ • എܠɾ • Transformerͷֶशʹ͔͔Δ࣌ؒΛ͍ͨ͘͠ • ΞΠσΞ • TransformerͷҰ෦ͷϨΠϠʔΛॳظͷ··ݻఆ •
ύϥϝʔλߋ৽ͷܭࢉ͕αϘΕΔͷͰߴԽ • ߩݙ • Ϟσϧͷऩଋ͕~30%΄Ͳ͘ͳΔʢ߹͋Δʣ • Ϟσϧͷੑೳ্͕͕Δʢ߹͋Δʣ September 17, 2021 ࠷ઌNLPษڧձ 4 -
ੈ·͞ʹҰେݴޠϞσϧ࣌ • All you need is Transformer? • ߴύϑΥʔϚϯε •
ݴޠɺը૾ɺԻͰେྲྀߦ • Vanillaઃఆ͕ڧ͗͢Δ • Vanillaͱѥछʹ΄ͱΜͲࠩͳ͍ [Narang+2021] • ॳొ͔ΒมΘͬͨͷLayer-normalizationͷҐஔ͘Β͍ʁ • Scaling Law [Kaplan+2020] • σʔλ͕ଟ͚ΕੑೳUP • Ϟσϧ͕େ͖͚ΕੑೳUP September 17, 2021 ࠷ઌNLPษڧձ 5 https://ruder.io/research-highlights-2020/ Transformer Dataset Size tokens Parameters non-embedding Compute PF-days, non-embedding Test Loss Figure 1 Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute2 used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two. [Kaplan+2020] -
ڊେχϡʔϥϧωοτͷ܇࿅ͭΒ͍ • ͦͦϋʔυΣΞʢGPUʣ͕Γͳ͍ • 32GBΧʔυͷݶք1.4Bύϥϝλ [Ren+2021] • ܭࢉࢿݯͷݶΒΕͨݚڀάϧʔϓͰ܇࿅͕Ͱ͖ͳ͍ • ܇࿅ʹ͕͔͔࣌ؒΓ͗͢Δ
• GPT-3ɿ1ຕͷGPUͰ355ʢΒ͍͠ʣ • Transformer (big)ɿDGX-2ΛͬͯҰ • ίϯϖͷ߹16ϞσϧඞཁͩͬͨΓ͢Δ à 16ʁʁʁ • ڥͷෛՙʢԹࣨޮՌΨεʣແࢹͰ͖ͳ͍ September 17, 2021 ࠷ઌNLPษڧձ 6 ࠓճͬͪ͜ͷղܾΛࢦ͢ -
͓͞Β͍ɿTransformer • ಉ͡େ͖͞ͷΛੵΜͰੑೳ্ΛਤΔͷ͕ओྲྀ September 17, 2021 ࠷ઌNLPษڧձ 7 Transformer-layer 🔥
Transformer-layer 🔥 Transformer-layer🔥 Transformer-layer🔥 Transformer-layer🔥 Multi-Head Attention Add & Norm Feed Forward Add & Norm Input + -
ΞΠσΞɿϨΠϠʔΛߋ৽͠ͳ͍ • Ұ෦ͷϨΠϠʔॳظͰݻఆ❄ • ଞͷϨΠϠʔ௨ৗ௨Γߋ৽🔥 • ߋ৽͢Δͱ͠ͳ͍Λࠞͥͯஔ • ֤ݻఆ͕ͳΔ͘ΕΔΑ͏ʹ September
17, 2021 ࠷ઌNLPษڧձ 8 Transformer-layer 🔥 Transformer-layer ❄ Transformer-layer🔥 Transformer-layer❄ Transformer-layer🔥 ɿ͜ͷίʔυಈ͖·ͤΜ -
ఏҊख๏ɿResavoir Transformer • Resavoir Transformer • ॳظͰݻఆ͞ΕͨϨΠϠʔ❄Λ ؚΉΑ͏ͳTransformer • ఏҊख๏ͷؾ࣋ͪ
• ύϥϝʔλߋ৽ΛαϘΕΔͷͰ ߴԽ • ॳظϨΠϠʔ͕ಛநग़Λ ؤுͬͯ͘ΕΔ͜ͱΛفΔ • ѱ͘ݴ͑ͬͯΈͨܥͷ September 17, 2021 ࠷ઌNLPษڧձ 9 Transformer-layer 🔥 Transformer-layer ❄ Transformer-layer🔥 Transformer-layer❄ Transformer-layer🔥 -
ϨβϘΞͷ͖ͪ • χϡʔϥϧωοτϫʔΫͬͯదͩ͠ɺ ϊΠζతʹ͏·͘ಈ࡞ͯ͘͠ΕΔΜ͡Όͳ͍…ʁ • Residualଓͷ͓͔͛Ͱେৎʁ • ݻఆϨΠϠʔ❄͕༗ӹͳಛΛ࡞Δ͜ͱΛظ͢Δ • ࡞Εͳ͍߹ɿֶशϨΠϠʔ🔥
ͷϊϧϜ > ݻఆϨΠϠʔ❄ ͷ ϊϧϜͱ͢ΕɺݻఆϨΠϠʔແࢹͰ͖Δʁ • ྨࣅʁख๏ɿAdapter • ֶशࡁΈͷTransformerʹখ͞ͳϞδϡʔϧΛ Ճͯ͠finetuning • ϥϯμϜॳظԽ෦߃ࣸ૾͔Βελʔτ͢ΔͨΊ finetuneݩͷใഁյ͞Εͳ͍ • ResidualଓͷҐஔ͕ඇৗʹॏཁΒ͍͠ • [Pfeiffer+2020] September 17, 2021 ࠷ઌNLPษڧձ 10 ॳظͷ··ݻఆͯ͠ຊʹେৎͳͷ͔ʁ 2.2.1 Single-task Adapters For each of the N tasks, the model is initialized with parameters ⇥0. In addition, a set of new and randomly initialized parameters n are introduced (the adapter parameters). To share the same set of parameters ⇥0 across all otherwise independent tasks, the parameters in ⇥0 are fixed and only the parameters n are trained. This makes it possible to efficiently parallelize the training of adapters for all N tasks. The objective for each task n 2 1, . . . , N is of the form: n argmin Ln(Dn; ⇥0, ) For common adapter architectures, contains considerably fewer parameters than ⇥, e.g., only 3.6% of the parameters of the pre-trained model in (Houlsby et al., 2019). Feed Forward Add & Norm Adapter Multi-Head Attention Add & Norm Add & Norm LayerNorm LayerNorm FF Down FF Up Add & Norm Feed Forward Multi-Head Attention Add & Norm Add & Norm FF Down FF Up Add & Norm Adapter Adapter Figure 2: Different architectural components of the adapter. On the left, we show all components for which -
ݻఆϨΠϠʔ ❄ ͷطଘݚڀ • ॳظݻఆͷLSTM͕ڧ͍จΤϯίʔμͰ͋Δ [Wieting+2019] • Skip-thoughtInfersentͱͷ͕ࠩখ͍͜͞ͱΛࣔͨ͠ • ϥϯμϜࣸ૾Λ༻͍ͨ࣍ݩݮ
[Sahlgren 2005] • ෦ۭؒͷࣸ૾͕ݩۭؒͰͷڑΛอ࣋ September 17, 2021 ࠷ઌNLPษڧձ 11 Published as a conference paper at ICLR 2019 Model Dim MR CR MPQA SUBJ SST2 TREC SICK-R SICK-E MRPC STSB BOE 300 77.3(.2) 78.6(.3) 87.6(.1) 91.3(.1) 80.0(.5) 81.5(.8) 80.2(.1) 78.7(.1) 72.9(.3) 70.5(.1) BOREP 4096 77.4(.4) 79.5(.2) 88.3(.2) 91.9(.2) 81.8(.4) 88.8(.3) 85.5(.1) 82.7(.7) 73.9(.4) 68.5(.6) RandLSTM 4096 77.2(.3) 78.7(.5) 87.9(.1) 91.9(.2) 81.5(.3) 86.5(1.1) 85.5(.1) 81.8(.5) 74.1(.5) 72.4(.5) ESN 4096 78.1(.3) 80.0(.6) 88.5(.2) 92.6(.1) 83.0(.5) 87.9(1.0) 86.1(.1) 83.1(.4) 73.4(.4) 74.4(.3) InferSent-1 = paper, G 4096 81.1 86.3 90.2 92.4 84.6 88.2 88.3 86.3 76.2 75.6 InferSent-2 = fixed pad, F 4096 79.7 84.2 89.4 92.7 84.3 90.8 88.8 86.3 76.0 78.4 InferSent-3 = fixed pad, G 4096 79.7 83.4 88.9 92.6 83.5 90.8 88.5 84.1 76.4 77.3 InferSent-3, BestRand - 1.6 3.4 0.4 0.0 0.5 2.0 2.4 1.0 2.3 2.9 ST-LN 4800 79.4 83.1 89.3 93.7 82.9 88.4 85.8 79.5 73.2 68.9 ST-LN, BestRand - 1.3 3.1 0.8 1.1 -0.1 0.5 -0.3 -3.6 -0.9 -5.5 Table 1: Performance (accuracy for all tasks except SICK-R and STSB, for which we report Pear- son’s r) on all ten downstream tasks where all models have 4096 dimensions with the exception of BOE (300) and ST-LN (4800). Standard deviations are show in parentheses. InferSent-1 is the paper [Wieting+2019] -
࣮ݧઃఆ • త • Reservoir TransformerͷޮՌΛݕূ • ϕʔεϥΠϯ • Transformer
• λεΫ • ػց༁ʢ8.5*84-5ʣ • ݴޠϞσϧʢenwiki8ʣ • ํݴޠϞσϧʢMNLIɺSST-2ͳͲʣ • ධՁࢦඪ • BLEUʢࢀর༁ͱͷҰகʣ • AUCCʢAUC-ROCͷֶश൛; ޙड़ʣ September 17, 2021 ࠷ઌNLPษڧձ 12 ࣌ؒͷ߹্͚ͩ͜͜ -
ධՁࢦඪɿAUCC September 17, 2021 ࠷ઌNLPษڧձ 13 ResavoirԽʹΑΔֶशͷӨڹΛௐ͍ͨ • AUCC: area
under the convergence curve • ֶशۂઢΛੵͯ͠ܭࢉ • ͕େ͖͍ → ऩଋ͕͍ ʢand ੑೳ͕ྑ͍ʣ ͜͜Λੵ ܇࿅࣌ؒ BLEU దͳ෯ͰଧͪΓ -
࣮ݧ݁Ռ on September 17, 2021 ࠷ઌNLPษڧձ 14 Figure 8:
Validation BLEU AUCC an IWSLT (high is good). Comparison former and reservoir transformer with former reservoir layers added. • Τϯίʔμͷ͕… • খ͍͞ͱ͖ɿϕʔεϥΠϯͷ΄͏͕ߴʁ • େ͖͍ͱ͖ɿఏҊख๏ͷ΄͏͕ߴʁ • Ұ؏ͯ͠ྑ͍ख๏ଘࡏ͠ͳ͍ • ֶश࣌ؒΛ1ׂݮΒͤΔ͔ɺͱ͍͏ఔ Կݻఆ ͍ͯ͠Δ͔Ṗ Ͳ͜ʹ͕ ଧͨΕ͍ͯΔ͔Ṗ -
࣮ݧ݁Ռ on BLEU • ResavoirϨΠϠʔ͕ଟ͍ͱ͖ʹޮՌ͋Γʁ • ਖ਼ଇԽͱͯ͠ಈ࡞ʁ • Ϟσϧ͕Over-parametrize͞Εͨঢ়گԼͰ༗ޮ͔ •
ੑೳ΄ͱΜͲಉ͡ɺͱ͍͏ͷ͕࣮ͦ͏ September 17, 2021 ࠷ઌNLPษڧձ 15 ion BLEU AUCC and test BLEU for good). Comparison of regular trans- voir transformer with FFN or Trans- layers added. Figure 1 enwik8 parison ing dep Կݻఆͯ͠ ͍Δ͔Ṗ -
ؾʹͳΔ͜ͱɿϕʔεϥΠϯ • ϕʔεϥΠϯͷTransformerͷܗ͕͓͔͍͠ • Encoder Nʹରͯ͠Decoder 1 [Kasai et al.
2020] • ී௨6ɿ6Ͱ࣮ݧ • ஶऀʮߴͳઃఆ͔ͩΒ࠾༻ͨ͠ʯ • ଥͳઃఆͰͳ͍ؾ͕͢Δ… • ੈͷதͷΤϯίʔμɾσίʔμͱ͔͚Εͨઃఆ • 2ͭͷͷ͕औΓҧ͑ΒΕ͍ͯΔʁ • [Kasai et al. 2020]ɿσίʔυͷߴԽͷ • ຊจɿ܇࿅ͷߴԽͷ • ͜͜ΛऔΓҧ͑ͯٞͯ͠Δ࣌Ͱ͔ͳΓո͍͕͠… September 17, 2021 ࠷ઌNLPษڧձ 16 -
ϕʔεϥΠϯͷܗ͕ͳͥॏཁ͔ʁ • ࣮ݧͷݟ͕৴༻Ͱ͖ͳ͍ • ϕʔεϥΠϯͷऩଋेʹߴ͔ʁ • 6:6ઃఆͷ΄͏͕ऩଋ͕͍ɺͱ͍͏͜ͱͳ͍͔ʁ • ϕʔεϥΠϯͷੑೳेʹߴ͍ͷ͔ʁ •
6:6ઃఆͷ΄͏͕ੑೳ͕ߴ͍ɺͱ͍͏͜ͱͳ͍͔ʁ • ඞཁͳݟ͕ಘΒΕ͍ͯͳ͍ • DecoderଆͰReservoir͢Δ͜ͱՄೳ͔ʁ September 17, 2021 ࠷ઌNLPษڧձ 17 -
ϕʔεϥΠϯͷௐࠪ September 17, 2021 ࠷ઌNLPษڧձ 18 [Xu+2021] ͷࢼͱরΒ͠߹ΘͤͯΈͨ • ϕʔεϥΠϯͷऩଋेʹߴ͔ʁ
à ଟ0, • 😃 σίʔμΛઙͨ͘͠΄͏͕܇࿅͕͍ • 😰 ͨͩɺऩଋ͕͍͔Ͳ͏͔Θ͔Βͳ͍ • AUCC͕ߴ͍͔Ͳ͏͔Θ͔Βͳ͍ • ϕʔεϥΠϯͷੑೳेʹߴ͍ͷ͔ʁ à /( • 😨 ઃఆͷ΄͏͕ߴੑೳʹݟ͑Δ [Xu+2021] -
·ͱΊʢ࠶ܝʣ • എܠɾ • Transformerͷֶशʹ͔͔Δ࣌ؒΛ͍ͨ͘͠ • ΞΠσΞ • TransformerͷҰ෦ͷϨΠϠʔΛॳظͷ··ݻఆ •
ύϥϝʔλߋ৽ͷܭࢉ͕αϘΕΔͷͰߴԽ • ߩݙ • Ϟσϧͷऩଋ͕~30%΄Ͳ͘ͳΔʢ߹͋Δʣ • Ϟσϧͷੑೳ্͕͕Δʢ߹͋Δʣ September 17, 2021 ࠷ઌNLPษڧձ 19