TaskͰ΄ͱΜͲͷνʔϜ͕࠾༻ March 20, 2020 RIKEN AIP / Tohoku University 3 5 4 + reduce batch size (4k ! 1k tokens) 12.40 ± 0.08 31.97 ± 0 6 5 + lexical model 13.03 ± 0.49 31.80 ± 0 7 5 + aggressive (word) dropout 15.87 ± 0.09 33.60 ± 0 8 7 + other hyperparameter tuning (learning rate, 16.57 ± 0.26 32.80 ± 0 model depth, label smoothing rate) 9 8 + lexical model 16.10 ± 0.29 33.30 ± 0 Table 2: German!English IWSLT results for training corpus size of 100k words and 3.2M words (full c Mean and standard deviation of three training runs reported. 105 106 0 10 20 30 32.8 30.8 28.7 24.4 20.6 16.6 26.6 24.9 23 20.5 18.3 16 25.7 18.5 11.6 1.8 1.3 0 corpus size (English words) BLEU neural MT optimized phrase-based SMT neural MT baseline Figure 2: German!English learning curve, showing BLEU as a function of the amount of parallel training data, for PBSMT and NMT. 4.3 NMT Systems We train neural systems with Nematus (Sennrich et al., 2017b). Our baseline mostly follows the size, model depth, regularization paramete learning rate. Detailed hyperparameters ported in Appendix A. 5 Results Table 2 shows the effect of adding different ods to the baseline NMT system, on the ult data condition (100k words of training dat the full IWSLT 14 training corpus (3.2M w Our ”mainstream improvements” add aroun BLEU in both data conditions. In the ultra-low data condition, reduci BPE vocabulary size is very effective BLEU). Reducing the batch size to 1000 to sults in a BLEU gain of 0.3, and the lexical yields an additional +0.6 BLEU. Howeve gressive (word) dropout6 (+3.4 BLEU) and other hyperparameters (+0.7 BLEU) has a st effect than the lexical model, and adding t Figure from [Sennrich and Zhang 2019]
[Zhao+2019] • จʹϥϯμϜͳϊΠζΛ༩͢Δ • ֤ख๏ͷৄࡉจΛࢀর͍ͯͩ͘͠͞ March 20, 2020 RIKEN AIP / Tohoku University 10 rea@@ ter London . DIRECTNOISE: hmaski hmaski hmaski for hmaski hmaski hmaski hmaski Original: The cli@@ p is mixed with images of Toronto streets during power failure . BACKTRANS (NOISY): The cli@@ p is mix with images of Toronto streets during power failure . DIRECTNOISE: The hmaski is mixed hmaski images si@@ of The hmaski streets large hmaski power R@@ failure place hmaski Original: At the in@@ stitute , she introduced tis@@ sue culture methods that she had learned in the U.@@ S. BACKTRANS (NOISY): At in@@ stitute , She introduced tis@@ sue culture method that she learned in U.@@ S. DIRECTNOISE: hmaski the the hmaski hmaski hmaski hmaski tis@@ culture R@@ methods , she P hmaski the s U.@@ hmaski Figure 5: Examples of sentences generated by BACKTRANS (NOISY) and DIRECTNOISE methods. Fig. 6 shows examples generated by DIRECTNOISE, when changing the mask probability (µmask). µmask Output Sentence N/A He threw the sand@@ wi@@ ch at his wife . 0.1 He ale threw , ch his ne@@ wife dar@@ hmaski 0.3 hmaski hmaski hmaski hmaski ch at ament his Research . 0.5 He o threw the sand@@ ch hmaski his hmaski . 0.7 hmaski hmaski sand@@ hmaski hmaski hmaski hmaski wife hmaski Figure 6: Examples generated when varying µmask . N/A denotes original text.
/ Tohoku University 16 • ੜݩίʔύεͷੑೳͷӨڹখ͍͞ʁ • ͔͠͠ɺ(JHBXPSE͕Ұ؏ͯ͠ྑ͍ • จ๏త ͖Ε͍ͳจΛ׆༻͢Δ͜ͱͷॏཁੑΛࣔࠦ Method Seed Corpus T Prec. Rec. F0.5 Baseline N/A 46.6 23.1 38.8 BACKTRANS (NOISY) Wikipedia 43.8 30.8 40.4 BACKTRANS (NOISY) SimpleWiki 42.5 31.3 39.7 BACKTRANS (NOISY) Gigaword 43.1 33.1 40.6 DIRECTNOISE Wikipedia 48.3 25.5 41.0 DIRECTNOISE SimpleWiki 48.9 25.7 41.4 DIRECTNOISE Gigaword 48.3 26.9 41.7 Table 3: Performance on BEA-valid when changing the seed corpus T used for generating pseudo data (|Dp | = 1.4M). DIRECTNOISE with Gigaword achieves the best value of F0.5 among all the configurations. Optimization Metho N/A Baseli PRETRAIN BACK PRETRAIN DIREC JOINT BACK JOINT DIREC PRETRAIN BACK PRETRAIN DIREC JOINT BACK JOINT DIREC Table 4: Performan mization settings o Wikipedia. ੑೳมԽখ ੑೳมԽখ
/ Tohoku University 17 • ੜݩίʔύεͷੑೳͷӨڹখ͍͞ʁ • ͔͠͠ɺ(JHBXPSE͕Ұ؏ͯ͠ྑ͍ • จ๏త ͖Ε͍ͳจΛ׆༻͢Δ͜ͱͷॏཁੑΛࣔࠦ Method Seed Corpus T Prec. Rec. F0.5 Baseline N/A 46.6 23.1 38.8 BACKTRANS (NOISY) Wikipedia 43.8 30.8 40.4 BACKTRANS (NOISY) SimpleWiki 42.5 31.3 39.7 BACKTRANS (NOISY) Gigaword 43.1 33.1 40.6 DIRECTNOISE Wikipedia 48.3 25.5 41.0 DIRECTNOISE SimpleWiki 48.9 25.7 41.4 DIRECTNOISE Gigaword 48.3 26.9 41.7 Table 3: Performance on BEA-valid when changing the seed corpus T used for generating pseudo data (|Dp | = 1.4M). DIRECTNOISE with Gigaword achieves the best value of F0.5 among all the configurations. Optimization Metho N/A Baseli PRETRAIN BACK PRETRAIN DIREC JOINT BACK JOINT DIREC PRETRAIN BACK PRETRAIN DIREC JOINT BACK JOINT DIREC Table 4: Performan mization settings o Wikipedia. ੑೳมԽখ ੑೳมԽখ 1: GigawordΛ͏͠
LargePretrain+Ensemble+SSE+R2L LargePretrain (Single Model) Grundkiewicz et al. (2019) Zhao et al. (2019) Lichtarge et al. (2019) Junczys-Dowmunt et al. (2018) Chollampatt and Ng (2018) F0.5 (CoNLL2014) γϯάϧϞσϧͷ࣌Ͱߴੑೳ March 20, 2020 RIKEN AIP / Tohoku University 24 શͯΞϯαϯϒϧϞσϧ γϯάϧϞσϧͷੑೳ͕ɺ[Grundkiewicz+2019]Λআ͘ શͯͷΞϯαϯϒϧϞσϧΑΓߴੑೳ
LargePretrain+Ensemble+SSE+R2L LargePretrain (Single Model) Grundkiewicz et al. (2019) Zhao et al. (2019) Lichtarge et al. (2019) Junczys-Dowmunt et al. (2018) Chollampatt and Ng (2018) F0.5 (CoNLL2014) Ճख๏ʹΑΓߋʹੑೳ্ March 20, 2020 RIKEN AIP / Tohoku University 25 զʑͷϞσϧ͕ੈք࠷ߴੑೳΛୡ '
butsugiri
atsukomasano2026
fkske
trafficbrain
gpeyre
.png){kind=avatar}
satai
kanekomasahiro
koide3
signer
fufufukakaka
masakat0
analokmaus
k_sato
csswizardry
inesmontani
skipperchong
soudai
andyhume
codingconduct
jeffersonlam
aleyda
dougneiner
marktimemedia
eileencodes
![Q1: ٙࣅσʔλੜͷख๏ΛͲ͏͢Δʁ • BACKTRANS (NOISY) [Xie+2018] • ϊΠζ͖ٯ༁ʹΑΓٖࣅσʔλΛੜ • DIRECTNOISE](https://files.speakerdeck.com/presentations/ea43838567c443c7ac9360173c18d16b/slide_9.jpg){kind=link}
![࣮ݧઃఆɾσʔληοτ • ʮҰൠతʯͳઃఆΛ࠾༻ • Ϟσϧ: Transformer (Big) [Vaswani+2017] • ࠷దԽ:](https://files.speakerdeck.com/presentations/ea43838567c443c7ac9360173c18d16b/slide_14.jpg){kind=link}
