Understanding Back-Translation at Scale

機械翻訳のデータ拡大手法の一つである逆翻訳について、大量データで評価するとどうなるか検証した論文を紹介します。

Transcript

1. Understanding Back-Translation at Scale Yasumasa Sasano (@SquirrelYellow) ٯ຋༁࿦จͷσʔλΛಡΉ Edunov et al. 2018ˏEMNLP 2018
2. Back-Translation = BT ͱ͸Կ͔ 5BSHFU
   จষσʔλ 4PVSDF
   จষσʔλ ֶश ٯ຋༁Ϟσϧ
   BT https://qiita.com/tkmaroon/items/4b8f469db1534d5e265b ͪ͜ΒͷهࣄͷදݱΛआΓ·ͨ͠ (1) ຊ໋ͱ͸ٯํ޲ͷ຋༁ϞσϧΛֶश(೔ӳͳΒӳ೔)
3. 5BSHFU
   จষσʔλ 4PVSDF
   จষσʔλ 5BSHFU
   ୯ݴޠσʔλ 4PVSDF
   ߹੒ 4ZOUIFUJD
   ୯ݴޠσʔλ ਪ࿦ ٯ຋༁Ϟσϧ
   BT Back-Translation = BT ͱ͸Կ͔ (2) BTΛ࢖ͬͯσʔλΛ૿΍͢
4. 5BSHFU
   จষσʔλ 4PVSDF
   จষσʔλ ຊ໋Ϟσϧ 5BSHFU
   ୯ݴޠσʔλ 4PVSDF
   ߹੒ 4ZOUIFUJD
   ୯ݴޠσʔλ ֶश Back-Translation = BT ͱ͸Կ͔ (3) ૿΍ͨ͠σʔλͰֶश ࿦จʹॻ͍ͯͳ͍͕ɺΘ͟Θ͟ʮٯʯ຋༁͢Δͷ͸ ਖ਼͍͠จষΛڭࢣʹ࠷దԽ͍ͨ͠ͱ͍͏͜ͱͩͱߟ͑Δ
5. BTͰେ෯ਫ਼౓UPͱ࿩୊ʹ http://deeplearning.hatenablog.com/entry/back_translation
6. ͜ͷ࿦จΛબΜͩಈػ ࣮͸৽ख๏ͷఏҊ࿦จͰ͸ͳ͍ طଘͷॾख๏ΛେྔσʔλͰධՁ͢ΔͱͲ͏ͳΔ͔ݕূ at Scale σʔλ֦େʹର͢ΔݕূσʔλΛಡΜͰٞ࿦͍ͨ͠ BT͸Ұछͷσʔλ֦େ - ࢓ࣄͷؔ܎্ɺࠓ͋ΔσʔλΛϑϧʹ׆͔͢ಈػ͕͋Δ - ͲΜͳσʔλ֦େ͕༗ޮ͔͸ղ໌͞Ε͍ͯͳ͍෦෼͕ଟ͍ͷͰڵຯ͕͋Δ ͷ͕ಈػ
7. ฆΕ͕ͳ͍Α͏ʹ ΤϏσϯε ࿦จͷओு ݸਓͷॴײ ؾʹͳΔϙΠϯτ
8. Synthetic data generation method #5Ͱ࡞Δ߹੒σʔλʹ͍ͭͯ
9. ߹੒σʔλͷ࡞ΓํʹΑΔҧ͍ΛධՁ Greedy Search ෩अ ෩अ פ͍ פ͍ ࠓ೔ ͷ ෩अ פ͍ ࡢ೔ ͸ Beam Search ArgmaxΛ࢖͏ͱ༁จͷଟ༷ੑ͕ͳ͘ͳͬͯ·͍ͣ ࠓ೔ ͷ ෩अ פ͍ ࡢ೔ ͸ εςοϓຖʹҐΛ ֬ఆͯ࣍͠ͷ୯ޠ΁ ௨͠Ͱߴ֬཰ͷΛબ୒ શ୳ࡧ͸ແཧͳͷͰ Beam ༗ݶ෯ Ͱ୳ࡧ 1Ґ લޙ৚݅෇1Ґ Greedy Search Beam Search Top 10 Sampling Beam + Noise Argmax Noised Middle ୯ޠ ֬཰෼෍ (ιʔτࡁ)
10. ߹੒σʔλͷ࡞ΓํʹΑΔҧ͍ΛධՁ Top 10 ηʔλʔ פ͍ פ͍ ࠓ೔ ͷ ෩अ פ͍ ࡢ೔ ͸ Beam + Noise Sampling ྫྷଂݿ ϥϯμϜαϯϓϦϯά 1Ґ͔Β10ҐݶఆͰϥϯμϜαϯϓϦϯά ࠓ೔ פ͍ ͸ ࠓ೔ ͸ פ͍ ࠓ೔ ͸ פ͍ ࠓ೔ ͸ פ͍ BLANK ม͑ͯ΋͕ࠩͳ͍ p=0.1 p=0.1 uniform+maxҠಈ3 k=5, 10, 20, 50Ͱࢼ͕ͨ͠ɺ Otto et al. 2018a ʹΑΔͱෆ֬ఆੑ͕ ͔ͳΓେ͖͘มͳ ୯ޠΛग़͢Մೳੑ͕େ͖͍ ॳग़͸Imamura et al. 2018 (NICT) ڭࢣͳֶ͠शख๏ͰఏҊ Lample et al. 2018a ෩अ ෩अ ୯ޠ ֬཰෼෍ (ιʔτࡁ) ੜ੒จʹଟ༷ੑΛ࣋ͨͤΔ͜ͱ͕Ͱ͖Δ จষੜ੒ٕ๏ͱͯ͠͸ݹ͘ɺ Graves et al. 2003ͳͲͰ࢖ΘΕ͍ͯΔ
11. ߹੒σʔλͷ࡞ΓํʹΑΔҧ͍ΛධՁ sampling΍beam+noise͸ɺbeam΍greedyΑΓ΋1.7-2.0 BLEU΋ੑೳ͕ྑ͍
    top10͸beam΍greedyΑΓ΋ྑ͍͕sampling΍beam+noiseΑΓѱ͍ sampling΍beam+noise͸.ͷ࣌ʹbeamͷഒۙ͘ੑೳվળ͍ͯ͠Δ
12. ੜ੒͞Εͨจষͷ෼ੳ Greedy search΍Beam search͸ଟ༷ͰϦονͳσʔλ෼෍Λ࿪ΊΔ Ott et al.2018aͷ ࿦จʹΑΔͱ௿ස౓ޠ͕ग़ͳ͘ͳΔ܏޲ʹ͋Δ ͷͰSamplingख๏͕Α͍
    denoising autoencodersͱͷྨࣅੑ
    sampling΍beam+noiseͰग़དྷ্͕ͬͨจ͸ݱ࣮཭Ε͍ͯ͠Δ͕ɺzஔ׵z΍zॱংมߋzͱ ͍͏ݱ৅͸ී௨ʹى͖ΔͷͰͦ͏͍ͬͨॲཧΛೖΕΔͱϩόετʹͳΔ
    ࣍ͷ୯ޠ͕༧ଌͰ͖ͳ͍ͨΊɺ೉қ౓͕Ҿ্͖͕ͬͯਫ਼౓্͕͕Δ
13. ੜ੒͞Εͨจষͷ෼ੳ ໌Β͔ʹ͓͔͍͠୯ޠ͕ೖΔͷ͸zہॴతzͩͱΘ͔Δ Ծઆ
    ͲΜͳϊΠζ୯ޠ͕དྷͯ΋໰୊ͳ͍Α͏ɺ࢒ͬͨਖ਼ৗ෦෼ͷ൚Խੑೳ͕޲্ͨ͠ʁ 0, /( ڐ༰Ͱ͖Δ୯ޠΛ੨ɺ໌Β͔ʹ͓͔͍͠୯ޠΛ੺ͰృͬͯΈΔͱɺ ʮہॴతͳϊΠζʯʹΑΔ൚Խੑೳ޲্
    ࣭ʹؔΘΒͣଟ༷ੑ͕૿͔͑ͨΒ0,ͱ͍͏ղऍ͸Ͱ͖ͳ͘͸ͳ͍͕ɺ ͦΕʹͯ͠͸ਫ਼౓্͕Γ͗͢Ͱ͸ʁͱ͍͏͜ͱͰ΋͏গ͠۷ΓԼ͍͛ͨ (ݸਓతߟ࡯)
14. (ݸਓతߟ࡯ͷଓ͖) ݘ͕޷͖Ͱ͢ ΫτΡϧϑਆ࿩͕޷͖Ͱ͢ I like dog I am scared of Cthulhu ہॴతϊΠζΛ෇༩ ଟ͘ͷࣗવݴޠॲཧͷϞσϧ͸ গ͠ม͑Δ͚ͩͰ؆୯ʹὃͤΔಛੑ͕͋Δ Deep Text Classification Can be Fooled Liang et al. 2016 ຋༁ ະֶशͷσʔλ ޡࠩٯ఻೻ ͜ͷ໰୊ʹରԠ͢Δଧͪख ʹͳ͍ͬͯΔՄೳੑ ԾʹΫτΡϧϑ͕ປࢺͰ΋ ʮ޷͖ʯ͸ʮlikeʯ (ϊΠζ෦෼ʹޡࠩΛ఻೻͢Δͷ͸׬ᘳʹແବͳͷͰվળͰ͖Δ͔΋)
15. Low Resource & High Resource #5ͷݩखͱͳΔର༁Ϧιʔεྔͷҧ͍ʹ͍ͭͯ
16. 5BSHFU 4PVSDF ຊ໋Ϟσϧ 5BSHFU
    ୯ݴޠσʔλ 4PVSDF
    ߹੒ 4ZOUIFUJD
    ୯ݴޠσʔλ ֶश ݩख͕গͳ͍ͱԿ͕ى͜Δ͔ ͜͜ͷྔ͕গͳ͍(80Kจఔ౓) จݿຊ࡭͘Β͍ (1࡭12ສࣈ, 80ࣈ/จ)
17. ݩख͕গͳ͍ͱԿ͕ى͜Δ͔ 80KจͰ͸sampling
    
    beam searchͷٯసݱ৅͕ى͖͍ͯΔ
    σʔλ਺͕ଟ͚Ε͹ଟ͍΄Ͳsampling͕ڧ͘ͳΔ
    ݩख͕গͳ͍৔߹ɺBTͷਫ਼౓͕ߴ͘ͳ͍ͷͰɺsamplingͰϊΠζΛՃ͑ͨͱ͖ͷѱӨ ڹʹ੬͘ͳΔ BTͷਫ਼౓ͷҾ্͖͕͛ඞཁ
18. ݩख͕গͳ͍໰୊ͷܰݮ 5BSHFU 4PVSDF &ODPEFS %FDPEFS 4PVSDF 4PVSDF 5BSHFU 5BSHFU 4PVSDFݴޠϞσϧ 5BSHFUݴޠϞσϧ సҠֶशorॏΈڞ༗ సҠֶशorॏΈڞ༗ (1) ୯ݴޠͰݴޠϞσϧΛ࡞ͬͯసҠֶश ʮݴޠϞσϧͷసҠ͕ࠔ೉ʯͱ͍͏໰୊͕Devlin et al. 2018 (BERT)Ͱղফ͞ΕͨͷͰਐల͋Δ͔΋
19. ͍ͭͷؒʹ͔΋ͷ͍͢͝࿦จ͕ൃද͞Ε͍ͯͨ ࢀߟ࿦จ: Lample et al. 2019 (XLM) #&35ΛసҠֶशɺ຋༁Λ&ODPEFS%FDPEFSͷܗͰ͸ͳ͘ҰͭͷݴޠϞσϧ ͱֶͯ͠श͠ɺ8.5`
    ಠӳ຋༁ͷڭࢣͳֶ͠शͷ405"Λ
    #-&6ߋ৽ 
    BSYJW
    TVCNJU
20. ݩख͕গͳ͍໰୊ͷܰݮ (2) ૒ରֶश (Dual Learning) ຊ໋Ϟσϧ 5BSHFU
    ୯ݴޠσʔλ 4PVSDF
    ୯ݴޠσʔλ l૒ରzϞσϧ ର༁Ͱͳͯ͘OK
21. Domain of synthetic data ߹੒σʔλͷυϝΠϯʹؔ͢Δݕূ
22. υϝΠϯదԠ 5BSHFU
    จষσʔλ 4PVSDF
    จষσʔλ ຊ໋Ϟσϧ χϡʔε
    5BSHFU
    ୯ݴޠσʔλ χϡʔε
    4PVSDF
    ߹੒ 4ZOUIFUJD
    ୯ݴޠσʔλ ֶश χϡʔεͷର༁σʔλ͕ͳͯ͘΋χϡʔεʹڧ͘ͳΔ͔ʁ
23. υϝΠϯదԠ ධՁ༻σʔλͷυϝΠϯʹBTͷυϝΠϯ news ͷ৔߹͸ຊ෺ͷσʔλ ૝ఆͰ83%ͷվળ཰
    ධՁ༻σʔλͷυϝΠϯͱ#5ͷυϝΠϯ news ͕·ΔͰ߹͍ͬͯͳ͍ ৔߹ʹ΋ຊ෺ͷσʔλ૝ఆͰ32.5%ͷվળ཰
    ͲͪΒ΋վળ͍ͯ͠Δ͕ɺυϝΠϯ߹க͍ͯ͠Δ৔߹͸൚༻ͷσʔλҎ ্ͷਫ਼౓ʹͳΔ ʓʓδϟϯϧͷର༁σʔλ͕ͳͯ͘΋
    ୯ݴޠσʔλ͕͋Ε͹ʓʓδϟϯϧͷ຋༁ΛڧԽՄೳ
24. ·ͱΊ ·ͱΊ Ͳͷख๏Ͱ΋ٯ຋༁ΛೖΕΕ͹ਫ਼౓͸্͕Δ͕ɺٯ ຋༁͢Δͱ͖ͷѻ͍Ͱਫ਼౓޲্෯͕਺ഒʹͳΔ͜ͱ ΋͋Δ σʔλ͕গͳ͍࣌ʹ͸૬ରతʹੑೳ͕Լ͕ΔͷͰ҆ қʹαϯϓϦϯά͕࢖͑ͳ͍ υϝΠϯదԠʹ΋࢖͑Δ