Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding Back-Translation at Scale

ysasano
February 12, 2019

Understanding Back-Translation at Scale

機械翻訳のデータ拡大手法の一つである逆翻訳について、大量データで評価するとどうなるか検証した論文を紹介します。

ysasano

February 12, 2019
Tweet

Other Decks in Technology

Transcript

  1. Back-Translation = BT ͱ͸Կ͔ 5BSHFU จষσʔλ 4PVSDF จষσʔλ ֶश ٯ຋༁Ϟσϧ

    BT https://qiita.com/tkmaroon/items/4b8f469db1534d5e265b ͪ͜ΒͷهࣄͷදݱΛआΓ·ͨ͠ (1) ຊ໋ͱ͸ٯํ޲ͷ຋༁ϞσϧΛֶश(೔ӳͳΒӳ೔)
  2. 5BSHFU จষσʔλ 4PVSDF จষσʔλ 5BSHFU ୯ݴޠσʔλ 4PVSDF ߹੒ 4ZOUIFUJD 

    ୯ݴޠσʔλ ਪ࿦ ٯ຋༁Ϟσϧ BT Back-Translation = BT ͱ͸Կ͔ (2) BTΛ࢖ͬͯσʔλΛ૿΍͢
  3. 5BSHFU จষσʔλ 4PVSDF จষσʔλ ຊ໋Ϟσϧ 5BSHFU ୯ݴޠσʔλ 4PVSDF ߹੒ 4ZOUIFUJD

     ୯ݴޠσʔλ ֶश Back-Translation = BT ͱ͸Կ͔ (3) ૿΍ͨ͠σʔλͰֶश ࿦จʹॻ͍ͯͳ͍͕ɺΘ͟Θ͟ʮٯʯ຋༁͢Δͷ͸ ਖ਼͍͠จষΛڭࢣʹ࠷దԽ͍ͨ͠ͱ͍͏͜ͱͩͱߟ͑Δ
  4. ߹੒σʔλͷ࡞ΓํʹΑΔҧ͍ΛධՁ Greedy Search ෩अ ෩अ פ͍ פ͍ ࠓ೔ ͷ ෩अ

    פ͍ ࡢ೔ ͸ Beam Search ArgmaxΛ࢖͏ͱ༁จͷଟ༷ੑ͕ͳ͘ͳͬͯ·͍ͣ ࠓ೔ ͷ ෩अ פ͍ ࡢ೔ ͸ εςοϓຖʹҐΛ ֬ఆͯ࣍͠ͷ୯ޠ΁ ௨͠Ͱߴ֬཰ͷΛબ୒ શ୳ࡧ͸ແཧͳͷͰ Beam ༗ݶ෯ Ͱ୳ࡧ 1Ґ લޙ৚݅෇1Ґ Greedy Search Beam Search Top 10 Sampling Beam + Noise Argmax Noised Middle ୯ޠ ֬཰෼෍ (ιʔτࡁ)
  5. ߹੒σʔλͷ࡞ΓํʹΑΔҧ͍ΛධՁ Top 10 ηʔλʔ פ͍ פ͍ ࠓ೔ ͷ ෩अ פ͍

    ࡢ೔ ͸ Beam + Noise Sampling ྫྷଂݿ ϥϯμϜαϯϓϦϯά 1Ґ͔Β10ҐݶఆͰϥϯμϜαϯϓϦϯά ࠓ೔ פ͍ ͸ ࠓ೔ ͸ פ͍ ࠓ೔ ͸ פ͍ ࠓ೔ ͸ פ͍ BLANK ม͑ͯ΋͕ࠩͳ͍ p=0.1 p=0.1 uniform+maxҠಈ3 k=5, 10, 20, 50Ͱࢼ͕ͨ͠ɺ Otto et al. 2018a ʹΑΔͱෆ֬ఆੑ͕ ͔ͳΓେ͖͘มͳ ୯ޠΛग़͢Մೳੑ͕େ͖͍ ॳग़͸Imamura et al. 2018 (NICT) ڭࢣͳֶ͠शख๏ͰఏҊ Lample et al. 2018a ෩अ ෩अ ୯ޠ ֬཰෼෍ (ιʔτࡁ) ੜ੒จʹଟ༷ੑΛ࣋ͨͤΔ͜ͱ͕Ͱ͖Δ จষੜ੒ٕ๏ͱͯ͠͸ݹ͘ɺ Graves et al. 2003ͳͲͰ࢖ΘΕ͍ͯΔ
  6. ੜ੒͞Εͨจষͷ෼ੳ Greedy search΍Beam search͸ଟ༷ͰϦονͳσʔλ෼෍Λ࿪ΊΔ Ott et al.2018aͷ ࿦จʹΑΔͱ௿ස౓ޠ͕ग़ͳ͘ͳΔ܏޲ʹ͋Δ ͷͰSamplingख๏͕Α͍ denoising

    autoencodersͱͷྨࣅੑ sampling΍beam+noiseͰग़དྷ্͕ͬͨจ͸ݱ࣮཭Ε͍ͯ͠Δ͕ɺzஔ׵z΍zॱংมߋzͱ ͍͏ݱ৅͸ී௨ʹى͖ΔͷͰͦ͏͍ͬͨॲཧΛೖΕΔͱϩόετʹͳΔ ࣍ͷ୯ޠ͕༧ଌͰ͖ͳ͍ͨΊɺ೉қ౓͕Ҿ্͖͕ͬͯਫ਼౓্͕͕Δ
  7. (ݸਓతߟ࡯ͷଓ͖) ݘ͕޷͖Ͱ͢ ΫτΡϧϑਆ࿩͕޷͖Ͱ͢ I like dog I am scared of

    Cthulhu ہॴతϊΠζΛ෇༩ ଟ͘ͷࣗવݴޠॲཧͷϞσϧ͸ গ͠ม͑Δ͚ͩͰ؆୯ʹὃͤΔಛੑ͕͋Δ Deep Text Classification Can be Fooled Liang et al. 2016 ຋༁ ະֶशͷσʔλ ޡࠩٯ఻೻ ͜ͷ໰୊ʹରԠ͢Δଧͪख ʹͳ͍ͬͯΔՄೳੑ ԾʹΫτΡϧϑ͕ປࢺͰ΋ ʮ޷͖ʯ͸ʮlikeʯ (ϊΠζ෦෼ʹޡࠩΛ఻೻͢Δͷ͸׬ᘳʹແବͳͷͰվળͰ͖Δ͔΋)
  8. 5BSHFU 4PVSDF ຊ໋Ϟσϧ 5BSHFU ୯ݴޠσʔλ 4PVSDF ߹੒ 4ZOUIFUJD  ୯ݴޠσʔλ

    ֶश ݩख͕গͳ͍ͱԿ͕ى͜Δ͔ ͜͜ͷྔ͕গͳ͍(80Kจఔ౓) จݿຊ࡭͘Β͍ (1࡭12ສࣈ, 80ࣈ/จ)
  9. ݩख͕গͳ͍໰୊ͷܰݮ 5BSHFU 4PVSDF &ODPEFS %FDPEFS 4PVSDF 4PVSDF 5BSHFU 5BSHFU 4PVSDFݴޠϞσϧ

    5BSHFUݴޠϞσϧ సҠֶशorॏΈڞ༗ సҠֶशorॏΈڞ༗ (1) ୯ݴޠͰݴޠϞσϧΛ࡞ͬͯసҠֶश ʮݴޠϞσϧͷసҠ͕ࠔ೉ʯͱ͍͏໰୊͕Devlin et al. 2018 (BERT)Ͱղফ͞ΕͨͷͰਐల͋Δ͔΋
  10. υϝΠϯదԠ 5BSHFU จষσʔλ 4PVSDF จষσʔλ ຊ໋Ϟσϧ χϡʔε 5BSHFU ୯ݴޠσʔλ χϡʔε

    4PVSDF ߹੒ 4ZOUIFUJD  ୯ݴޠσʔλ ֶश χϡʔεͷର༁σʔλ͕ͳͯ͘΋χϡʔεʹڧ͘ͳΔ͔ʁ