Sequence to Sequence Learning with Neural Networks

࿦จྠಡձ 4FRVFODFUP4FRVFODF-FBSOJOH XJUI/FVSBM/FUXPSLT IUUQTBSYJWPSHBCT ౻ຊ༟հ

"HFOEB ࣄલ஌ࣝ ͜ͷ࿦จʹ͍ͭͯ -45. %FFQ-45.
༻ޠ੔ཧ 1BQFS "CTUSBDU *OUSPEVDUJPO 5IFNPEFM &YQFSJNFOUT Ұ෦ 3FMBUFEXPSL $PODMVTJPO

ࣄલ஌ࣝɿ͜ͷ࿦จʹ͍ͭͯ ౰࣌ ೥ (PPHMF຋༁ͷΞοϓάϨʔυͰ࿩୊ʹͳͬͨ ػց຋༁ͷओྲྀΛม͑ͨ ैདྷ ɿ౷ܭతػց຋༁
4.5 4UBUJTUJDBM.BDIJOF5SBOTMBUJPO ຊ࿦จҎ߱ɿχϡʔϥϧػց຋༁ /.5 /FVSBM.BDIJOF5SBOTMBUJPO IUUQEFFQMFBSOJOHIBUFOBCMPHDPNFOUSZOFVSBM@NBDIJOF@USBOTMBUJPO@UIFPSZ -45.͕ྲྀߦΔ͖͔͚ͬͷͭ ܥྻσʔλΛѻ͏ͳΒ࢝Ίʹީิʹڍ͓͛ͯ͘͘Β͍ʹ͸ྲྀߦͬͯΔΑ͏ͳʜ ը૾ܥʹ΋Ԡ༻͞Ε͍ͯΔ $POW-45. ౳

ࣄલ஌ࣝɿ-45. -45. -POH4IPSU5FSN.FNPSZ IUUQTVQMPBEXJLJNFEJBPSHXJLJQFEJBDPNNPOTC5IF@-45.@DFMMQOH М Λ௨͢ͱ dͷ஋ʹͳΔ
๨٫౓߹͍ͱ͔Λද͢

ࣄલ஌ࣝɿ-45. -45. -POH4IPSU5FSN.FNPSZ IUUQTVQMPBEXJLJNFEJBPSHXJLJQFEJBDPNNPOTC5IF@-45.@DFMMQOH ͦΕ·ͰͷهԱΛɺ΄ͲΑ͘๨Εͯ΄ͲΑ͘Ա͑ͯΔΠϝʔδ O:
output " tanh(Ct) ,EB?>) LA8 KEG; f: forget " #EF96 " i: input " g ,EB3 C H* 7/ %&(< g: = F9 7/ = 7/H* tanh(&. (-1, 1)) Ct: F9 LSTM 'J 1C $4+@ (2D private !!0:) ht: L4 I54- $4@

ࣄલ஌ࣝɿ-45. -45. -POH4IPSU5FSN.FNPSZ هԱηϧ $ʹ͍ͭͯ͸ɺ࣍ͷ $·ͰʹΞμϚʔϧੵͱ଍͠ࢉ͔͠ߦΘΕͳ͍ ޡࠩٯ఻ൖ࣌ͷ଍͠ࢉͱΞμϚʔϧੵΛ͍ͯ͠ΔՕॴͷޯ഑
଍͠ࢉ ྲྀΕ͖ͯͨޯ഑Λͦͷ··ྲྀ͢ ΞμϚʔϧੵ GPSHFUήʔτͰ๨ΕΔ΂͖Ͱͳ͍ͱ൑அ͞Εͨཁૉ͸ফࣦͤͣʹྲྀΕΔ ্هͷ݁Ռޯ഑ফࣦ͕ى͖ͮΒ͍ʢΒ͍͠ʣ f: forget

ࣄલ஌ࣝɿ%FFQ-45. %FFQͳ -45. ૚ɾΠϝʔδਤ <> Dͱ Iͷ্Լ͕ٯͰ͋Δ͜ͱʹ஫ҙ
-45. -45. -45. l"z -45. -45. -45. -45. l#z -45. -45. -45. -45. l&04z -45. 4PGUNBY lЋz -45. -45. -45. -45. 4PGUNBY lЌz lЋz -45. -45. -45. -45. 4PGUNBY l&04z lЌz I D

ࣄલ஌ࣝɿ༻ޠ੔ཧ QISBTFCBTFE4.5 4UBUJTUJDBM.BDIJOF5SBOTMBUJPO lݪݴޠΛ༩͑ͨ࣌ʹର༁ͷ໬౓͕࠷େͱͳΔ֬཰ϞσϧΛֶशͯ͠ ໨తݴޠʹ຋༁͢ΔγεςϜΛࢦ͢z IUUQEFFQMFBSOJOHIBUFOBCMPHDPNFOUSZOFVSBM@NBDIJOF@USBOTMB
UJPO@UIFPSZ χϡʔϥϧػց຋༁ /.5 /FVSBM.BDIJOF5SBOTMBUJPO ʹର͠ɺ ֬཰Ϟσϧʹ //Λ࢖͍ͬͯͳ͍΋ͷΛओʹࢦ͢ #-&6 #J-JOHVBM &WBMVBUJPO6OEFSTUVEZ dͰߴ͍΄ͲΑ͍ IUUQVOJDPSOJLFUPUUPSJ VBDKQTQBQFSHSBEVBUJPOUIFTJTOPEFIUNM

"CTUSBDU େن໛ͳϥϕϧ෇σʔληοτͰ͸ %//͸͏·͘ػೳ͢Δ ͔͠͠จ͔Βจ΁ͷϚοϐϯάʹ͸࢖͑ͳ͔ͬͨ ຊߘͰ͸ छྨͷଟ૚ -45.Λ࢖ͬͨ
ೖྗจΛݻఆ࣍ݩͷϕΫτϧʹϚοϐϯά͢Δ -45. ϕΫτϧ͔Βର৅ݴޠͷจʹσίʔυ͢Δ -45. 8.5`EBUBTFUͰ੒Ռग़ͨ #-&6TDPSF &OHMJTI'SFODIλεΫ QISBTFCBTFE4.5γεςϜͰ͸ ্ه 4.5γεςϜʹΑͬͯग़ྗ͞Εͨ ݸͷԾઆʁΛ-45. ʹΑͬͯ࠶ϥϯΫ෇͚͢Δͱɺ ʹ૿Ճ ۩ମతʹ͸ޙड़͞Ε͍ͯΔʁ ͜Ε͸౰࣌ͷ 405"ʹ͍ۙείΞ

"CTUSBDU -45.͸ޠॱʹରͯ͠ӶහʢޠॱͰҙຯ͕େ͖͘มΘΔ ౳ʣ۟΍จͷදݱΛֶशͨ͠ ·ͨ͜ͷදݱ͸ೳಈଶ BDUJWFWPJDF ΍डಈଶ QBTTJWF WPJDF
ʹରͯ͠ෆม ೳಈଶओޠ ಈࢺ ໨తޠ<˞> डಈଶओޠ #Fಈࢺ ʴ աڈ෼ࢺ CZ ೳಈଶͷओޠ <˞> ྫʮࢲ͸͜ͷ࿦จΛಡΜͩʯ㲈ʮ͜ͷ࿦จ͸ࢲʹಡ·Εͨʯ ೖྗจΛٯॱʹֶͯ͠श͢Δͱ -45.ͷਫ਼౓͕ܶతʹ޲্ ͢Δ͜ͱΛൃݟͨ͠ ্هͷૢ࡞ʹΑΓೖྗจͱର৅จͷؒͷ୹͍۠ؒͰͷґଘؔ܎ TIPSUUFSNEFQFOEFODJFT ͕ଟ͘΋ͨΒ͞ΕͨͨΊ ্هʹΑͬͯ࠷దԽ໰୊͕؆୯ʹͳͬͨ ͭΊͷ -45.͸ TFRVFODFWFDUPSͱ Ұͭલͷର৅จͷ୯ޠ͔ Β࣍ͷ୯ޠΛग़ྗ͢ΔͷͰɺ࠷ॳͷ୯ޠ͕େࣄͬΆ͍ʁ <˞>IUUQTXXXLBQMBOJOUFSOBUJPOBMDPNKQCMPHBDUJWFQBTTJWFWPJDF

*OUSPEVDUJPO %FFQ/FVSBM/FUXPSLT %//T ͸ڧྗͳػցֶशϞσϧ ฒྻܭࢉՄೳɺදݱྗߴ͍ ϥϕϧ෇͖σʔλ͕े෼ͳ৘ใΛ͍࣋ͬͯΕ͹ɺେ͖͍
%//΋ ڭ ࢣ෇͖ CBDLQSPQBHBUJPOʹΑֶͬͯशՄೳ ͕ͨͬͯ͠ྑ͍݁ՌΛୡ੒Ͱ͖Δ %//ͷύϥϝʔλ͕ଘࡏ͢Ε͹ɺ ڭࢣ෇͖ CBDLQSPQBHBUJPO͸ύϥϝʔλΛݟ͚ͭΒΕΔ ͨͩ͠ɺ%//͸ ೖྗͱλʔήοτจΛݻఆ࣍ݩͷϕΫτϧ ʹΤϯίʔυͰ͖Δ໰୊ʹ͔͠ద༻Ͱ͖ͳ͍ ڧ੍͍ݶ ܥྻ໰୊ʹ͓͍ͯܥྻ௕͸ࣄલʹ෼͔Βͳ͍͜ͱ͕ଟ͍ͨΊ Ի੠ೝࣝ΍ػց຋༁΋ܥྻ໰୊ ࣭ٙԠ౴͸ɺ࣭໰จΛճ౴จʹϚοϐϯά͢Δ໰୊ͱ΋ΈΕΔ υϝΠϯʹґଘͤͣʹܥྻ͔Βܥྻ΁ͷϚοϐϯάΛֶश͢Δํ๏͸ ༗༻

*OUSPEVDUJPO ݻఆ࣍ݩ໰୊ʹର͠ɺ-45.Λ௚઀దԠͯ͠ରԠ ͭΊͷ -45.Ͱೖྗจͷݻఆ௕දݱϕΫτϧΛಘΔ ͭΊͷ -45.Ͱग़ྗจΛ্هͷϕΫτϧ͔Βੜ੒͢Δ
ͭΊͷ -45.͸ຊ࣭తʹ͸ 3//MBOHVBHFNPEFM ͜ͷํ๏Ͱ͸ೖྗจͱͦΕʹରԠͨ͠ग़ྗจΛੜ੒͢Δ·ͰλΠϜϥ ά͕͋Δ ௕ڑ཭ͷ࣌ؒґଘੑʹର্ͯ͠खֶ͘शͰ͖Δ -45.Λબ୒͢Δͷ ͸ଥ౰ -45. -45.

*OUSPEVDUJPO ୡ੒ͨ͠είΞͱطଘ CBTFMJOFͱͷൺֱ #-&6TDPSFPG FOTFNCMFPGEFFQ-45.T
.QBSBNFUFSTBOE EJNFOTJPOBMTUBUFFBDI ࢀߟ7((ͷύϥϝʔλ਺͕໿ . VTJOHBTJNQMFMFGUUPSJHIUCFBNTFBSDIEFDPEFS CFBNTFBSDI ޙड़ େ͖͍ //ʹΑΔ EJSFDUUSBOTMBUJPOͰ͸͘͢͝ྑ͍݁Ռ Ұํ 4.5γεςϜʹΑΔείΞ CBTFMJOF͸ ্هͷ -45.ʹΑΔείΞ͸ޠኮ਺ L ධՁର৅จ͕͜ͷ Lݸʹؚ·Εͳ͍ޠኮΛؚΜͰ͍Δ৔߹ ˠ #-&6TDPSF͕Լ͕Δ L͸খن໛ Β͍͠ ·ͩ࠷దԽͷ༨஍͕͋Δ VOPQUJNJ[FE ޠኮ਺΍ͲͷޠኮΛબͿ͔ʁ ࠷దԽͷ༨஍Λ࢒ͭͭ͠΋ QISBTFCBTFE4.5TZTUFNΛ௒͍͑ͯ ΔͷͰ -45.CBTF͕༏Ε͍ͯΔ͜ͱΛ֬ೝͰ͖Δ

*OUSPEVDUJPO lCFTUMJTUTPGUIF4.5CBTFMJOFzʹର͠ -45.Λ ࢖ͬͯϦείΞͨ͠ΒείΞ͕ ʹվળ طଘϞσϧͷϕετείΞ͕ ͳͷͰͦΕʹ͍ۙ
-45.͸௕จʹۤઓ͠ͳ͔ͬͨ ͜Ε·ͰͷྨࣅͷݚڀΛͯ͠Δਓ͸ۤઓͨ͠ܦݧ͕͋Δ ೖྗจͷޠॱΛٯʹͨ͠Өڹ͕େ͖͍ ग़ྗจͷޠॱ͸ͦͷ·· ͦΕʹΑΓ࠷దԽ໰୊Λ؆୯ʹ͢Δ୹ظతͳґଘؔ܎Λ૿΍ͤͨͨΊ TFDUJPOͱ Ͱ৮Ε·͢ ͦͷ͓͔͛Ͱ 4(%Ͱ໰୊ͳֶ͘शͰ͖ͨ ͜ͷ࿦จͷ LFZUFDIOJDBMDPOUSJCVUJPOͷͭ

*OUSPEVDUJPO -45.͸ೖྗจͷݻఆ࣍ݩ΁ͷϚοϐϯάΛֶश͢Δ ຋༁͸ݩͷจͷݴ͍׵͑Ͱ͋Δ͜ͱ͕ଟ͍ ҧ͏৔߹͕͋Δʁʢݸਓతٙ໰ʣ ຋༁Λ໨తͱͨ͠৔߹ɺ-45.͕จষͷҙຯදݱ
ݻఆ࣍ݩ ϕΫτϧ Λ֫ಘ͢Δ͜ͱΛޙԡ͢͠Δ ҙຯ͕ࣅ͍ͯΔͳΒ͍ۙɺҙຯ͕ҧ͏ͳΒԕ͍ ݁Ռͱͯ͠ҙຯදݱΛಘͨͱ͍͏෦෼͸ XPSEWFDʹࣅ͍ͯΔʁ ͜ͷޙͷఆੑతධՁͰ֬ೝͨ͠ ಛʹೳಈଶͱडಈଶʹର͠ϩόετͰ͋Δ͜ͱΛ֬ೝͨ͠

5IFNPEFM 3FDVSSFOU/FVSBM/FUXPSL 3// ͸ܥྻੜ੒ͷࣗવͳҰൠ ԽϞσϧ *OQVU(x1 ,
x2 , ... , xT ) 0VUQVU(y1 , y2 , ... , yT ) ௕͕͞ҰॹͰ͋Δ͜ͱʹ஫ҙ 3//͸ೖྗจͱग़ྗจͷରԠ͕୯७ͳ৔߹͸༗ޮ ௕͕͞ಉ͡ ೖྗจͷ୯ޠͱग़ྗจͷ୯ޠ͕ POରԠ౳ ௕͕͞ҧͬͨΓରԠ͕ෳࡶͰ͋Δ৔߹͸೉͍͠ ௚લͷӅΕঢ়ଶ Λ࢖͍ͬͯΔ

5IFNPEFM 3//ͰҰൠతͳܥྻ໰୊ʹରԠ͢ΔͨΊʹ͸ʜ ͭͷ 3// 3//ͱ 3// Λ࢖ͬͯԼه
TUFQ TUFQ3//ͰೖྗจΛݻఆ࣍ݩϕΫτϧʹϚοϐϯά TUFQ3//Ͱݻఆ࣍ݩϕΫτϧ͔Βग़ྗจΛϚοϐϯά ݪཧతʹ͸શͯͷؔ࿈৘ใ͕ 3//ʹ༩͑ΒΕ͍ͯΔͷͰ͏ ·ֶ͘शͰ͖Δ͸ͣ ͔͠͠ 3//Ͱ͸௕ظతͳґଘؔ܎Λֶशͮ͠Β͍ -45.

5IFNPEFM -45.ͷΰʔϧ p(ग़ྗจ | ೖྗจ) ΛٻΊΔ ԼهͷΑ͏ʹ৚݅෇͖֬཰ΛఆࣜԽ
ೖྗจͷදݱϕΫτϧΛ࠷ॳͷӅΕঢ়ଶͱͨ͠ -45.-BOHVBHF .PEFMΛߏ੒͢Δ ֤୯ޠͷ֬཰͸ޠኮ਺࣍ݩͷ TPGUNBY Ͱදݱ͞ΕΔ Ճ͑ͯจষͷऴΘΓͱͯ͠ l&04zͱ͍͏ޠኮ΋௥Ճͨ͠ GJH εϥΠυ Q ͷྫ ೖྗl"z l#z l$z l&04z ग़ྗl8z l9z l:z l;z l&04z ͜ͷఆࣜԽͰ͸ -45.͸ ͭͰ΋ྑ͍૝ఆ ೖྗจͱ௕͕͞ ҧ͏Մೳੑ͋Γ ೖྗจͷදݱ ϕΫτϧ

5IFNPEFM ࣮ࡍʹ࢖ͬͨϞσϧ͸ҎԼͷҧ͍͕͋Δ छྨͷ -45.TΛ࢖ͬͨ ϞσϧͷύϥϝʔλΛ૿΍ͯ͠΋ܭࢉίετ͕͋·Γ૿͑ͳ͍ͨΊ
ಉ࣌ʹෳ਺ͷݴޠϖΞΛࣗવʹֶशͰ͖ΔͨΊ EFFQͳ -45.TΛ࢖ͬͨ TIBMMPX-45.TΑΓਫ਼౓ྑ͔ͬͨ lTPXFDIPTFBO-45.XJUIGPVSMBZFSTz ೖྗจͷޠॱΛٯʹͨ͠ ྫ lB C DzΛ lЋ Ќ Ѝzͱ຋༁͍ͨ͠৔߹ -45.ʹ͸ lD C BzΛ lЋ Ќ ЍzʹϚοϐϯά͢ΔΑ͏ʹཁٻ͢Δ ্هͷྫͰ͸ lBzͱ lЋz͕ۙ઀ ೖྗ୯ޠͱग़ྗ୯ޠͷฏۉڑ཭͸มΘΒͳ͍ ͜ͷ͓͔͛Ͱ 4(%͸ೖྗจͱग़ྗจͷؒͷ l΍ΓͱΓΛཱ֬z͠΍͘͢ ͳͬͨ ݸਓॴײ -45.-.΋࠷ॳΒ΁Μͷ୯ޠ͕େࣄͳͷͰɺ·ͣ͸ͦ͜Λ ͔ͬ͠Γग़ྗ͢ΔΑ͏ʹֶश͢Δ͜ͱͰ࠷ॳҎ߱ͷ୯ޠ΋ֶश͠΍ͦ͢͏

&YQFSJNFOU %BUBTFUEFUBJMT ࣮ݧ͸ ௨Γͷํ๏Ͱߦͬͨ 4.5γεςϜͷग़ྗΛࢀߟʹͤͣʹ௚઀຋༁ 4.5γεςϜͷग़ྗΛ
SFTDPSF %BUBTFUEFUBJMT 8.5`&OHMJTIUP'SFODIEBUBTFU USBJO .TFOUFODFT .'SFODIXPSETBOE.&OHMJTIXPSET UFTU CFTUMJTUTGSPNUIFCBTFMJOF4.5 ޠኮ਺ʹ͍ͭͯ TPGUNBY ౳Ͱݻఆ࣍ݩʹ͢Δඞཁ͋Γ ೖྗจ &OHMJTI ͔Β͸සग़ ޠኮ ग़ྗจ 'SFODI ͔Β͸සग़ ޠኮ ্هʹଐ͞ͳ͍୯ޠ͸શͯ l6/,zͱͨ͠

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH ܇࿅࣌͸Լهͷ࠷େԽΛ໨తؔ਺ͱֶͯ͠श ༧ଌ ຋༁จੜ੒ ࣌͸ԼهͷࣜΛຬͨ͢Α͏ͳจΛ୳͢
୯ޠͣͭ࠷΋֬཰͕ߴ͍୯ޠΛબ΂͹Α͍ͷͰ͸ʁ ˠ ֤୯ޠ͕ಠཱʹੜ੒͞Ε͍ͯΔΘ͚Ͱ͸ͳ͍ͷʹʁ Ͳ͏΍ͬͯ୳͢ͷ͔ ୯७ͳΫϥεͱ͸୳ࡧൣғ͕͔ͳΓҧ͏ จ௕Λݻఆͯ͠΋ɺ୳ࡧൣғ͸ޠኮ਺?จ௕ ྫ ͷ ৐ ࣮ࡍʹ͸จ௕͸ݻఆ͍ͯ͠ͳ͍ͷͰࣄ্࣮ແݶ ˠ lTJNQMFMFGUUPSJHIUCFBNTFBSDIEFDPEFSzͰղܾ ͍͍ײ͡ʹ୯ޠΛॱ൪ʹੜ੒͢Δํ๏ ग़ྗ࣌ʹ TPGUNBY Λ௨ͨ͠෼෍Λ ಘΒΕΔ͚Ͳɺͦ͜Ͱग़ྗΛ lXz ʹ֬ఆͯ͠͠·ͬͯΑ͍ͷ͔ʁ!

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH CFBNTFBSDI ϏʔϜ୳ࡧ <> ۙࣅత୳ࡧ๏ͷͭ
ղ͖͍ͨ໰୊ ࠓճ͸࠷େ஋ΛͱΔ୯ޠྻͷ୳ࡧ Λ෦෼໰୊ʹ෼ׂ֤ ෦෼໰୊Λղ͍͍ͯ͘ ࠷ॳͷ୯ޠ͔Βॱ൪ʹ୯ޠͣͭબΜͰ͍͘Πϝʔδ ֤ॱʹ࠷దͳ୯ޠΛબΜͰ΋࠷దղʹͳΒͳ͍Մೳੑ͕͋Δ ଞʹܥྻม׵ϞσϧͰ༻͍ΒΕΔͷ͸ᩦཉ๏ HSFFEZBMHPSJUIN ֤෦෼໰୊ͷධՁ஋͕΋ͬͱ΋ߴ͍΋ͷΛͭબ୒͢Δ ϏʔϜ୳ࡧͰ͸ɺࣄલʹܾΊ্ͨҐ ,ݸͷީิΛอ࣋ͭͭ͠ॲཧΛਐ ΊΔ ,͸ CFBNTJ[F ʹ͋ͨΔ ᩦཉ๏͸ϏʔϜ୳ࡧͷಛघͳ৔߹ , ͱ΋ߟ͑ΒΕΔ ࣌ࠁ K ࠨ͔Β਺͑ͯ K൪໨ͷ୯ޠ ͷ༧ଌʹ͓͍ͯɺҐஔ͔Β K· Ͱͷྦྷੵੜ੒֬཰͕୳ࡧൣғ಺Ͱ্Ґ ,ݸͷ༧ଌ݁ՌΛग़ྗީิͱ ͯ͠อ࣋͢Δ ᩦཉ๏ΑΓਫ਼౓͕ߴ͍͕ ,ഒͷܭࢉίετ

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH CFBNTFBSDI ϏʔϜ୳ࡧ ͷٖࣅίʔυ<>

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH CFBNTJ[Fʹ͍ͭͯ CFBNTJ[FͰ΋্ख͘ػೳͨ͠ CFBNTJ[F͕࠷΋ར఺͕͋ͬͨ
-45.TXJUIBCFBNTJ[Fͷํ͕ TJOHMF-45.XJUIBCFBN TJ[FΑΓ DIFBQFS ܭࢉίεύ͕ྑ͍ʁ Ճ͑ͯ -45.Ͱ CBTFMJOFTZTUFNͷ CFTUMJTUTΛ SFTDPSFͨ͠ શͯͷ IZQPUIFTJTʹ͍ͭͯ MPHQSPCBCJMJUZͱͬͯ -45.ͷ IZQPUIFTJT͔Βͱͬͨ΋ͷͱฏۉͯ͠ OFXTDPSFͱͨ͠

&YQFSJNFOU 3FWFSTJOHUIF4PVSDF4FOUFODFT ೖྗจΛٯॱʹͨ͠Βਫ਼౓͕͔ͳΓߴ͘ͳͬͨͷΛൃݟͨ͠ Ͳ͏ͯ͜͠͏ͳΔͷ͔׬શͳઆ໌Ͱ͖ͳ͍ ୹ظతͳґଘؔ܎Λଟֶ͘शʹಋೖͰ͖ͨͨΊͩͱࢥΘΕΔ
ೖྗจͱର৅จΛͨͩ DPODBU ͢Δͱɺೖྗจͷ֤୯ޠͱର৅จͷ֤ ୯ޠͷڑ཭͸௕͘ͳΔ lNJOJNBMUJNFMBHz͕ى͖͍ͯΔ ࠓճ͜Ε͸ௐ΂ͯͳ͍Ͱ͢ ͨͩ͠ೖྗจΛٯॱʹ͢Δͱɺฏۉڑ཭͸มΘΒͳ͍͕ɺ࠷ॳͷ਺୯ ޠͷڑ཭͸୹͘ͳΔ ˠ lNJOJNBMUJNFMBHz͕ݮগ ޡࠩ఻ൖָ͕ʹͳΔ ౰ॳ͸จͷલ൒͸ྑ͘ͳͬͯޙ൒͸ѱ͘ͳΔͱࢥͬͯͨ શମͰਫ਼౓ྑ͘ͳͬͨ

3FMBUFEXPSL ͜Ε·ͰݟͨதͰ؆୯ͰޮՌతͳํ๏͸ //-.ϕʔε 3//-BOHVBHF.PEFM 3//-. 'FFEGPSXBSE/FVSBM/FUXPSL-BOHVBHF.PEFM
//-. վྑ൛ //-. 5PQJDNPEFMͱ૊Έ߹Θͤͯਫ਼౓޲্ "VMJ FUBM %FWMJOΒ͸্هʹ͞Βʹ৭ʑՃ͑ɺߴ͍ਫ਼౓Λ࣮ݱ //-.Λ .5γεςϜͷσίʔμʹ૊ΈࠐΉ σίʔμͷΞϥΠϝϯτ৘ใΛ࢖͏͜ͱͰɺೖྗจதͰ༗༻ͳ୯ޠΛΛ //-.ʹ༩͑ΔΑ͏ʹͨ͠

3FMBUFEXPSL ຊݚڀ͸ ,BMDICSFOOFS BOE#MVOTPN ͷख๏ʹ͍ۙ Ұ౓ೖྗจΛϕΫτϧʹϚοϐϯάͯ͠ग़ྗจΛϚοϐϯά
ͨͩ͠ $//ͰϕΫτϧʹϚοϐϯάͯͨ͠ͷͰޠॱͷ৘ใΛࣦͬͨ $IPΒ͸ -45.MJLFͳ 3//Λ࢖͕ͬͨɺ͋͘·Ͱ 4.5γεςϜ ʹ૊ΈࠐΉͨΊͩͬͨ #BIEBOBV Β͸ $IPΒͷख๏ʹର͠ "UUFOUJPO ஫ҙػߏ Λ࢖ͬͯ ௕จʹ͓͚ΔੑೳΛվળͨ͠ 1PVHFU"CBEJFΒ͸ $IPFUΒͷख๏ʹରͯ͠෦෼తʹ຋༁Λ͢Δ ͜ͱͰεϜʔζͳ຋༁Λੜ੒͠Α͏ͱͨ͠ ্هख๏ʹରͯ͠΋ɺೖྗจΛٯॱʹ͢Δ͚ͩͰվળ͢ΔՄ ೳੑΛײͯ͡Δ )FSNBOOΒ͸ FOEUPFOEͰೖྗจͱग़ྗจΛۭؒ಺ͷ࠲ ඪʹϚοϐϯά͢ΔϞσϧΛߟ͑ͨ ͔͠͠Ϛοϐϯά͢Δ͚ͩ ຋༁͢Δʹ͸Ұ൪͍ۙ఺Λ୳͔͢ɺSFTDPSF͢Δඞཁ͕͋ͬͨ

$PODMVTJPO ैདྷͷ 4.5CBTFETZTUFNΛ௒͑ͨ ೖྗจΛٯॱʹ͢Δͱਫ਼౓͕͔ͳΓྑ͘ͳͬͨ TIPSUUFSNEFQFOEFODJFT ୹͍۠ؒͰͷґଘؔ܎
͕૿͑ͯɺ࠷ద Խ͕؆୯ʹͳͬͨͱࢥΘΕΔͨΊ ͓ͦΒ͘ 3//Ͱͷྑ͘ͳΔ͜ͱ͕૝ఆ͞ΕΔ ຊߘͰ͸ࢼͯ͠ͳ͍ -45.͸ඇৗʹ௕͍จষʹରͯ͠΋຋༁ೳྗ͕ߴ͍ ୯७Ͱ௚઀ 4FR4FRͰ͖ΔϞσϧʹ΋ؔΘΒͣɺطଘͷ 4.5Λϕʔεͱͨ͠γεςϜΑΓྑ͔ͬͨ

͓·͚Ϟσϧͷύϥϝʔλ਺ͷٻΊํ 1Z5PSDI ࢖ͬͯΔͳΒ QZUPSDITVNNBSZ͕ खܰʹݟΕ͓ͯ͢͢ΊͰ͢ ,FSBT Ͱ͋Ε͹ͨͩ
QSJOU͢Δ͚ͩͰݟΕ·͕͢ʜ

ࢀߟࢿྉ <>ݩ࿦จ IUUQTBSYJWPSHBCT <>ਂ૚ֶशʹΑΔࣗવݴޠॲཧ <>θϩ͔Β࡞Δ %FFQ-FBSOJOHᶄ
ࣗવݴޠॲཧฤ

Sequence to Sequence Learning with Neural Networks

Sequence to Sequence Learning with Neural Networks

fam_taro

More Decks by fam_taro

Other Decks in Science

Featured

Transcript

࿦จྠಡձ 4FRVFODFUP4FRVFODF-FBSOJOH XJUI/FVSBM/FUXPSLT IUUQTBSYJWPSHBCT ౻ຊ༟հ

"HFOEB ࣄલ஌ࣝ ͜ͷ࿦จʹ͍ͭͯ -45. %FFQ-45.

ࣄલ஌ࣝɿ͜ͷ࿦จʹ͍ͭͯ ౰࣌ ೥ (PPHMF຋༁ͷΞοϓάϨʔυͰ࿩୊ʹͳͬͨ ػց຋༁ͷओྲྀΛม͑ͨ ैདྷ ɿ౷ܭతػց຋༁

ࣄલ஌ࣝɿ-45. -45. -POH4IPSU5FSN.FNPSZ IUUQTVQMPBEXJLJNFEJBPSHXJLJQFEJBDPNNPOTC5IF@-45.@DFMMQOH М Λ௨͢ͱ dͷ஋ʹͳΔ

ࣄલ஌ࣝɿ-45. -45. -POH4IPSU5FSN.FNPSZ IUUQTVQMPBEXJLJNFEJBPSHXJLJQFEJBDPNNPOTC5IF@-45.@DFMMQOH ͦΕ·ͰͷهԱΛɺ΄ͲΑ͘๨Εͯ΄ͲΑ͘Ա͑ͯΔΠϝʔδ O:

ࣄલ஌ࣝɿ-45. -45. -POH4IPSU5FSN.FNPSZ هԱηϧ $ʹ͍ͭͯ͸ɺ࣍ͷ $·ͰʹΞμϚʔϧੵͱ଍͠ࢉ͔͠ߦΘΕͳ͍ ޡࠩٯ఻ൖ࣌ͷ଍͠ࢉͱΞμϚʔϧੵΛ͍ͯ͠ΔՕॴͷޯ഑

ࣄલ஌ࣝɿ%FFQ-45. %FFQͳ -45. ૚ɾΠϝʔδਤ <> Dͱ Iͷ্Լ͕ٯͰ͋Δ͜ͱʹ஫ҙ

ࣄલ஌ࣝɿ༻ޠ੔ཧ QISBTFCBTFE4.5 4UBUJTUJDBM.BDIJOF5SBOTMBUJPO lݪݴޠΛ༩͑ͨ࣌ʹର༁ͷ໬౓͕࠷େͱͳΔ֬཰ϞσϧΛֶशͯ͠ ໨తݴޠʹ຋༁͢ΔγεςϜΛࢦ͢z IUUQEFFQMFBSOJOHIBUFOBCMPHDPNFOUSZOFVSBM@NBDIJOF@USBOTMB

"CTUSBDU େن໛ͳϥϕϧ෇σʔληοτͰ͸ %//͸͏·͘ػೳ͢Δ ͔͠͠จ͔Βจ΁ͷϚοϐϯάʹ͸࢖͑ͳ͔ͬͨ ຊߘͰ͸ छྨͷଟ૚ -45.Λ࢖ͬͨ

"CTUSBDU -45.͸ޠॱʹରͯ͠ӶහʢޠॱͰҙຯ͕େ͖͘มΘΔ ౳ʣ۟΍จͷදݱΛֶशͨ͠ ·ͨ͜ͷදݱ͸ೳಈଶ BDUJWFWPJDF ΍डಈଶ QBTTJWF WPJDF

*OUSPEVDUJPO %FFQ/FVSBM/FUXPSLT %//T ͸ڧྗͳػցֶशϞσϧ ฒྻܭࢉՄೳɺදݱྗߴ͍ ϥϕϧ෇͖σʔλ͕े෼ͳ৘ใΛ͍࣋ͬͯΕ͹ɺେ͖͍

*OUSPEVDUJPO ݻఆ࣍ݩ໰୊ʹର͠ɺ-45.Λ௚઀దԠͯ͠ରԠ ͭΊͷ -45.Ͱೖྗจͷݻఆ௕දݱϕΫτϧΛಘΔ ͭΊͷ -45.Ͱग़ྗจΛ্هͷϕΫτϧ͔Βੜ੒͢Δ

*OUSPEVDUJPO ୡ੒ͨ͠είΞͱطଘ CBTFMJOFͱͷൺֱ #-&6TDPSFPG FOTFNCMFPGEFFQ-45.T

*OUSPEVDUJPO lCFTUMJTUTPGUIF4.5CBTFMJOFzʹର͠ -45.Λ ࢖ͬͯϦείΞͨ͠ΒείΞ͕ ʹվળ طଘϞσϧͷϕετείΞ͕ ͳͷͰͦΕʹ͍ۙ

*OUSPEVDUJPO -45.͸ೖྗจͷݻఆ࣍ݩ΁ͷϚοϐϯάΛֶश͢Δ ຋༁͸ݩͷจͷݴ͍׵͑Ͱ͋Δ͜ͱ͕ଟ͍ ҧ͏৔߹͕͋Δʁʢݸਓతٙ໰ʣ ຋༁Λ໨తͱͨ͠৔߹ɺ-45.͕จষͷҙຯදݱ

5IFNPEFM 3FDVSSFOU/FVSBM/FUXPSL 3// ͸ܥྻੜ੒ͷࣗવͳҰൠ ԽϞσϧ *OQVU(x1 ,

5IFNPEFM 3//ͰҰൠతͳܥྻ໰୊ʹରԠ͢ΔͨΊʹ͸ʜ ͭͷ 3// 3//ͱ 3// Λ࢖ͬͯԼه

5IFNPEFM -45.ͷΰʔϧ p(ग़ྗจ | ೖྗจ) ΛٻΊΔ ԼهͷΑ͏ʹ৚݅෇͖֬཰ΛఆࣜԽ

5IFNPEFM ࣮ࡍʹ࢖ͬͨϞσϧ͸ҎԼͷҧ͍͕͋Δ छྨͷ -45.TΛ࢖ͬͨ ϞσϧͷύϥϝʔλΛ૿΍ͯ͠΋ܭࢉίετ͕͋·Γ૿͑ͳ͍ͨΊ

&YQFSJNFOU %BUBTFUEFUBJMT ࣮ݧ͸ ௨Γͷํ๏Ͱߦͬͨ 4.5γεςϜͷग़ྗΛࢀߟʹͤͣʹ௚઀຋༁ 4.5γεςϜͷग़ྗΛ

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH ܇࿅࣌͸Լهͷ࠷େԽΛ໨తؔ਺ͱֶͯ͠श ༧ଌ ຋༁จੜ੒ ࣌͸ԼهͷࣜΛຬͨ͢Α͏ͳจΛ୳͢

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH CFBNTFBSDI ϏʔϜ୳ࡧ <> ۙࣅత୳ࡧ๏ͷͭ

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH CFBNTFBSDI ϏʔϜ୳ࡧ ͷٖࣅίʔυ<>

&YQFSJNFOU %FDPEJOHBOE3FTDPSJOH CFBNTJ[Fʹ͍ͭͯ CFBNTJ[FͰ΋্ख͘ػೳͨ͠ CFBNTJ[F͕࠷΋ར఺͕͋ͬͨ

&YQFSJNFOU 3FWFSTJOHUIF4PVSDF4FOUFODFT ೖྗจΛٯॱʹͨ͠Βਫ਼౓͕͔ͳΓߴ͘ͳͬͨͷΛൃݟͨ͠ Ͳ͏ͯ͜͠͏ͳΔͷ͔׬શͳઆ໌Ͱ͖ͳ͍ ୹ظతͳґଘؔ܎Λଟֶ͘शʹಋೖͰ͖ͨͨΊͩͱࢥΘΕΔ

3FMBUFEXPSL ͜Ε·ͰݟͨதͰ؆୯ͰޮՌతͳํ๏͸ //-.ϕʔε 3//-BOHVBHF.PEFM 3//-. 'FFEGPSXBSE/FVSBM/FUXPSL-BOHVBHF.PEFM

3FMBUFEXPSL ຊݚڀ͸ ,BMDICSFOOFS BOE#MVOTPN ͷख๏ʹ͍ۙ Ұ౓ೖྗจΛϕΫτϧʹϚοϐϯάͯ͠ग़ྗจΛϚοϐϯά

$PODMVTJPO ैདྷͷ 4.5CBTFETZTUFNΛ௒͑ͨ ೖྗจΛٯॱʹ͢Δͱਫ਼౓͕͔ͳΓྑ͘ͳͬͨ TIPSUUFSNEFQFOEFODJFT ୹͍۠ؒͰͷґଘؔ܎

͓·͚Ϟσϧͷύϥϝʔλ਺ͷٻΊํ 1Z5PSDI ࢖ͬͯΔͳΒ QZUPSDITVNNBSZ͕ खܰʹݟΕ͓ͯ͢͢ΊͰ͢ ,FSBT Ͱ͋Ε͹ͨͩ

ࢀߟࢿྉ <>ݩ࿦จ IUUQTBSYJWPSHBCT <>ਂ૚ֶशʹΑΔࣗવݴޠॲཧ <>θϩ͔Β࡞Δ %FFQ-FBSOJOHᶄ