TensorFlow & DeepMind Lab & UNREAL

TensorFlowで実装したUNREALアルゴリズムでDeepMind Labの3D迷路を解く

Transcript

1. 5FOTPS'MPX
   
   %FFQNJOE
   -BC OBSSBUJWF
   OJHIUTגࣜձࣾ
   
   ࡾ޷߁༞ 5FOTPS'MPX
   6TFS
   (SPVQ
   
   
   
   
   
2. %FFQ.JOE
   -BC
3. 6/3&"- ڧԽֶशͷ"$ΞϧΰϦζϜΛϕʔεʹ&YQFSJFODF
   3FQMBZΛ࢖ͬͨิॿλεΫΛ૊Έ߹Θͤͯ%໎࿏Ͱ Yഒͷֶशͷߴ଎ԽΛ࣮ݱ REINFORCEMENT LEARNING WITH UNSUPERVISED AUXILIARY TASKS 
 Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki et. al (DeepMind, 2016)
4. ಈ෺ͷເ w ಈ෺͸ເͷதͰܦݧͨ͠ग़དྷࣄΛ࠶ݱ ϦϓϨΠ ͠ ͳ͕Βւഅ৽ൽ࣭΁ͷهԱͷݻఆΛߦ͍ͬͯΔ
   w ߠఆత൱ఆతͳใुʹؔΘΔग़དྷࣄͷເΛಘʹස ൟʹݟֶͯशΛߦ͍ͬͯΔ
   w FYʮਫҿΈ৔ͰϥΠΦϯΛݟ͔͚ͯةݥͳ໨ʹ͋ͬ ͨʯ
   w 6/3&"-Ͱ͸͜ΕΛώϯτʹ͍ͯ͠Δ
5. ڧԽֶश ؀ڥ ΤʔδΣϯτ "DUJPO ⬆ ➡ ⬇ ঢ়ଶ T ใु S
6. 6/3&"-΁ͷྲྀΕ %2/ "$ 6/3&"-   
7. "$ "TZODISPOPVT
   "EWBODFE
   "DUPS$SJUJD
   w ෳ਺ͷ؀ڥΛඇಉظʹฒྻʹಈֶ͔ͯ͠शΛߴ଎Խ ҆ఆԽͤͨ͞
8. К 1PMJDZ 7 ֤"DUJPOΛऔΔ֬཰ ݱࡏͷঢ়ଶՁ஋ ⬆ ➡ ⬇ TPGUNBY MJOFBS $POW $POW '$ -45. "$ͷωοτϫʔΫߏ଄
9. ֤-PDBM
   /FUXPSLͰ͸ɺֶश݁Ռͷޯ഑ EВ ͷΈΛٻΊɺ
   ΢ΣΠτʹ൓өͤͣ(MPCBMͷ΢ΣΠτ В ʹݸผʹ൓өɻ
   (MPCBMͷ΢ΣΠτΛ·֤ͨ-PDBM
   ͷ΢ΣΠτʹίϐʔɻ EВ EВ EВ EВ В ʜ
10. 1PMJDZ К
    7
    ͷޯ഑ R= = = w 7͸3ʹ͚ۙͮΔ༷ʹߋ৽
    w 37͕ਖ਼ͳΒɺऔͬͨBDUJPO͕ग़Δ֬཰Λ૿΍༷͢ʹߋ৽
 37͕ෛͳΒɺऔͬͨBDUJPO͕ग़Δ֬཰ΛݮΒ༷͢ʹߋ৽
    
 V network: Policy network: ˞্هͷදهͰ7͸(SBEJFOU
    %FTDFOU
    1PMJDZ͸(SBEJFOU
    "TDFOU
    
    
    
    
    
    
    
    
    
    θv = θv - α * dθv, θ = θ + α * dθ 1PMJDZ 7
11. 6/3&"- w "$ʹɺ&YQFSJFODF
    3FQMBZΛޮՌతʹ࢖ͬͨิ ॿλεΫΛಋೖ͠ɺ͞ΒʹֶशΛߴ଎Խͤ͞Δ
    w 1JYFM
    $POUSPM
    w 3FXBSE
    1SFEJDUJPO
    w 7BMVF
    'VODUJPO
    3FQMBZ 6/TVQFSWJTFE
    3&JOGPSDFNFOU
    "VYJMJBSZ
    -FBSOJOH
12. &YQFSJFODF
    3FQMBZ w <ঢ়ଶ
    "DUJPO
    ใु
    ࣍ঢ়ଶ>
    ͷϖΞΛେྔʹอଘ͠ ͯɺ͔ͦ͜ΒαϯϓϦϯάͯ͠ωοτϫʔΫΛֶश
    w %2/͸ɺ͜Ε͕ͳ͍ͱֶश͕҆ఆ͠ͳ͔ͬͨ
    w "$Ͱ͸࢖͍ͬͯͳ͍
14. 1JYFM
    $POUSPM w ը໘ͷϐΫηϧ஋ͷมԽྔΛΑΓେ͖͘͢Δ༷ʹ͞ ͍ͤͨ
    w ը໘ͷϐΫηϧ஋ͷมԽΛٖࣅใुͱ͢Δิॿλε Ϋ
15. 1JYFM
    $POUSPM w ը໘ΛYͷϐΫηϧάϦουʹ෼͚ɺάϦουຖʹ2ֶशΛߦ͏
    w %VFMJOH
    /FUXPSLΛ࢖ͬͨ2ֶश ˞1JYFM
    $POUSPMͰಘΒΕͨ2஋͕BDUJPOͷબ୒ʹ࢖ΘΕΔ༁Ͱ͸ͳ͍ YͷάϦου BDUJPO਺෼ ֤άϦουͷϐΫηϧมԽྔฏۉΛใुͱͨ࣌͠ͷׂҾՃࢉใु߹ܭ2
16. 3FXBSE
    1SFEJDUJPO w &YQFSJFODF
    3FQMBZ͔Β࿈ଓͨ͠ϑϨʔϜऔΓग़ ͠ɺϑϨʔϜ໨ͷใु͕ɺਖ਼͔ෛ͔θϩ͔Λ༧ଌ ͢ΔิॿλεΫ
    w ༧ଌ͢Δใु͸ɺ
 ʴ ʔ
    PS
    ͷൺ཰͕ʹͳΔ༷ʹαϯϓϦϯά
    
 ༗ӹͳใुΠϕϯτ͸ϨΞͰ͋ͬͯ΋ɺසൟʹαϯϓϦϯά͞ΕΔ
17. 3FXBSE
    1SFEJDUJPO ࣍ͷใु͕
    PS
    
    PS
    Λ༧ଌ
18. 7BMVF
    'VODUJPO
    3FQMBZ w "$Ͱ΍͍ͬͯΔɺঢ়ଶՁ஋ 7 ͷਪఆ "DUPS$SJUJDͷ$SJUJDଆ Λɺ&YQFSJFODF
    3FQMBZ͔ΒαϯϓϦϯάͨ͠ϑϨʔϜͰ࠶౓ ߦ͏
    w 3FXBSE
    1SFEJDUJPOͱҧͬͯɺαϯϓϦϯά͸ಛʹภΒͤͳ͍
19. ิॿλεΫ͸ɺ௚઀"DUJPOબ୒ʹӨڹ͸༩͑ͳ͍͕ɺϕʔ εͷ"$ͱ$POW૚ɺ-45.૚ͷ8FJHIUΛڞ༗͍ͯ͠Δͷ ͰɺิॿλεΫΛೖΕΔ͜ͱʹΑΓɺͦΕΛղ͘ޮՌతͳ ಛ௃දݱ͕ಘΒΕΔ͜ͱʹΑΓɺؒ઀తʹ"DUJPOબ୒ʹӨ ڹΛ༩͑Δ
20. ଛࣦؔ਺ #BTF
    "$ 7BMVF
    'VODUJPO
    
    3FQMBZ 1JYFM
    $POUSPM
    YάϦου෼ 3FXBSE
    1SFEJDUJPO
22. "$ͱͷൺֱ %FFQ.JOE
    -BC؀ڥʹͯฏۉͰYഒͷߴ଎Խ
23. ΓΜ͝ΛऔΔͱ ఺ ϫʔϓ஍఺ʹ౸ୡ͢Δͱ
    ఺ΛಘͯϥϯμϜͳ
    ৔ॴʹϫʔϓ
24. ࠶ݱݕূಈը IUUQTZPVUVCFY),R#F)* ˞
    4QFBLFS
    %FDL಺Ͱද͍ࣔͯ͠Δ৔߹͸ɺ63-ϦϯΫ͕ΫϦοΫͰ͖ͳ͍ͷͰɺQEGΛμ΢ϯϩʔυͯ͠ΫϦοΫ͍ͯͩ͘͠͞
25. 1JYFM
    $POUSPM ֤άϦουͷલϑϨʔϜͱͷ
 ϐΫηϧมԽྔ ֤άϦουͷ2஋
 औͬͨ"DUJPOʹର͢Δ2஋
26. 1PMJDZ
    К ֤ΞΫγϣϯΛऔΔ֬཰
 લਐ ޙୀ ࠨӈճస ࠨӈεϥΠυ ֶश͕ਐΉͱ΄΅ͷ֬཰Ͱ֤"DUJPOΛબͿΑ͏ʹͳͬͯ͘Δ
27. 7BMVF
    'VODUJPO ݱࡏͷঢ়ଶՁ஋
 ϫʔϓ஍఺ ఺ ʹۙͮ͘ʹͭΕ্͕͍ͯͬͯ͘
28. 3FXBSE
    1SFEJDUJPO ϓϥεใु͕དྷΔͱ༧ଌ͍ͯ͠Δ
29. 4PVSDF w IUUQTHJUIVCDPNNJZPTVEBVOSFBM