Share
Embed
第32回 強化学習アーキテクチャ勉強会 状態表現学習と世界モデルの最近の研究,および深層生成モデルライブラリPixyzの紹介 #rlarch
1) 強化学習のための状態表現学習と世界モデル
強化学習問題において,「状態」は所与のものとして考えがちであるが,必ずしもエージェントの観測そのものを用いることが良いとは限らない.例えば,部分観測問題であれば,エージェントが過去の観測を何らかの形で記憶して利用することが有益であろう.そのため,効率的な強化学習のためには,エージェントの過去の観測から有益な「状態」の表現を学習するようにモデルを設計することが有望である.このような状態表現や状態遷移を学習し,エージェントの環境のモデリングを行うモデルは「世界モデル」[1]や,「内部モデル」と呼ばれており,近年,画像など高次元の入力に対応するために状態表現学習に深層生成モデルを用いる研究が数多く発表されている.これらの研究を,2018年にarXivに投稿されたレビュー論文[2]に基づきながら整理して議論する.
2) 深層生成モデルライブラリPixyzハンズオン
様々な深層生成モデルを簡潔に記述することのできる,PyTorchベースのライブラリであるPixyz[3]のハンズオンを行う(PyTorchが使用可能なラップトップがあると便利だと思います).
3) 最近の世界モデル研究紹介: GQN・TD-VAE
英DeepMind社から2018年に発表された世界モデル関連の研究である,Generative Query Network (GQN)[4] とTemoral Difference Variational Auto-Encoder (TD-VAE) [5]の2つのモデルに関して,Pixyzによる実装例を交えながら説明を行う.これらのモデルの応用やその先の展望を議論したい.
Tatsuya Matsushima
February 05, 2019
Transcript
104 slides
-
ࣗݾհ ౦ژେֶ ֶܥݚڀՊ ٕज़ܦӦઓֶུઐ߈ দඌݚڀࣨ M1 দౢ ୡ (Tatsuya Matsushima)
• ਓؒͱڞੜͰ͖ΔΑ͏ͳదԠతͳϩϘοτͷ։ൃͱɼ ͦͷΑ͏ͳϩϘοτΛ࡞Δ͜ͱͰੜ໋ੑਓؒͷೳΛ ߏతʹཧղ͢Δ͜ͱʹڵຯ͕͋Γ·͢ɽ • ࠷ۙɼܦΫϩετϨϯυ͞ΜͰهࣄΛॻ͖·ͨ͠ • দඌݚ͕ʂ AIͷʮମੑʯΛάʔάϧϑΣΠεϒοΫ͕ݚڀ https://trend.nikkeibp.co.jp/atcl/contents/technology/00007/00001/ • ϩϘοτ੍ޚʹେͳʮঢ়ଶʯදݱɹσʔλ͔ΒڥͷදݱΛֶͿ https://trend.nikkeibp.co.jp/atcl/contents/technology/00007/00015/ 2 @__tmats__ -
͓͠ͳ͕͖ ୈ1෦: 19:00-19:25 ڧԽֶशͷͨΊͷঢ়ଶදݱֶशͱੈքϞσϧ • ڧԽֶशʹ͓͚Δঢ়ଶͷදݱΛֶश͢Δํ๏Λ·ͱΊΔɽ ͜ͷදݱɼڥΛԿΒ͔ͷܗͰϞσϧԽͨ͠ʮੈքϞσϧʯͱͳ͍ͬͯΔ͜ͱ͕·͍͠ ୈ2෦: 19:30-20:00 ਂੜϞσϧϥΠϒϥϦPixyzϋϯζΦϯ
• ۙɼʮੈքϞσϧʯਂੜϞσϧΛ༻͍࣮ͯ͞ΕΔ͜ͱ͕ଟ͍ɽ ਂੜϞσϧΛ؆ܿʹॻ͚ΔϥΠϒϥϦPixyzͷνϡʔτϦΞϧΛߦ͏ɽ ୈ3෦: 20:05-20:35 ࠷ۙͷੈքϞσϧݚڀհ: GQNɾTD-VAE • 2018ʹӳDeepMind͔Βൃද͞Εͨ2ͭͷੈքϞσϧʮGQNʯͱʮTD-VAEʯΛ PixyzʹΑΔ࣮ྫΛަ͑ͳ͕Βղઆ͢Δɽ 3 -
ൃද༰ʹ͍ͭͯ (ຊൃදͰϕʔεͱ͍ͯ͠Δจ) State Representation Learning for Control: An Overview •
https://arxiv.org/abs/1802.04181 (Last revised 5 Jun 2018) • Timothée Lesort, Natalia Díaz-Rodríguez, Jean-François Goudou, David Filliat • S-RL Toolboxͱ͍͏πʔϧ࡞͍ͯ͠Δ https://github.com/araffin/robotics-rl-srl • ੍ޚλεΫʹ༻͍Δঢ়ଶͷදݱֶशʹؔ͢ΔϨϏϡʔจ • UC BerkeleyΛத৺ʹΜʹݚڀ͞Ε͍ͯΔ • ຊͰ͋Μ·Γݟͳ͍ؾ͕͢Δ • Χόʔ͞Ε͍ͯͳ͍ଞͷจຊൃදͰՃͨ͠ 5 -
ঢ়ଶදݱֶशͱʁ දݱֶश (representation learning) • σʔλ͔ΒabstructͳಛΛݟ͚ͭΔֶश ঢ়ଶදݱֶश(state representation learning, SRL)
• ঢ়ଶදݱ(state representation)ͱɼ ֶशͨ͠ಛ͕࣍ݩͰɼ࣌ؒతʹൃల͠ɼΤʔδΣϯτͷߦಈͷӨڹΛड͚Δͷ • ͜ͷΑ͏ͳදݱϩϘςΟΫε੍ޚʹ༗ӹͰ͋Δͱߟ͑ΒΕΔ cf)࣍ݩͷढ͍ • ྫ) ը૾ใඇৗʹߴ࣍ݩ͕ͩɼϩϘοτͷ੍ޚͷతؔͬͱ࣍ݩʹදݱ͞Ε͏Δ • ϚχϐϡϨʔγϣϯͷ߹ɼମͷ3࣍ݩͷҐஔใ • ੜͷ؍ଌσʔλ͔Β͜ͷঢ়ଶදݱΛݟ͚ͭΔख๏ͷݚڀ͕ओཁͳςʔϚ 6 -
ੈքϞσϧ ೳʹ͓͚ΔModel Building [Lake+ 2016]ͷॏཁੑ • ਓؒ͋ΒΏΔͷΛ֮Ͱ͖ΔΘ͚Ͱ͘ɼใ(ܹ)͔ΒੈքΛϞσϧԽͨ͠෦Ϟ σϧΛ࡞Γɼਓؒͷೳʹେ͖ͳׂΛ୲͍ͬͯΔͱࢥΘΕΔ • ੈքϞσϧͱ͍͏
• [DLྠಡձ]GQNͱؔ࿈ݚڀɼੈքϞσϧͱͷؔʹ͍ͭͯ https://www.slideshare.net/DeepLearningJP2016/dlgqn-111725780 • ࠓ·ͰͷهԱ͔ΒະདྷΛ༧ଌ͢Δྗ͕ೳ • δΣϑɾϗʔΩϯεʰߟ͑Δɾߟ͑Δίϯϐϡʔλʱ • ֶशͨ͠෦ϞσϧΛ༻͍ͯະདྷΛγϛϡϨʔγϣϯ͠ͳ͕Βߦಈ͍ͯ͠Δ ͱߟ͑ΒΕΔ 7 -
ੈքϞσϧ ೳʹ͓͚ΔModel Building [Lake+ 2016]ͷॏཁੑ • Josh TenenbaumઌੜʹΑΔMITͰͷߨٛ • MIT
AGI: Building machines that see, learn, and think like people (Josh Tenenbaum) https://www.youtube.com/watch?v=7ROelYvo8f0 8 -
ྑ͍දݱͱʁ ੜͷ؍ଌใͷແؔ࿈ͳ෦Λແࢹͯ͠ɼڧԽֶशʹར༻͢ΔͨΊʹඞཁෆՄܽͳ ใΛΤϯίʔυ͢Δ͜ͱ͕ඞཁ [Böhmer et al., 2015]ʹΑΔྑ͍ঢ়ଶදݱͷఆٛ • Ϛϧίϑੑ͕͋Δ •
ݱࡏͷঢ়ଶͷΈΛݟΔ͚ͩͰɼ͋ΔํࡦΛ༻͍ͯߦಈΛબ͢Δ͜ͱ͕Ͱ͖Δ͙Β͍ेͳ ใΛཁ͍ͯ͠Δ • ํࡦͷվળͷͨΊʹར༻Ͱ͖Δ • ಉ͡Α͏ͳಛΛ࣋ͭݟͨ͜ͱͷͳ͍ঢ়ଶʹɼֶशͨ͠ՁؔΛ൚ԽͰ͖Δ • ࣍ݩͰ͋Δ 9 -
SRLͰਅͷঢ়ଶɹɹɹΛΘͣʹɼ͜ΕΛۙࣅ͢ΔΑ͏ͳঢ়ଶɹɹɹΛֶश͢Δ • աڈͷ؍ଌɹɹ͔Βݱࡏͷঢ়ଶɹͷϚοϐϯάɹɹɹɹɹɹͷֶश SRLͷҰൠԽ 10 at ∈ ot ∈ ؍ଌ
ߦಈ at ot ot+1 ਅͷঢ়ଶ(ෆ໌) ˜ st ˜ st+1 ˜ st ∈ ˜ ใु ˜ st ∈ ˜ st ∈ o1:t st st = ϕ (o1:t) -
SRLͷΞϓϩʔν ࣗݾූ߸Խث(auto-encoder)ͷར༻ • ࠶ߏޡࠩͷ࠷খԽΛ͢Δ͜ͱͰɼΤϯίʔμɹͱσίʔμɹɹΛֶश • ͦͷࡍɼঢ়ଶɹ͕͋Δੑ࣭Λ࣋ͭΑ͏ʹ੍Λ͔͚Δ • ྫ)࣍ݩͷ੍ɼϊΠζͷআڈ(denoising)ɼεύʔεੑͷ੍ 12 st
st ϕ ϕ−1 st = ϕ (ot ; θϕ) ̂ ot = ϕ−1 (st ; θϕ−1) ࠶ߏޡࠩ Τϯίʔμ σίʔμ -
SRLͷΞϓϩʔν ॱϞσϧ(forward model)ͷར༻ • ॱϞσϧɹঢ়ଶɹͱߦಈɹΛ༻͍ͯ࣍ͷঢ়ଶɹɹΛ༧ଌ • ॱϞσϧʹઢܗมͳͲͷ੍Λ͔͚Δ͜ͱ͕Ͱ͖Δ • Τϯίʔμɹ࣍ͷঢ়ଶͷ༧ଌޡࠩΛٯͤ͞Δ͜ͱͰֶश͞ΕΔ 13
̂ st+1 = f (st , at ; θfwd) ॱϞσϧ ࣍ͷঢ়ଶͷ༧ଌޡࠩ st = ϕ (ot ; θϕ) Τϯίʔμ ϕ st at st+1 f -
SRLͷΞϓϩʔν ٯϞσϧ(inverse model)ͷར༻ • ঢ়ଶɹͱ࣍ͷঢ়ଶɹɹ͔Β࣮ࡍʹऔΒΕͨߦಈɹΛਪఆ͢Δ • Τϯίʔμɹ࣮ࡍʹͱΒΕͨߦಈɹͷ༧ଌޡࠩΛٯͤ͞Δ͜ͱͰֶश͞ΕΔ 14 st st+1
at ϕ at st = ϕ (ot ; θϕ) Τϯίʔμ ̂ at = g (st , st+1 ; θinv) ٯϞσϧ ࣮ࡍʹऔΒΕͨ ߦಈͷ༧ଌޡࠩ -
SRLͷΞϓϩʔν ࣄલࣝ(prior)ͷಋೖ • ಛఆͷ੍μΠφϛΫεʹؔ͢ΔࣄલࣝΛར༻͢Δ • ྫ) ࣌ؒతͳ࿈ଓੑ • ࣄલࣝ͋Δ݅ɹͷͱͰɼঢ়ଶͷू߹ɹɹʹద༻͞ΕΔlossΛ௨ͯ͡ఆٛ͞ΕΔ •
15 Loss = ℒprior (s1:n ; θϕ |c) s1:n c ঢ়ଶͷۭؒࣗମʹ ੍Λ͓͘ st = ϕ (ot ; θϕ) Τϯίʔμ -
ͳͥSRLΛߟ͑Δ͖ͳͷ͔? • ੜͷ؍ଌ͔Βend-to-endʹڧԽֶश͢Δͷίετ͕ߴ͍ • SRLͰྑ͍priorΛೖΕͯ͋͛Δ͜ͱ͕Ͱ͖Δ͔ • ϚϧνϞʔμϧͳ؍ଌʹ֦ு͠ಘΔ • ؔ࿈ͨ͠λεΫΛࣄલʹղ͘͜ͱͰసҠֶशʹར༻Ͱ͖Δ •
ਐԽઓུ(ES)ͳͲͷɼ࣍ݩ͕୳ࡧεϐʔυʹ݁͢ΔΑ͏ͳΞϧΰϦζϜΛ࠾༻͢Δ͜ͱ ͕ՄೳʹͳΔ Why SRL? 16 -
ֶशͷతؔ • ؍ଌͷ࠶ߏ • ॱϞσϧ(forward model)ͷֶश • ٯϞσϧ(inverse model)ͷֶश •
ಛͷఢରతֶशͷ׆༻ • ใुͷ׆༻ • ͦͷଞͷతؔ • ϋΠϒϦουͳతؔ 19 -
ֶशͷతؔ ؍ଌͷ࠶ߏ • ࣍ݩѹॖͱͯ͠Α͘ΘΕΔํ๏ • ྫ) PCA[Curran+ 2015]ɼDAEɼVAE[van Hoof+ 2016]ɽ
• ࣗݾූ߸Խث(auto-encoder)Λ͏ख๏͕ଟ͍ • ը૾ͷ؍ଌΛͦͷ··͏[Mattner+ 2012] • ΦϒδΣΫτͷҐஔΛදݱ͢ΔΑ͏ʹ੍͢Δ ྫ)Spatial Softmax [Finn+ 2015] • ؍ଌʹཱͭಛ͕ଘࡏͯ͠ͳ͍ͱ୯ʹ؍ଌΛ࠶ߏ͢Δ͚ͩͰྑ͍දݱʹͳΒͳ͍ • ྫ)ήʔϜʹ͓͚Δখ͍͞ΞΠςϜ • ҧ͏࣌ؒεςοϓ͔Β࠶ߏͨ͠Γɼ࣌ؒൃలʹ੍ؔͯ͠Λ͔͚Δ͜ͱͰରԠ 20 -
(ྫ) E2C [Watter+ 2015] Embed to Control: A Locally Linear
Latent Dynamics Model for Control from Raw Images • VAEΛ༻͍ͨॱϞσϧɽঢ়ଶ(જࡏදݱ)ɹͷભҠΛઢܗͰ͋ΔͱԾఆ. • ࣍ͷ࣌ؒεςοϓͷঢ়ଶͷ༧ଌɹɹͱͦͷঢ়ଶɹɹͷKLΛ ͚ۙͮΔ͜ͱͰॱϞσϧΛֶश • ΧϧϚϯϑΟϧλͱͯ͠ఆࣜԽͨ͠ͷ͋Δ(DVBF) [Karl+ 2016] 22 st ̂ st+1 ∼ (μ = Wst + Uat + V, σ) ઢܗ ̂ st+1 st+1 -
World Models • VAEͱMDN-RNNΛར༻ͨ͠ॱϞσϧ • Vision model (V): ߴ࣍ݩͷ؍ଌσʔλΛVAEΛ༻͍ͯ ࣍ݩͷίʔυ(ঢ়ଶ)ʹѹॖ
• Memory RNN (M): աڈͷίʔυ͔Β࣍ͷεςοϓͷ ίʔυ(ঢ়ଶ)Λ༧ଌ [DLྠಡձ]World Models https://www.slideshare.net/DeepLearningJP2016/dlworld-models-95167842 (ྫ) World Model [Ha+ 2018] 23 -
ֶशͷతؔ ٯϞσϧ(inverse model)ͷֶश • ͱͬͨߦಈΛਪఆͰ͖ΔΑ͏ʹঢ়ଶͷදݱʹ੍Λ՝͢ • ྫ) Learning to Poke
by Poking [Agrawal+ 2016] • ͍ͭͬͭͨҐஔ(ɹ)ɼ֯(ɹ)ɼڑ(ɹ)Λਪఆ 24 lt θt pt -
(ྫ) ICM [Pathak+ 2017] Curiosity-driven Exploration by Self-supervised Prediction •
ॱϞσϧͷ༧ଌޡࠩɹɹΛڧԽֶशͷతใुͱͯ͠ར༻ • ΤʔδΣϯτͷ֎෦͔Βͷใु͕εύʔεͳͱ͖ʹ୳ࡧΛଅਐ͢Δ • ٯϞσϧʹΑΔLossར༻ [DLྠಡձ]Large-Scale Study of Curiosity-Driven Learning https://www.slideshare.net/DeepLearningJP2016/dllargescale-study-of-curiositydriven-learning 25 ℒfwd ( ̂ ϕ (ot+1), ̂ f ( ̂ ϕ (ot), at)) = 1 2 ̂ f ( ̂ ϕ (ot), at) − ̂ ϕ (ot+1) 2 2 ℒfwd min θP ,θI ,θF [−λπ(st ; θP) [Σt rt] + (1 − β)ℒinv + βℒfwd] ٯϞσϧ ॱϞσϧ ֎తใु -
ֶशͷతؔ ಛͷఢରతֶश • ྫ) Causal InfoGAN [Kurutach+ 2018] • GANͷతؔʹঢ়ଶͱGeneratorͷग़ྗ(؍ଌͷϖΞ)ͷ૬ޓใྔʹؔ͢Δਖ਼ଇԽ߲ΛՃ
26 min G,Q,ℳ max D V(G, D) − λIVLB (G, Q) ૬ޓใྔ -
ֶशͷతؔ ใुͷ׆༻ • SLRʹ͓͍ͯใुΛར༻͢Δ͜ͱඞͣ͠ඞཁͰͳ͍͕ɼঢ়ଶΛ۠ผ͢ΔͨΊͷ Ճతͳใͱͯ͠ར༻͠͏Δ • ྫ) VPN [Oh+ 2017]
• ࣍ͷঢ়ଶͱͦͷঢ়ଶՁ༧ଌ 27 ߦಈ ※optionͷo ࣍ͷঢ়ଶ ࣍ͷঢ়ଶՁ ؍ଌ ঢ়ଶ -
ֶशͷతؔ ͦͷଞͷతؔ • ࣮ੈքʹؔ͢Δࣄલࣝ(prior)Λঢ়ଶۭؒʹө͢ΔͨΊʹɼతؔΛ͢Δ • ͍Ζ͍Ζͳͷ͕ఏҊ͞Ε͍ͯΔ • Slowness prior [Lesort+
2017, Jonschkowski+ 2017] • ॏཁͳͷΏͬ͘Γͱ࿈ଓతʹಈ͖ɼٸܹͳมԽ͕ى͜ΔՄೳੑ͍ • Variability [Jonschkowski+ 2017] • ؔͷ͋Δͷಈ͘ͷͰɼঢ়ଶදݱֶशಈ͍͍ͯΔͷʹ͖͢ 28 ℒSlowness (D, ϕ) = [ Δst 2 ] ℒVariabilty (D, ϕ) = [e− st1 − st2 ] -
ֶशͷతؔ ͦͷଞͷతؔ • Robotic Priors [Jonschkowski+ 2015]Ͱಋೖ͞Ε͍ͯΔͷ • Proportionality •
ҧ͏ঢ়ଶͰಉ͡ߦಈΛͨ͠߹ʹɼঢ়ଶʹٴ΅͢ӨڹಉఔͰ͋Δ • Repeatability • ࣅͨঢ়ଶͰಉ͡ߦಈΛͨ͠߹ʹɼঢ়ଶʹٴ΅͢Өڹಉఔɾಉ͡ํͰ͋Δ 29 ℒProp (D, ϕ) = [( Δst2 − Δst1 ) 2 |at1 = at2] ℒRep (D, ϕ) = [e− st2 − st1 2 Δst2 − Δst1 2 |at1 = at2] -
ֶशͷతؔ ϋΠϒϦουͳతؔ • ࣮ࡍࠓ·Ͱʹڍ͛ͨతؔͷ͏ͪɼෳΛΈ߹ΘͤͯSRL͕ߦΘΕΔ͜ͱ͕ଟ͍ 30 ߦಈ/࣍ͷঢ়ଶ ͷ੍ ॱϞσϧ ※࣍ͷঢ়ଶͷ༧ଌ ٯϞσϧ
؍ଌͷ࠶ߏ ࣍ͷ؍ଌͷ ༧ଌ ใुͷ׆༻ E2C [Watter+ 2015] ✔ ✔ ✔ ✔ World Model [Ha+ 2018] ✔ ✔ ✔ ICM [Pathak+ 2017] ✔ ✔ ✔ Causal InfoGAN [Kurutach+ 2018] ✔ ✔ ✔ ✔ VPN [Oh+ 2017] ✔ ✔ Robotic Priors [Jonschkowski+ 2015] ✔ ✔ -
؍ଌɾঢ়ଶɾߦಈۭؒͷઃܭ • ؍ଌɾঢ়ଶɾߦಈۭؒͷઃܭͷෳࡶੑʹӨڹΛٴ΅͢ • Ͳͷ͘Β͍ͷ࣍ݩͷେ͖͔͞ɼߦಈ͕ࢄ͔ɾ࿈ଓ͔ • ௨ৗɼਅͷঢ়ଶΑΓେ͖ͳঢ়ଶۭؒͷ࣍ݩΛઃܭ͢Δ͜ͱ͕ଟ͍ • ঢ়ଶΛͲͷ͙Β͍ͷ࣍ݩʹ͢Ε͍͍͔Α͘Θ͔Βͳ͍λεΫଟ͍ ྫ)Atari
31 ڥ ؍ଌͷछྨ ؍ଌۭؒͷ࣍ݩ ঢ়ଶͷ࣍ݩ ߦಈ Robotic Priors [Jon-schkowski+ 2015] slot car racing ը૾ 16×16×3 2 ࢄ(25) E2C [Watter+ 2015] cart-pole ը૾ 80×80×3 8 ࢄ ICM [Pathak+ 2017] Mario Bros. ը૾ 42×42×3 2 ࢄ(14) -
ঢ়ଶදݱͷධՁࢦඪ Ͳ͏ͬͯঢ়ଶදݱͷྑ͞ΛධՁ͢Δ͔ʁ • ΤʔδΣϯτʹ࣮ࡍʹڧԽֶशλεΫΛղ͔ͤͯɼλεΫؒͰసҠͰ͖Δ͙Β͍൚Խ͞Ε ͨදݱʹͳ͍ͬͯΔ͔ΛௐΔ • ͬͱҰൠతͳํ๏͕ͩɼ࣮ݧίετ͕ߴ͍ • ͲͷڧԽֶशΞϧΰϦζϜΛͬͯධՁ͢Ε͍͍͔Θ͔Βͳ͍ •
ͳͷͰɼֶशͨ͠ঢ়ଶදݱ͕ྑ͍͔Ͳ͏͔ͷதؒతͳධՁख๏͕ཉ͍͠ • ࠷ۙ๏Λ͏ • ࣭తධՁ • ྔతධՁ (KNN-MSE [Lesort+ 2017]) 32 KNN − MSE(s) = 1 k ∑ s′∈KNN(s,k) ˜ s − ˜ s′ 2 -
ঢ়ଶදݱͷධՁࢦඪ Ͳ͏ͬͯঢ়ଶදݱͷྑ͞ΛධՁ͢Δ͔ʁ • ͭΕͷͳ͍දݱ(disentangled)͔Ͳ͏͔ΛΈΔ • disentangled metric score [Higgins+ 2016]
• σʔλͷഎޙͷੜཁҼ͕͔͍ͬͯΔલఏ • ༰ྔ͕খ͘͞VC࣍ݩͷখ͍͞ผثͷaccuracyΛ༻͍Δํ๏ • ਅͷঢ়ଶͷճؼϞσϧΛ࡞Δ [Jonschkowski+ 2015] • ςετηοτͷਫ਼ΛධՁ͢Δ 33 -
ධՁʹ༻͍ΔλεΫ SRLͰఆ൪ͷλεΫ • ৼࢠɾཱৼࢠ • ϥϯμϜͳҐஔ͔Βελʔτ͢ΔৼࢠΛཱͯΔ • Cart-Pole • ंͷཱ͍ͭͨৼࢠΛཱͯΔ
• ਨํ͔Β15°ͣΕΔ͔த৺͔Β2.4ϢχοτͿΜͣΕͯ͠·͏ͱΤϐιʔυ͕ऴྃ͢Δ 35 -
ධՁʹ༻͍ΔλεΫ SRLͰఆ൪ͷλεΫ • ϏσΦήʔϜ • ྫ) AtariɼDoomɼSuper Mario Bros. •
ཧγϛϡϨʔλ • ྫ) OpenAI Gymɼ DeepMind Labs • ࣮ϩϘοτ • ྫ) ϚχϐϡϨʔγϣϯ[Finn+ 2015]ɼϘλϯԡ͠[Lesort+ 2015]ɼ࣋[Finn+ 2015] 36 -
S-RL Toolbox SRLΞϧΰϦζϜͷධՁʹؔ͢Δ͍Ζ͍ΖΛղܾ͢Δπʔϧ [Raffin+ 2018] • https://github.com/araffin/robotics-rl-srl • ଟ༷ͳػೳ •
10छྨͷڧԽֶशΞϧΰϦζϜ • Open AI GymܗࣜͷΠϯλʔϑΣΠεΛ࣋ͭධՁڥ • ϩΨʔɾՄࢹԽπʔϧ • ϋΠύʔύϥϝʔλαʔνπʔϧ • ࣮ػͷbaxterͰूΊͨσʔληοτ • SRLͷ࣮ूSRL-Zooͱؚͯ͠·Ε͍ͯΔ • https://github.com/araffin/srl-zoo • PyTorchͰ͏Ε͍͠ 37 -
ײ • ঢ়ଶදݱʹؔͯ͠ͲΕ͚ͩෆ࣮֬ੑ͕͋Δͷ͔ΛධՁ͢Δݚڀ͋ΔͷͩΖ͏͔ʁ • ྫ͑ɼ࠷ॳͷ1ϑϨʔϜ͚ͩݟͨͱ͖ͱɼ20ϑϨʔϜ࿈ଓͰݟͨͱ͖Ͱͦͷঢ়ଶදݱͷෆ ࣮֬ੑҟͳΔͣ • ͦͷෆ࣮֬ੑΛөͨ͠policy͕࡞ΕΕޮతͳ୳ࡧʹͭͳ͕Δʁ • ͨ͘͞ΜͷλεΫΛղ͔ͤͯSRLͯ͠ɼྑ͍SRLͷύϥϝʔλΛֶशͨ͠ͷͪɼfew-shot
Ͱ৽͍͠λεΫʹద߹ͤ͞ΔMAMLతͳΞϓϩʔν͕༗ޮ͔ • ͦͦɼSRLΛ͍ͨ͠ؾ࣋ͪɼͨ͘͞ΜͷλεΫͰڞ༗Ͱ͖ΔදݱΛֶश͍͔ͨ͠Βͩͬ ͨͷͰʁ • (·͋ɼ࣮ݧίετ͕ߴ͍ͷͰɼจͰͨ͘͞ΜͷυϝΠϯΛͬͨڧԽֶशΛͨ͘͠ͳ͍ͷΘ ͔Δ͚Ͳ…) 39 -
σΟεΧογϣϯ ੈքϞσϧͷֶशͱํࡦͷֶशͷ • ੈքϞσϧ͕ෆશͳͱ͖ʹํࡦΛͲ͏ֶश͢Δͷ͔ʁ • ϞσϧΛΞϯαϯϒϧ͢Δํ๏ දݱֶशͱ͍͏ઃఆࣗମͷ • ݁ہɼਅͷdownstreamͷλεΫ͕Θ͔Βͳ͍ͱ͖ʹɼͳΜΒ͔ͷྑ͍දݱ͕ଘࡏͯ͠ ͍Δͣͱ͍͏ԾఆΛ͓͘ɼදݱֶशͷʹߦ͖ண͘ͷͰ
• meta-priorͷ֓೦ʹ૬[Bengio+ 2013] • ͜ͷ͠͞ͷഎܠʹɼλεΫ͕ࢄతʹಘΒΕΔͱ͍͏ઃఆࣗମͷԾఆ͕͋ΔΑ͏ ͳؾ͢Δ 40 -
References [Agrawal+ 2016] Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra
Malik, Sergey Levine (2016). Learning to Poke by Poking: Experiential Learning of Intuitive Physics. https://arxiv.org/abs/1606.07419 [Bengio+ 2013] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798–1828, 2013. https://ieeexplore.ieee.org/document/6472238 [Böhmer+ 2015] Böhmer, W., Springenberg, J. T., Boedecker, J., Riedmiller, M., and Obermayer, K. (2015). Autonomous learning of state representations for control: An emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations. KI - Künstliche Intelligenz, pages 1–10. http://www.ni.tu-berlin.de/fileadmin/fg215/articles/ boehmer15b.pdf [Curran+ 2015] William Curran, Tim Brys, Matthew Taylor, William Smart (2015). Using PCA to Efficiently Represent State Spaces. https:// arxiv.org/abs/1505.00322 [Finn+ 2015] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, Pieter Abbeel (2015). Deep Spatial Autoencoders for Visuomotor Learning. https://arxiv.org/abs/1509.06113 [Ha+ 2018] David Ha, Jürgen Schmidhuber (2018). World Models. https://arxiv.org/abs/1803.10122 [Higgins+ 2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner (2016). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. https:// openreview.net/forum?id=Sy2fzU9gl [Jonschkowski+ 2015] Jonschkowski, R. and Brock, O. (2015). Learning state representations with robotic priors. Auton. Robots, 39(3): 407–428. http://www.robotics.tu-berlin.de/fileadmin/fg170/Publikationen_pdf/Jonschkowski-15-AURO.pdf [Jonschkowski+ 2017] Rico Jonschkowski, Roland Hafner, Jonathan Scholz, Martin Riedmiller (2017). PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations. https://arxiv.org/abs/1705.09805 [Karl+ 2016] Maximilian Karl, Maximilian Soelch, Justin Bayer, Patrick van der Smagt. Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data. https://arxiv.org/abs/1605.06432 42 -
References [Kurutach+ 2018] Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart
Russell, Pieter Abbeel (2018). Learning Plannable Representations with Causal InfoGAN. https://arxiv.org/abs/1807.09341 [Lake+ 2016} Building Machines That Learn and Think Like People (2016). Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman. https://arxiv.org/abs/1604.00289 [Oh+ 2017] Junhyuk Oh, Satinder Singh, Honglak Lee (2017). Value Prediction Network. https://arxiv.org/abs/1707.03497 [Pathak+ 2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell (2017). Curiosity-driven Exploration by Self- supervised Prediction. https://arxiv.org/abs/1705.05363 [Raffin+ 2018] Antonin Raffin, Ashley Hill, René Traoré, Timothée Lesort, Natalia Díaz-Rodríguez, David Filliat (2018). S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning. https://arxiv.org/abs/1809.09369 [Lesort+ 2017] Timothée Lesort, Mathieu Seurin, Xinrui Li, Natalia Díaz Rodríguez, David Filliat (2017). Unsupervised state representation learning with robotic priors: a robustness benchmark. https://arxiv.org/abs/1709.05185 [Mattner+ 2012] Mattner, J., Lange, S., and Riedmiller, M. A. (2012). Learn to swing up and balance a real pole based on raw visual input data. In Neural Information Processing - 19th International Conference, ICONIP 2012, Doha, Qatar, November 12-15, 2012, Proceedings, Part V, pages 126–133. https://ieeexplore.ieee.org/document/7759578 [van Hoof+ 2016] van Hoof, H., Chen, N., Karl, M., van der Smagt, P., and Peters, J. (2016). Stable reinforcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3928–3934. https://ieeexplore.ieee.org/document/7759578/ [Watter+ 2015] Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, Martin Riedmiller (2015). Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images. https://arxiv.org/abs/1506.07365 43 -
ਂֶशʹ͓͚ΔੜϞσϧ ਂੜϞσϧ (Deep Generative Model, DGM) • ʹχϡʔϥϧωοτϫʔΫΛ༻͍Δ • VAEͱGAN͕Α͘ΒΕ͍ͯΔ
• ͱ͘ʹɼVAEࠓ·Ͱͷ؍ଌͷܥྻͷ࣍ݩදݱ(ঢ়ଶදݱ)Λֶश͢ΔͨΊʹ Α͘༻͍ΒΕ͍ͯΔ(ୈ1෦) 48 VAE ग़య: [Tschannen+ 2018] GAN ग़య: [Tschannen+ 2018] -
VAE Variational Autoencoder (VAE) [Kingma+ 2014] • જࡏมϞσϧΛֶश͢ΔͨΊʹɼ܇࿅σʔλͷରͷ࠷େԽΛࢦ͢ • KLඇෛͳͷͰɼɹɹɹɼର
ͷԼքʹͳ͍ͬͯΔ(ELBO) • ͭ·ΓELBOͷ࠷େԽΛ͢Εྑ͍(VAEͷloss ͷ࠷খԽ) 49 ℒVAE (θ, ϕ) = ̂ p(x) [qϕ (z|x) [−log pθ (x|z)]] + ̂ p(x) [DKL (qKL (z|x)∥p(z))] ※ ܦݧσʔλɹɹͰظΛͱΔ͜ͱΛ໌ࣔతʹ͍ࣔͯͯ͠ɼݟ׳Εͳ͍͕ී௨ͷVAEͷELBO ̂ p(x) [−log pθ (x)] = ℒVAE (θ, ϕ) − ̂ p(x) [DKL (qϕ (z|x)∥pθ (z|x))] −ℒVAE ̂ p(x) [−log pθ (x)] ℒVAE ̂ p(x) ग़య: [Tschannen+ 2018] KL߲ ࠶ߏ -
VAE VAEͷloss • ୈ1߲ɼɹɹɹɹɹʹΑΔαϯϓϧΛ༻͍ɼޯreparametrization trickΛͬͯٯ • ୈ2߲ɼclosed-formʹٻΊΔ͔ɼαϯϓϧ͔Βਪఆ͢Δ • Τϯίʔμͱͯ͠,ɹɹɹɹɹɹɹɹɹɹɹɹɹɼࣄલͱͯ͠ɼ ΛબΜͩͱ͖
closed-formʹܭࢉͰ͖Δ • ͦͷ΄͔ͷͱ͖ɼؒͷڑΛαϯϓϧ͔Βਪఆ͢Δඞཁ͕͋Δ ྫ) GANʹ͓͚Δdensity ratio trick 50 ℒVAE (θ, ϕ) = ̂ p(x) [qϕ (z|x) [−log pθ (x|z)]] + ̂ p(x) [DKL (qϕ (z|x)∥p(z))] z(i) ∼ qϕ (z|x(i)) qϕ (z|x) = (μϕ (x), diag (σϕ (x))) p(z) = (0,I) KL߲ ࠶ߏ -
ఢରతֶशʹΑΔີൺਪఆ f-μΠόʔδΣϯε • ɹΛತؔͰɼ ཱ͕͢ΔͱԾఆͨ͠ͱ͖ɼ ͱ ͷf-μΠόʔδΣϯεΛ ͱఆٛ͢Δɽ
• ͷͱ͖ɼKL divergenceʹͳΔ • ɹͱɹ͔Βͷαϯϓϧ͕༩͑ΒΕͨͱ͖ɼdensity-ratio trickΛͬͯf-μΠόʔδΣϯεΛਪఆ Ͱ͖Δ • GANʹΑͬͯΒΕΔΑ͏ʹͳͬͨ 51 f f(1) = 0 px py Df (px ∥py) = ∫ f ( px (x) py (x) ) py (x)dx f(t) = t log t Df (px ∥py) = DKL (px ∥py) px py -
ఢରతֶशʹΑΔີൺਪఆ GANʹΑΔDensity-ratio TrickΛͬͨKLμΠόʔδΣϯεͷਪఆ • ɹͱɹΛϥϕϧɹɹɹɹʹΑ͚ͬͯ݅ͮΒΕͨͱͯ͠දݱ͢Δ • ͭ·Γɼɹɹɹɹɹɹɹɼ • 2ྨλεΫʹམͱ͠ࠐΈɼDiscriminator ͦͷೖྗ͕ɹɹ͔ΒಘΒΕͨͷͰ
͋Δ֬Λ༧ଌ͢Δ • ͜ͷͱ͖ɼີൺΫϥεͷ͕֬ಉͱͯ͠ɼ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹͱͳΔ • Ҏ্ΑΓɼɹ͔Βi.i.dͳɹݸͷαϯϓϧ͕ಘΒΕͨͱ͖ɼ 52 c ∈ {0,1} px py px (x) = p(x|c = 1) py (x) = p(x|c = 0) Sη px (x) px (x) py (x) = p(x|c = 1) p(x|c = 0) = p(c = 1|x) p(c = 0|x) ≈ Sη (x) 1 − Sη (x) px N DKL (px ∥py) = ∫ px (x)log ( px (x) py (x) ) dx ≈ 1 N N ∑ i=1 log ( Sη (x(i)) 1 − Sη (x(i))) ग़య: [Tschannen+ 2018] -
Pixyzͱ Pixyz • ෳࡶͳਂੜϞσϧΛ؆୯ʹ࣮ɾར༻͢Δ͜ͱʹ ಛԽͨ͠PyTorchϕʔεͷϥΠϒϥϦ • ϨϙδτϦ: https://github.com/masa-su/pixyz • υΩϡϝϯτ:
https://docs.pixyz.io • ౦େদඌݚ ླ͞Μ͕։ൃ • ڧԽֶशΞʔΩςΫνϟษڧձͷΦʔΨφΠβͷ1ਓ • ਂੜϞσϧΛهड़͢ΔϥΠϒϥϦͱͯ͠ ֬มɹɹɹͷಉ࣌ɹɹɹɹΛҙ໊͚ࣝͯ͠ΒΕ͍ͯΔ 54 x, y, z P(x, y, z) -
1. Distribution API ֬ͷAPI • DistributionΫϥεΛܧঝͯ͠ωοτϫʔΫΛఆٛ͢Δ • torch.distributions ʹؚ·ΕΔͷͱ΄΅ಉ͡ॻ͖ํ •
ಉ࣌ͷҼղΛɼͷֻ͚ࢉͱͯ͠هड़Ͱ͖Δ • ͷੵͱͯ͠ߏ͞ΕΔɼಉ༷ʹͱͯ͠αϯϓϦϯάܭࢉ͕Մೳ 56 -
2. Loss API DistributionΫϥεΛͱʹɼޡࠩؔԼքΛܭࢉ͢Δ • σʔλΛҾͱͯ͠estimateϝιουΛ͏͜ͱͰΛධՁͰ͖Δ (define-and-run) • ༷ʑͳLoss͕طʹఆٛ͞Ε͍ͯΔ •
ྫ) ෛͷର(NLL)ɼKLμΠόʔδΣϯε(KullbackLeibler)… • Lossؒͷ࢛ଇԋࢉ͕ՄೳͳͷͰɼෳࡶͳਂੜϞσϧ͕༻ҙʹهड़Ͱ͖Δ 57 − ∑ x,y∼pdata (x,y) [ Eq(z|x,y) [log p(x, z|y) q(z|x, y) ] + α log q(y|x) ] − ∑ xu ∼pdata(xu) Eq(z|xu ,y)q(y|xu ) [ log p (xu , z|y) q(z|xu , y)q(y|xu ) ] ྫ) M2Ϟσϧ[Kingma+ 2014] -
3. Model API LossoptimizerΛModelΫϥεʹͯ͠ϞσϧΛఆٛ • trainϝιουͰֶशɼtestϝιουͰධՁ ग़དྷ߹͍ͷϞσϧ༻ҙ͞Ε͍ͯΔ • ؆୯ͳϞσϧͰͬ͞͞ͱ࣮͍ͨ͠ਓ͚ •
ྫ) VAEɼGANɼมਪ(VI)ɼ࠷ਪఆ(ML) https://docs.pixyz.io/en/latest/models.html 58 -
1. Πϯετʔϧ લఏ: PyTorch͕Πϯετʔϧࡁ • ͜ͷลΛΈ͍ͯͩ͘͞ https://pytorch.org/get-started/locally/ • ;ͭ͏ɼɹɹɹɹɹɹɹɹɹɹɹͰΑ͍ͱࢥΘΕΔ 1)
PixyzͷgithubϨϙδτϦ͔Βclone 2) pip install • কདྷɼόʔδϣϯ͕҆ఆͨ͠ΒPyPIʹొ͢Δ༧ఆͩͦ͏Ͱ͢(git clone͢Δඞཁͳ͘ͳΔ) 60 git clone https://github.com/masa-su/pixyz.git pip install -e pixyz pip install torch torchvision -
2. ͬͯΈΔ PixyzʹΑΔ࣮ͷجຊతͳྲྀΕ 1. Λఆٛ͢Δ • ͷੵͱͯ͠ΈͳͤΔʂ 2. తؔɾϞσϧΛఆٛ͢Δ •
Model APIɼLoss APIɼDistribution APIͷ3ͭͷॻ͖ํ͕ଘࡏ • Lossಉ࢜ͷ࢛ଇԋࢉ͕Ͱ͖Δʂ 3. ֶश͢Δ • ModelΫϥεΛܧঝͨ͠߹ɼmodel.train()ͰOKʂ 61 -
ࠓͷνϡʔτϦΞϧࢿྉ ʮश͏ΑΓ׳ΕΑʯͱ͍͏͜ͱͰ༻ҙͯ͠Έ·ͨ͠ • https://github.com/TMats/rlarch-pixyz-tutorial • 00: PixyzͰѻ͏֬ʹ͍ͭͯ • 01: Model
APIͷVAEΫϥεΛͬͯɼvanillaͳVAE[Kingma+ 2014]Λ࣮͢Δ • 02: Loss APIΛͬͯɼΑΓෳࡶͳਂੜϞσϧΛ࣮͢Δ • M2Ϟσϧ[Kingma+ 2014] • ॳΊͯ͜ͷࢿྉΛར༻͢ΔͷͰɼࠓޙͷࢀߟͷͨΊʹɼ࣭ɾίϝϯτͳͲ͋Εͥͻ͓ ئ͍͠·͢ 62 -
Pixyzͷ͏Ε͍͠ͱ͜Ζ define-by-runͱdefine-and-runͷ͍͍ͱ͜ͲΓΛ͍ͯ͠Δ • ʮωοτϫʔΫPyTorchͷΑ͏ʹॊೈʹධՁ͍͚ͨ͠ΕͲɼLossઌʹܾΊ͓͍ͯͯॻ ͍ͨͷ͕ਖ਼͍͔͠Ͳ͏͔ࣜΛݟͯ֬ೝ͍ͨ͠ʯͱ͍͏ؾ࣋ͪʹԠ͑ͯ͘ΕΔϥΠϒϥϦ • ωοτϫʔΫͱ֬Λॻ͘ϨΠϠΛΓ͔ͨ͠ΒͰ͖ٕͨ • ݁Ռͱͯ͠ɼจͷॻ͔ΕͨLossͷࣜΛͦͷ··ࣸ͠औΔײ͡Ͱ࣮Ͱ͖Δ •
࣮ݧ͢Δͱ͖ɼωοτϫʔΫͷͱLossͷΛΓ࣮ͯ͠ݧͰ͖Δ Loss APIʹΑͬͯɼҟͳΔਂੜϞσϧΛಉҰͷϑϨʔϜϫʔΫͰࠞͥͯॻ͚Δ • ྫ) GANͱVAEͷLossͷ͕औΕΔ 63 -
Pixyzoo Pixyzoo • PixyzΛར༻ͨ͠ਂੜϞσϧͷ࣮ϨϙδτϦ • ͪΖΜGan ZooΈ͍ͨͳͷΛҙ͍ࣝͯ͠Δ • https://github.com/masa-su/pixyzoo •
ݱࡏɼGQNɾVIBɾFactorVAEͳͲ͕ೖ͍ͬͯΔ • ଓʑՃ͍ͨ͠ • ϓϧϦΫେܴͰ͢ • pixyzooϨϙδτϦΛforkͯ͠ϓϧϦΫΛૹ͍ͬͯͩ͘͞ 64 -
References [Kingma+ 2014] Diederik P. Kingma, Danilo J. Rezende, Shakir
Mohamed, Max Welling. Semi-Supervised Learning with Deep Generative Models. https://arxiv.org/abs/1406.5298 [Tschannen+ 2018] Michael Tschannen, Olivier Bachem, Mario Lucic (2018). Recent Advances in Autoencoder-Based Representation Learning. https://arxiv.org/abs/1812.05069 66 -
GQNͱʁ /FVSBMTDFOFSFQSFTFOUBUJPOBOESFOEFSJOH<&TMBNJ > • 4."MJ&TMBNJ %BOJMP+3F[FOEF FUBM 4DJFODF
• ͪͳΈʹ4DJFODFຊࢽͷهࣄ্࣮શ͘ࢀߟʹͳΒͳ͍4VQQMFNFOUBMΛಡΈ·͠ΐ͏ • ෳͷࢹʹ͓͚Δը૾Λͱʹɼผͷࢹ͔Βͷը૾Λੜ͢Δ(FOFSBUJWF2VFSZ /FUXPSL (2/ ΛఏҊ IUUQTXXXZPVUVCFDPNXBUDI UJNF@DPOUJOVFW3#+'OH/2P • ͠ɼࢹͷҐஔʹΑΒͳ͍ঢ়ଶͷදݱ͕֫ಘͰ͖ΔͳΒ͏Ε͍͠ ண͍ͯ͠Δཧ༝ • ڊେͳDPOEJUJPOBM7"&Λར༻ 70 -
GQNͷৄ͍͠ࢿྉ <%-ྠಡձ>(2/ͱؔ࿈ݚڀɼੈքϞσϧͱͷؔʹ͍ͭͯ • ౦େদඌݚླ͞Μ IUUQTXXXTMJEFTIBSFOFU%FFQ-FBSOJOH+1EMHRO /FVSBMTDFOFSFQSFTFOUBUJPOBOESFOEFSJOHͷղઆ • ౦େ૬ᖒݚ.ۚࢠ͞Μ IUUQTXXXTMJEFTIBSFOFU.BTBZB,BOFLPOFVSBMTDFOFSFQSFTFOUBUJPOBOESFOEFSJOHE <%-)BDLT>1Z5PSDI
1JYZ[ʹΑΔ(FOFSBUJWF2VFSZ/FUXPSLͷ࣮ • ౦େদඌݚ#୩ޱ͞Μ IUUQTXXXTMJEFTIBSFOFU%FFQ-FBSOJOH+1EMIBDLTQZUPSDIQJYZ[HFOFSBUJWFRVFSZ OFUXPSL 72 -
GQNͷઃఆ σʔληοτ • ɹݸͷγʔϯ ڥ ͦΕͧΕʹର͠ɼɹݸͷ࠲ඪͱͦͷ࠲ඪ͔Βͷ 3(#ը૾͔ΒͳΔର • ɹݸͷγʔϯͷɹݸͷ3(#ը૾ •
ɹݸͷγʔϯͷɹݸͷࢹ WJFXQPJOU ઃఆ • ɹݸͷ؍ଌʢจ຺ʣɹɹɹɹɹɹͱҙͷࢹʢΫΤϦʣ͕༩͑ΒΕͨͱͰ ରԠ͢Δ3(#ը૾ɹɹΛ༧ଌ͢Δɽ • ༗ݶͷ࣍ݩతͳ ը૾ͷ ؍ଌ͔Βɼܾఆతʹ༧ଌ͢Δ͜ͱͰ͖ͳ͍ • จ຺Ͱ͚݅ͮͨ֬Ϟσϧ ਂੜϞσϧ ͱͯ͠ղ͘ 73 {(xk i , vk i )} (i ∈ {1,…, N}, k ∈ {1,…, K}) N K vk i xk i i k i k M x1,…,M i , v1,…,M i vq i xq i -
લఏ: Conditional VAE Conditional VAE [Sohn+ 2015] • VAEʹҙͷใɹΛ͚݅ͮͨ(conditioned)Ϟσϧ •
ࣄલΛ ͱͯ͠ϞσϧԽ͢Δ͜ͱͰɼςετ࣌ʹɹΛਪͰ͖Δ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹมԼք:ELBO (ෛͷLoss) • ࣄલͱͯ͠ɹʹґଘ͠ͳ͍ɹɹΛ͏όʔδϣϯ͋Δ[Kingma+ 2014] • PixyzνϡʔτϦΞϧʹొͨ͠M2Ϟσϧ͜ͷύλʔϯ 74 !(#|%, ') ' % # )(%|#, ') )(#|') y log $ %|' ≥ ) * + %, ' log $ % +, ' $(+|') /(+|%, ') = ) * + %, ' log $(%|+, ') − 23[/(+|%, ')||$(+|')] p(z|y) z Γ͍ͨ͜ͱରͷ࠷େԽˠELBOͷ࠷େԽ y p(z) -
Generative Query Network GQNͷϞσϧ • จ຺ɹɹɹɹɹɹɹɹɹͱΫΤϦɹɹͰ͚݅ͮͨConditional VAE • ɹܾఆతͳม(දݱωοτϫʔΫ) άϥϑΟΧϧϞσϧͱͯ͠ͷղऍɹɹɹɹมԼք:ELBO
(ෛͷLoss) 75 r = f(x1,…,M i , v1,…,M i ) vq i f !" # $ # !", &", ' ( !" #, &", ' )(#|&", ') ' &" จ຺ ΫΤϦ જࡏม log $ %&|(&, * ≥ , & - %&, (&, * log . %& -, (&, * /(-|(&, *) 2 - %&, (&, * = , & - %&, (&, * log . %& -, (&, * − 56[2 - %&, (&, * ||/(-|(&, *)] ࣄલ Τϯίʔμ σίʔμ Γ͍ͨ͜ͱରͷ࠷େԽˠELBOͷ࠷େԽ -
Generative Query Network มԼք (ෛͷଛࣦLoss) 76 ! " # $",
&", ' log + $" #, &", ' − -.[0 # $", &", ' ||2(#|&", ')] KL߲ ࠶ߏ Τϯίʔμͱࣄલ͕ۙͮ͘Α͏ʹֶश →ςετ࣌ʹࣄલΛ͑ ɹΫΤϦʹର͢Δਅͷը૾ɹ͕ͳͯ͘ ɹจ຺ɹͱΫΤϦɹ͔ΒରԠ͢Δ ɹજࡏมɹΛਪͰ͖ΔΑ͏ʹͳΔͣʂ Τϯίʔμɼจ຺ɹͱΫΤϦɹɼ ରԠ͢Δը૾ɹ͔ΒɼજࡏมɹΛਪɽ જࡏมɹ͔ΒɼΫΤϦʹରԠ͢Δը૾ɹ͕ ࠶ߏ͞ΕΔΑ͏ʹֶश →જࡏมɹͦͷγʔϯશମΛද͢Α͏ͳ ɹԿΒ͔ͷදݱֶ͕श͞ΕΔͣʂ xq vq r z r vq xq z z xq z -
ΞʔΩςΫνϟత: දݱωοτϫʔΫ දݱωοτϫʔΫΛ༻͍ͯɼɹݸͷ؍ଌɹɹɹɹɹɹΛ1ͭͷจ຺ɹʹཁ͢Δ • ը૾ͱ࠲ඪͷϖΞʹରͯͦ͠ΕͧΕมͨ͠ͷʹؔͯ͠ฏۉΛͱΔ ֤ࢹʹ͓͚ΔฏۉΛͱΔ͜ͱͰɼ ࢹͷॱ൪ʹґଘ͠ͳ͍(permutation invariant)දݱΛಘΔ • จ຺ʹ༻͍ΔࢹͷΛࣗ༝ʹઃఆͰ͖Δ
• 3//ͰϞσϧԽ͢Δͱॱং͕ؔ͢Δ (ฏۉ)Λऔ͍͍ͬͯͷʁͱ͍͏ٙ • ࠷ۙɼҰԠٞ͞Ε͍ͯΔΒ͍͠[Wagstaff+ 2019] 77 x1,…,M i , v1,…,M i M r rk = ψ (xk, vk) r = M ∑ k=1 rk -
ΞʔΩςΫνϟత: DRAWͷར༻ DRAW [Gregor+ 2015] • VAEͷ͓͚ΔજࡏมɹͷਪΛRNNΛ༻͍ͯෳճʹ͚ͯɼ ࣗݾճؼతʹߦ͏͜ͱͰɼϞσϧͷදݱྗΛߴΊΔ • ͜ͷͱ͖ͷELBO୩ޱ͞ΜͷࢿྉͰಋग़͞Ε͍ͯΔ(p11-13)
https://www.slideshare.net/DeepLearningJP2016/dlhackspytorch-pixyzgenerative-query- network-126329901 • ݁ͱͯ͠ɼ࠶ߏͷͱ֤εςοϓͷKLͷʹͳΔ • ࣄલͱΤϯίʔμͷ྆ํʹར༻ 78 z q(z|x) = L ∏ l=1 ql (zl |x, z<l) πθ (z|vq, r) = L ∏ l=1 πθl (zl |vq, r, z<l) qϕ (z|xq, vq, r) = L ∏ l=1 qϕl (zl |xq, vq, r, z<l) ࣄલ Τϯίʔμ -
จதͷ࣮ݧ݁Ռ Roomσʔληοτ • ϥϯμϜͳ࢛͍֯෦ʹϥϯμϜͳʢ1~3ʣͷ༷ʑͳମΛஔ • นͷςΫενϟ: 5छྨ চͷςΫενϟ: 3छྨ ମͷܗঢ়:
7छྨ • αΠζɼҐஔɼ৭ϥϯμϜɽϥΠτϥϯμϜ • 2ສछྨͷγʔϯΛϨϯμϦϯά • σʔληοτ(͚ͩ)ެ։͞Ε͍ͯΔ • Roomͷଞʹछྨͷσʔληοτ͕ଘࡏ https://github.com/deepmind/gqn-datasets • ৽͍͠ࢹͰͷը૾͕༧ଌͰ͖͍ͯΔ͜ͱ͕ ఆੑతʹΘ͔Δ(ӈਤ) 79 -
จதͷ࣮ݧ݁Ռ Scene Algebra • ֶशͨ͠ωοτϫʔΫΛ༻͍ͯɼจ຺ ্Ͱͷ͠ࢉɾҾ͖ࢉΛߦ͏ • word2vecͷΑ͏ʹ༧ଌ݁Ռ͕ԋࢉ௨Γʹͳ͓ͬͯΓɼߏతͳදݱʹͳ͍ͬͯΔ • Ͳ͜·Ͱߏతͳͷ͔͕ٙΔ͕…
• σʔληοτશମͰมԽͷόϦΤʔγϣϯ͕ൺֱత୯७ͳͷͰͰ͖͍ͯΔͷs͔ 80 r -
PixyzʹΑΔ࣮ Pixyzooͷதʹଘࡏ https://github.com/masa-su/pixyzoo/tree/master/GQN • ౦େদඌݚB4 ୩ޱ͞ΜʹΑΔ࣮ • Eslami͞Μ(1st Author)͔ΒϋΠύϥ •
DeepMindجຊతʹ࣮Λެ։͍ͯ͠ͳ͍ͷͰɼ ͓ͦΒ͘࠷࣮ͳ࣮ͳͣ • จͰɼK80(24GB)4ຕར༻͍ͯ͠Δͱͷ͜ͱ • खݩͰ֬ೝͯ͠ɼTitanX(12GB)4ຕʹΪϦΪϦΔ͙Β͍ • ύϥϝʔλݮΒͯͦ͠ΜͳʹӨڹͳ͍ • චऀ(Eslami͞ΜɼRezende͞Μ)ʹհͯ͠Β͍·ͨ͠ • PixyzͷDeepMindσϏϡʔ(?) 81 https://twitter.com/arkitus/status/1072845916850274304 -
PixyzʹΑΔ࣮ ݱࡏɼDRAWͷ෦ͰforϧʔϓͷதͰ1εςοϓ͝ͱʹlossΛධՁ͍ͯ͠Δ Q. ͬͱ៉ྷʹ͔͚ͳ͍ͷʁ A. ࣍ͷόʔδϣϯ(0.0.5)ͰࣗݾճؼϞσϧʹରԠ͢Δ༧ఆ • ۙதʹmasterʹϚʔδ༧ఆͩͦ͏Ͱ͢ • ͔ͳΓ؆ܿʹͳΔͣ(࣍ʹհ͢ΔTD-VAEͰར༻͍ͯ͠Δ)
ͱ͍͑ɼ ωοτϫʔΫͱ֬ͷ࣮͕͞Ε͓ͯΓɼPixyzͷྑ͕͞ੜ͖͍ͯΔ 84 -
σΟεΧογϣϯ ݁ہજࡏදݱͱͯ͠Կ͕֫ಘ͞Ε͍ͯΔͷ͔ʁ(ঢ়ଶදݱֶशత؍) • ମͷදݱɼγʔϯͦͷͷͷදݱɼࢹؒͷؔͱ؍ଌͷؔ • ͜ΕΒΛͲ͏ͬͯऔΓग़ͯ͠ར༻͢Δͷ͔ʁ • ࣮ੈքʹసҠͤ͞ΔͱͲ͏ͳΔ͔ʁ • ݱ࣮ੈքͰࡱӨͨ͠ը૾ΛͬͯGQNΛֶशͤ͞ΔϓϩδΣΫτ
https://github.com/brettgohre/still_life_rendering_gqn Ͳ͏ͬͯΤʔδΣϯτͷߦಈͷܾఆ(ڧԽֶश)ʹͬͯΏ͔͘ʁ • ͲΜͳΞϓϦέʔγϣϯ͕͋ΓಘΔ͔ʁ ϝλֶशͷจ຺ • λεΫΛͲ͏ఆٛ͢Δͷ͔ɼԿͰ͚݅ͮΔͷ͔͕ʹͳ͖͍ͬͯͯΔ 85 -
TD-VAEͱʁ 5FNQPSBM%JGGFSFODF7BSJBUJPOBM"VUP&ODPEFS<(SFHPS > • ,BSPM(SFHPS (FPSHF1BQBNBLBSJPT FUBM *$-30SBM
• ܥྻΛѻ͏ਂੜϞσϧΛఏҊͨ͠ • ܥྻΛѻ͏ਂੜϞσϧͰɼεςοϓ͝ͱʹਪΛߦ͏͜ͱ͕ओྲྀͰ͋ͬͨ ୈ෦ͷॱϞσϧ ͕ɼ5%7"&ҙͷεςοϓ·Ͱδϟϯϓͯ͠ਪͰ͖Δ • ͜ΕΛͬͯ࣌ܥྻͷநԽʹऔΓΉ͜ͱ͕Ͱ͖ͳ͍ͩΖ͏͔ ண͍ͯ͠Δཧ༝ • 3//Λ༻͍ͨʮ৴೦ঢ়ଶʯͷಋೖͱ&-#0ͷ ղͷํ͕ΧΪ 87 -
ͲΜͳঢ়ଶදݱ͕·͍͔͠ʁ จதͰݴٴ͞Ε͍ͯΔɼΤʔδΣϯτͷঢ়ଶදݱ͕͖࣋ͭੑ࣭ σʔλͷநతͳঢ়ଶදݱΛֶश͠ɼ؍ଌͰͳ͘ঢ়ଶͷϨϕϧͰ༧ଌͰ͖Δ͜ͱ ͋Δ࣌ؒ·Ͱͷશͯͷ؍ଌ͕༩͑ΒΕͨͱͰɼ ঢ়ଶͷϑΟϧλϦϯάΛܾఆతʹίʔυͨ͠৴೦ঢ়ଶ CFMJFGTUBUF ΛֶशͰ͖Δ͜ͱ • ৴೦ঢ়ଶΤʔδΣϯτ͕࣋ͭੈքʹؔ͢Δঢ়ଶͷશͯͷใͱɼ࠷దʹߦಈ͢Δํ๏ΛؚΜ Ͱ͍Δ
εςοϓઌͷδϟϯϓͨ͠ະདྷΛ༧ଌ͢Δ͜ͱɽ ࣌ܥྻશͯΛޡࠩٯͤͣʹ࣌ؒతʹΕ͔ͨ࣌ΒֶशͰ͖ΔΑ͏ʹ͢Δ͜ͱʹΑͬ ͯɼ࣌ܥྻతͳநԽΛߦΘΕΔ͜ͱ ͜ΕΒͷੑ࣭Λຬͨ͢Ϟσϧͱͯ͠ɼ5%7"&ΛఏҊ 89 -
લఏ: ࣗݾճؼϞσϧ ࣗݾճؼϞσϧ (Autoregressive Model) • ܥྻσʔλɹɹɹɹɹɹɹΛϞσϦϯά͢Δํ๏ • νΣʔϯϧʔϧΛ༻͍ͯɼΛ͖݅ͷੵʹղ (ࣜ྆ลରΛͱͬͨ)
• RNNΛ༻͍࣮ͯͰ͖Δ • • ؍ଌۭؒͰ͔͠༧ଌ͠ͳ͍ͨΊɼσʔλͷѹॖͨ͠දݱΛֶश͠ͳ͍ • ֤εςοϓͰσίʔυɾΤϯίʔυΛ͢ΔͨΊܭࢉྔ͕େ͖͍ • ܇࿅࣌ʹ࣍ͷεςοϓͷσʔλ͕ೖͬͯ͘Δ͕(ڭࢣڧ੍)ɼςετ࣌ʹࣗͷ༧ଌΛೖྗ ͢ΔͨΊෆ҆ఆ 90 x = (x1 , …, xT) log p (x1 , …, xT) = ∑ t log p (xt |x1 , …, xt−1) ht = f (ht−1 , xt) -
લఏ: ঢ়ଶۭؒϞσϧ ঢ়ଶۭؒϞσϧ (State-space Model) • ܥྻσʔλɹɹɹɹɹɹ Λજࡏม(ঢ়ଶ) ɹ Λ༻͍ͯϞσϦϯά͢Δํ๏
• ɹͱɹͷಉ࣌: • Τϯίʔμ: • ɹͷೖྗͱͯ͠ɹ·ͰͷܥྻɹɹɹɹɹΛ༻͍Δ߹ɿϑΟϧλϦϯά ɹɹɹɹɹɹɹܥྻશମɹΛ༻͍Δ߹ɿεϜʔδϯά • มԼք:ELBO (ෛͷLoss) • ঢ়ଶؒͰͷભҠΛϞσϧԽ͢Δ • ςετ࣌ʹ֤εςοϓͰͷσίʔυɾΤϯίʔυ͕ඞཁͳ͍ 91 x = (x1 , …, xT) z = (z1 , …, zT) x z p(x, z) = ∏ t p (zt |zt−1) p (xt |zt) q(z|x) = ∏ t q (zt |zt−1 , ϕt (x)) log p(x) ≥ z∼q(z|x) [∑ t log p (xt |zt) + log p (zt |zt−1) − log q (zt |zt−1 , ϕt (x)) ] σίʔμ ঢ়ଶભҠ ϕt t x (x1 , …, xt) !"#$ %"#$ !" %" -
ϑΟϧλϦϯάͷಋೖ ঢ়ଶۭؒϞσϧͰɼঢ়ଶɹΛಘΔͨΊʹલͷεςοϓͷঢ়ଶɹ ͕ඞཁ • ͦͷͨΊʹ࣍ʑʹɹɹɹɹɹɹɹͷϦαϯϓϦϯά͕ඞཁ ϑΟϧλϦϯάɹɹɹɹɹɹΛಋೖ • ؍ଌͷܥྻɹɹɹɹͷΈʹґଘ͢ΔΑ͏ʹ͢Δ • POMDPͷڧԽֶशʹ͓͚Δ৴೦ঢ়ଶʹ૬
92 zt zt−1 !"#$ %"#$ !" %" zt−1 , zt−2 , …, z1 p(zt |x1 , …, xt ) (x1 , …, xt) !"#$ !" %"#$ %" & %" !$ , . . , !" -
ϑΟϧλϦϯάͷಋೖ ϑΟϧλϦϯάɹɹɹɹɹɹΛಋೖͯ͠ELBOΛಋग़ • ϑΟϧλϦϯάʹΑͬͯɼજࡏม ͷ2͚ͭͩͰදݱͰ͖Δ 93 log(x) = ∑ t
log p(xt |xx<t ) = ∑ t log ∫ p(xt |zt )p(zt |xx<t )dzt ≥ ∑ t q(zt ,zt−1 |x≤t ) [ log p(xt |zt )p(zt |x<t ) q(zt , zt−1 |x≤t ) ] = ∑ t q(zt |x≤t )q(zt−1 |zt ,x≤t ) [log p (xt |zt) + log p (zt−1 |x<t) + log p (zt |zt−1) −log q (zt |x≤t) − log q (zt−1 |zt , x≤t)] p(zt |x1 , …, xt ) (zt−1 , zt ) ঢ়ଶભҠ ϑΟϧλϦϯά ϑΟϧλϦϯά Τϯίʔμ σίʔμ JensenͷෆࣜΑΓ !"#$ !" %"#$ %" Τϯίʔμաڈʹ ͔͏ਪʹͳ͍ͬͯΔ -
ϑΟϧλϦϯάͷ࣮ TD-VAEͰϑΟϧλϦϯάΛRNNΛ༻͍࣮͍ͯͯ͠Δ • ৴೦ঢ়ଶΛද͢มΛɹɹͱͯ͠ɼ֤εςοϓͷ৴೦ঢ়ଶΛɹɹɹɹɹɹͱϞσϧԽ • ৴೦ঢ়ଶɹաڈͷ؍ଌͷܥྻ ͷใΛؚΜͰ͍Δͱߟ͑ΒΕΔ • ͜ͷͱ͖ɼมԼք:ELBO (ෛͷLoss)
94 bt bt = f (bt−1 , xt) bt (x1 , …, xt) pB(zt |bt)q(zt−1 |zt , bt−1 , bt) [log p (xt |zt) + log pB (zt−1 |bt−1) + log p (zt |zt−1) −log pB (zt |bt) − log q (zt−1 |zt , bt−1 , bt)] ঢ়ଶભҠ ϑΟϧλϦϯά ϑΟϧλϦϯά Τϯίʔμ σίʔμ !"#$ %"#$ !" %" &"#$ &" -
࣌ؒεςοϓͷδϟϯϓ ࠓ·ͰͷٞΛ1εςοϓͷભҠ͔ΒɼεςοϓͷભҠʹ֦ு͢Δ • දه͕มΘΔ͚ͩɼมԼք:ELBO(ෛͷLoss) • ֶश࣌ɼδϟϯϓ͢Δεςοϓ ΛɹɹɹͷൣғͰαϯϓϦϯάֶͯ͠श • ঢ়ଶભҠͷೖྗʹɹΛՃ͑Δ •
ςετ࣌ɼ ɹˠϑΟϧλϦϯά→ →ঢ়ଶભҠ→ →σίʔμ→ ͱͯ͠༧ଌ͕Ͱ͖Δ 95 pB (zt2 |bt2 )q(zt1 |zt2 ,bt1 ,bt2 ) [log p (xt2 |zt2 ) + log pB (zt1 |bt1 ) + log p (zt2 |zt1 ) −log pB (zt2 |bt2 ) − log q (zt1 |zt2 , bt1 , bt2 )] ঢ়ଶભҠ ϑΟϧλϦϯά ϑΟϧλϦϯά Τϯίʔμ σίʔμ xt1 zt1 zt2 ̂ xt2 δ = t2 − t1 [1,D] δ p(zt2 |z t 1 , δ) -
TD-VAEͷֶश TD-VAEͷมԼք: ELBO (ෛͷLoss) • ᶃϑΟϧλϦϯάɹɹɹ ͔ΒɹΛαϯϓϧ • ᶄͦΕΛͬͯɼΤϯίʔμɹɹɹɹ ͔ΒɹɹΛαϯϓϧ
• ୈ2߲ͱୈ4߲ɼΤϯίʔμͱϑΟϧλϦϯάͷKLμΠόʔδΣϯεʹͳΔ 96 zt2 ∼pB (zt2 |bt2 ),zt1 ∼q(zt1 |zt2 ,bt1 ,bt2 ) [log p (xt2 |zt2 ) + log pB (zt1 |bt1 ) + log p (zt2 |zt1 ) −log pB (zt2 |bt2 ) − log q (zt1 |zt2 , bt1 , bt2 )] ᶃϑΟϧλϦϯά͔Β ɹαϯϓϧ ᶄΤϯίʔμ͔Β ɹαϯϓϧ pB (zt2 |bt2 ) zt2 zt1 q(zt1 |zt2 , bt1 , bt2 ) KL [q(zt1 |zt2 , bt1 , bt2 )||pB (zt1 |bt1 )] -
PixyzʹΑΔ࣮ Pixyzooͷதʹଘࡏ https://github.com/masa-su/pixyzoo/tree/master/TD-VAE • ModelΫϥεΛܧঝ • ࣗݾճؼ༻ͷ IterativeLoss Λ༻͍͍ͯΔ •
Pixyz v0.0.5Ҏ্͕ඞཁ • ΤϯίʔμपΓͷLoss͕ ਧͬඈͿࣗମ͕ى͖͍ͯΔͷͰ ϋΠύϥνϡʔχϯά͕ඞཁ͔ʁ • GQNͷͱ͖େมͩͬͨ 97 Loss -
จதͷ࣮ݧ ෦؍ଌMiniPacman • 1εςοϓ͝ͱʹ༧ଌ͢ΔϞσϧͰɼఏҊख๏ͷΤϯίʔμͱଞͷΤϯίʔμΛൺֱ ɹɹTD-VAEͷΤϯίʔμ ɹɹfilteringϞσϧͷΤϯίʔμ ɹɹmean-fieldϞσϧͷΤϯίʔμ • ELBOͱෛͷରʹؔͯ͠TD-VAEͷΤϯίʔμ͕ྑ͍͜ͱΛࣔ͢ 98
q (zt−1 |zt , bt−1 , bt) q (zt |zt−1 , bt) q (zt |bt) !"#$ %"#$ !" %" &"#$ &" -
จதͷ࣮ݧ MovingMNIST • ࣈ͕ࠨӈʹಈ͘MNISTͰɼεςοϓΛεςοϓΛඈͨ͠༧ଌΛͤ͞Δ࣮ݧ • 1͔Β4εςοϓͷؒͰඈֶͯ͠श • ͳΜ͔ͬͯΔMovingMNIST͡Όͳ͍Α͏ͳؾ͕͢Δ….. • ଞʹDeepMind
LabΛ༻͍࣮ͨݧΛ͍ͯ͠Δ • ΞʔΩςΫνϟʹConvDRAW[Gregor+ 2016]Λͬͨͱͷ͜ͱ 99 -
σΟεΧογϣϯ TDͷҙຯ͢Δͱ͜Ζ • Τϯίʔμ͕աڈͷਪʹͳ͍ͬͯΔ෦͕Temporal DIfferenceͬΆ͍ • ҰԠɼ4.3અʹهड़͋Δ ࣌ؒํͷநԽ • ࣌ܥྻͷநԽڧԽֶशʹͱͬͯେ͖ͳ՝
• TD-VAEͰɼతʹߦಈΛѻ͍ͬͯͳ͍ • τϧΫϨϕϧͷ੍ޚͷ࣌ܥྻ͔Βɼ࣌ܥྻతʹநԽ͞ΕͨߦಈϓϦϛςΟϒ͕࡞ΒΕͦͷϓϦ ϛςΟϒ্Ͱ୳ࡧͰ͖Δͱ୳ࡧޮతʹྑͦ͞͏ͩࣗ͠વͳؾ͢Δ TD-VAEͷRNNʹશͯΛୗ͍ͯ͠Δײ • RNNͷදݱྗͷ 100 -
References [Eslami+ 2018] Eslami, S. M. Ali, Danilo Jimenez Rezende,
Frédéric Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil C. Rabinowitz, Helen King, Chloe Hillier, Matthew M Botvinick, Daan Wierstra, Koray Kavukcuoglu and Demis Hassabis. “Neural scene representation and rendering.” Science 360 (2018): 1204-1210. http:// science.sciencemag.org/content/360/6394/1204 [Gregor+ 2015] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra. DRAW: A Recurrent Neural Network For Image Generation. https://arxiv.org/abs/1502.04623 [Gregor+ 2016] Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, Daan Wierstra. Towards Conceptual Compression. https://arxiv.org/abs/1604.08772 [Gregor+ 2019] Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, Theophane Weber. Temporal Difference Variational Auto-Encoder. https://openreview.net/forum?id=S1x4ghC9tQ [Kingma+ 2014] Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling. Semi-Supervised Learning with Deep Generative Models. https://arxiv.org/abs/1406.5298 [Sohn+ 2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (NIPS), pp. 3483–3491, 2015. https://papers.nips.cc/ paper/5775-learning-structured-output-representation-using-deep-conditional-generative-models [Tschannen+ 2018] Michael Tschannen, Olivier Bachem, Mario Lucic (2018). Recent Advances in Autoencoder-Based Representation Learning. https://arxiv.org/abs/1812.05069 [Wagstaff+ 2019] Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, Michael Osborne. On the Limitations of Representing Functions on Sets. https://arxiv.org/abs/1901.09006 102