Autoencoding Variational Inference for Topic Modelsの解説スライド

ICLR2017読み会のスライド https://connpass.com/event/57631/

1. Autoencoding Variational Inference For Topic Models Akash Srivastava and Charles Sutton ICLR2017ಡΈձ ಡΉਓ: @nzw0301
2. ֓ཁ 1. Latent Dirichlet Allocation (LDA) ΛNeural Variational Inference (NVI) Ͱ • Dirichlet ෼෍ͷ reparameterization trick 2. ৽ϞσϧͷఏҊ 3. ѱ͍ہॴղʹϋϚΔͷΛ༧๷ 2
3. ࣄલ஌ࣝɿLDAͱVAEͷ֓ཁ 3
4. LDA จॻͷ֬཰తੜ੒Ϟσϧ [Blei et al., 2003]      จॻͷτϐοΫ෼෍
   
   Q [cВ ݚڀ
   
   
   
   
   
   ՝୊
   
   
   
   
   
   ஌ࣝ
   
   
   
   
   
   Պֶऀ
   
   
   ʜ ػցֶश
   
   ਓ޻஌ೳ
   
   Ϟσϧ
   
   
   
   
   αϯϓϧ
   
   ʜ τϐοΫͷ୯ޠ෼෍ p(w|β) Ќ Ќ ػցֶश
   ػցֶश
   ݚڀ
   ਓ޻஌ೳ
   ՝୊
   
   Ϟσϧ
   -%"
   Պֶ
   ਺ֶ
   ण࢘ ίʔύε 4
5. VAE: Encoder • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ • ֬཰෼෍͸જࡏม਺Λੜ੒ • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 5
6. VAE: Decoder • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ • ֬཰෼෍͸જࡏม਺Λੜ੒ • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 6
7. VAE: Reparameterization trick • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ • ֬཰෼෍͸જࡏม਺Λੜ੒ • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 7
8. VAE: ϩεؔ਺ 8 L (⇥) = D X d=1 ( 1 2 ⇣ tr (⌃0) + µT 0 µ0 K log | ⌃0 | ⌘ + E ✏⇠N (0,1) ⇣ log p xd |f ( µ0 + ⌃ 1/2 0 ✏ ) ⌘ ) (Ⅰ) ࣄલ෼෍ͱͷKLμΠόʔδΣϯε (Ⅱ) ର਺໬౓ ࣜશମ: Evidence Lower Bound (I) (Ⅱ)
9. ຊ୊ 9
10. Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet෼෍͔Βαϯϓϧ • Scale family DistributionͰͳ͍ͨΊɼߏ੒Ͱ͖ͳ͍ 10      จॻͷτϐοΫ෼෍
    
    Q [cВ
11. Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet෼෍͔Βαϯϓϧ • Scale family DistributionͰͳ͍ͨΊɼߏ੒Ͱ͖ͳ͍ • Laplace approximation • ਖ਼ن෼෍ͷαϯϓϧʹsoftmaxؔ਺Λద༻ͯ͠୅༻ • ࣄલ෼෍ͷύϥϝʔλɿ µk = log( ↵k) 1 K K X i=1 log ↵i ⌃k,k = 1 ↵k (1 2 K ) + 1 K2 K X i=1 1 ↵k 11
12. ωοτϫʔΫͱϩεؔ਺ 12 X encoder µ( X ) ⌃ ( X ) KL {N( z ; µ( X ) , ⌃ ( X ))||N( z ; µ1, ⌃1)} ✏ ⇠ N(✏; 0, I ) + decoder: f ( Z ) loss ( x, f ( Z )) • σ: softmaxؔ਺ • β : DecoderͷॏΈʢunnormalizedʣ • σ(β): ୯ޠͷDiriclet෼෍͔ΒͷαϯϓϧʹରԠ L ( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) θ ස౓ϕΫτϧ
13. prodLDA: ఏҊϞσϧ • Products of Experts • βͱθͷੵʹsoftmaxؔ਺ 13 L ( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) ( ✓)
14. ࠷దԽͱωοτϫʔΫͷ޻෉ NVIͷ໰୊఺ɿ ֶशͷॳظஈ֊Ͱlocal optimumʹߦ͖΍͍͢ • AdamͷύϥϝʔλΛௐ੔ • ηͱβ1 ͷ஋ͷߴΊʹઃఆ • Batch NormalizationͱDropoutΛ࢖༻ 14
15. ࣮ݧ 1. CoherenceͱPerplexity • ޙड़ 2. ֶश཰ͱࣄલ෼෍Λม͑ͨͱ͖ͷޮՌ • ߴֶ͍श཰ & Dirichlet෼෍͕ϕλʔ 3. ςετσʔλʹର͢Δ࠷దԽͷ༗ແ • ͠ͳͯ͘΋͍͍ 4. p(w|β)ͷϦετ • লུ 15
16. Coherence 16 ද͸౰࿦จ͔ΒҾ༻ • LDA VAE: ఏҊਪ࿦๏ • prodLDA: ఏҊਪ࿦๏+ఏҊϞσϧ • LDA DMFVI: Online Mean-Field Variational Inference • NVDM: VAEϕʔεͷจॻϞσϦϯά දͷ஋: 40ճ࣮ߦͯ͠ࢉग़
17. Perplexity 17 ද͸౰࿦จ͔ΒҾ༻
18. ϨϏϡʔ: ؾʹͳͬͨ΋ͷΛ͍͔ͭ͘ Q1. NVDMͰ΋adamͷֶश཰Λม͑ͨํ͕ެฏ A1. ࿦จʹ൓ө Q2. ϋΠύʔύϥϝʔλ͸࠷దԽ͔ͨ͠ A2. ൺֱख๏͸͍ͯ͠ΔɼఏҊख๏͸BO Rating: 6-7-6-5 18
19. ͦͷଞ • ஶऀ࣮૷: TensorFlow • NVDMͷஶऀΒͷ৽Ϟσϧ͕ICML2017ʹ࠾࿥ 19