Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Autoencoding Variational Inference for Topic Mo...

Autoencoding Variational Inference for Topic Modelsの解説スライド

ICLR2017読み会のスライド https://connpass.com/event/57631/

Kento Nozawa

June 15, 2017
Tweet

More Decks by Kento Nozawa

Other Decks in Research

Transcript

  1. ֓ཁ 1. Latent Dirichlet Allocation (LDA) ΛNeural Variational Inference (NVI)

    Ͱ • Dirichlet ෼෍ͷ reparameterization trick 2. ৽ϞσϧͷఏҊ 3. ѱ͍ہॴղʹϋϚΔͷΛ༧๷ 2
  2. LDA จॻͷ֬཰తੜ੒Ϟσϧ [Blei et al., 2003]    

     จॻͷτϐοΫ෼෍Q [cВ ݚڀ ՝୊ ஌ࣝ Պֶऀ ʜ ػցֶश ਓ޻஌ೳ Ϟσϧ αϯϓϧ ʜ τϐοΫͷ୯ޠ෼෍ p(w|β) Ќ Ќ ػցֶश ػցֶशݚڀ ਓ޻஌ೳ՝୊ Ϟσϧ-%" Պֶ਺ֶण࢘ ίʔύε 4
  3. VAE: Encoder • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ • ֬཰෼෍͸જࡏม਺Λੜ੒

    • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 5
  4. VAE: Decoder • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ • ֬཰෼෍͸જࡏม਺Λੜ੒

    • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 6
  5. VAE: Reparameterization trick • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ •

    ֬཰෼෍͸જࡏม਺Λੜ੒ • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 7
  6. VAE: ϩεؔ਺ 8 L (⇥) = D X d=1 (

    1 2 ⇣ tr (⌃0) + µT 0 µ0 K log | ⌃0 | ⌘ + E ✏⇠N (0,1) ⇣ log p xd |f ( µ0 + ⌃ 1/2 0 ✏ ) ⌘ ) (Ⅰ) ࣄલ෼෍ͱͷKLμΠόʔδΣϯε (Ⅱ) ର਺໬౓ ࣜશମ: Evidence Lower Bound (I) (Ⅱ)
  7. Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet෼෍͔Βαϯϓϧ • Scale

    family DistributionͰͳ͍ͨΊɼߏ੒Ͱ͖ͳ͍ 10      จॻͷτϐοΫ෼෍Q [cВ
  8. Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet෼෍͔Βαϯϓϧ • Scale

    family DistributionͰͳ͍ͨΊɼߏ੒Ͱ͖ͳ͍ • Laplace approximation • ਖ਼ن෼෍ͷαϯϓϧʹsoftmaxؔ਺Λద༻ͯ͠୅༻ • ࣄલ෼෍ͷύϥϝʔλɿ µk = log( ↵k) 1 K K X i=1 log ↵i ⌃k,k = 1 ↵k (1 2 K ) + 1 K2 K X i=1 1 ↵k 11
  9. ωοτϫʔΫͱϩεؔ਺ 12 X encoder µ( X ) ⌃ ( X

    ) KL {N( z ; µ( X ) , ⌃ ( X ))||N( z ; µ1, ⌃1)} ✏ ⇠ N(✏; 0, I ) + decoder: f ( Z ) loss ( x, f ( Z )) • σ: softmaxؔ਺ • β : DecoderͷॏΈʢunnormalizedʣ • σ(β): ୯ޠͷDiriclet෼෍͔ΒͷαϯϓϧʹରԠ L ( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) θ ස౓ϕΫτϧ
  10. prodLDA: ఏҊϞσϧ • Products of Experts • βͱθͷੵʹsoftmaxؔ਺ 13 L

    ( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) ( ✓)
  11. ࣮ݧ 1. CoherenceͱPerplexity • ޙड़ 2. ֶश཰ͱࣄલ෼෍Λม͑ͨͱ͖ͷޮՌ • ߴֶ͍श཰ &

    Dirichlet෼෍͕ϕλʔ 3. ςετσʔλʹର͢Δ࠷దԽͷ༗ແ • ͠ͳͯ͘΋͍͍ 4. p(w|β)ͷϦετ • লུ 15
  12. Coherence 16 ද͸౰࿦จ͔ΒҾ༻ • LDA VAE: ఏҊਪ࿦๏ • prodLDA: ఏҊਪ࿦๏+ఏҊϞσϧ

    • LDA DMFVI: Online Mean-Field Variational Inference • NVDM: VAEϕʔεͷจॻϞσϦϯά දͷ஋: 40ճ࣮ߦͯ͠ࢉग़