Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CUGroup2015

Kyle Kastner
August 24, 2015

 CUGroup2015

Kyle Kastner

August 24, 2015
Tweet

More Decks by Kyle Kastner

Other Decks in Research

Transcript

  1. Conditional Modeling For Fun and Profit Kyle Kastner Université de

    Montréal - MILA Intern - IBM Watson @ Yorktown Heights
  2. Deep Learning, Simple Concepts • Universal function approximators • Learn

    the features • Desire hierarchy in learned features ◦ y = h(g(f(x)) ◦ {h, g, f} are nonlinear functions • Classification ◦ Learn p(y | x) = h(g(f(x)) [1]
  3. Basic Anatomy • Weights (W, V) • Biases (b, c)

    • Morph features using non-linear functions e.g. ◦ layer_1_out = tanh(dot(X, W) + b) ◦ layer_2_out = tanh(dot(layer_1_out, V) + c) ... • Backpropagation to “step” values of W,V,b,c [1, 2][]
  4. Mixture Density Networks • What are sufficient statistics? ◦ Describe

    an instance of a distribution ◦ Gaussian with mean u, variance s ◦ Bernoulli with probability p • Ties to neural networks ◦ Arbitrary output parameters ◦ Can we interpret parameters in a layer as sufficient statistics? YES! ◦ Cost / regularization forces this relationship [3, 1]
  5. Parameterizing Distributions • sigmoid -> Bernoulli • softmax -> Multinomial

    • linear, linear -> Gaussian with mean, log_var • softmax, linear, linear -> Gaussian mixture • Can combine with recurrence ◦ Learned, dynamic distributions over sequences ◦ Incredibly powerful [3, 1, 4, 5, 6, 7, 8, 9]
  6. Latent Factor Generative Models • Auto-Encoding Variational Bayes D. Kingma

    and M. Welling ◦ Model known as Variational Autoencoder (VAE) ◦ See also Stochastic Backpropagation and Approximate Inference in Deep Generative Models, Rezende, Mohamed, Wierstra [11, 12, 13]
  7. A Bit About VAE • Want to do latent variable

    modeling • Don’t want to do MCMC or EM • Sampling Z blocks gradient • Reparameterization trick ◦ Exact soln intractable for complex transforms (like NN) ◦ Lower bound on likelihood with KL divergence ◦ N(mu, sigma) -> mu + sigma * N(0, 1) ◦ Like mixture density networks, but in the middle ◦ Now trainable by backprop [11, 12, 13]
  8. Taking The Wheel • Specifics of MNIST digits ◦ Writing

    style and class ◦ Traits are semi-independent ◦ Can encode this in the model ◦ y -> softmax classifier (~y is sample) ◦ p(z | x, y), p(z | x, ~y) or p(z | x, f(x)) • Fully conditional version of M2 ◦ Semi-Supervised Learning with Deep Generative Models, Kingma, Rezende, Mohamed, Welling [13, 14]
  9. In Practice... • Conditioning is a strong signal ◦ p(x_hat

    | z) vs. p(x_hat | z, y) • Can give control or add prior knowledge • Classification is an even stronger form ◦ Prediction is learned by maximizing p(y | x) ! ◦ In classification, don’t worry about forming a useful z [1, 13, 14]
  10. Conditioning Feedforward • Concatenate features ◦ concatenate((X_train, conditioning), axis=1) ◦

    p(y | X_1 … X_n, L_1 … L_n) • One hot label L (scikit-learn label_binarize) • Could also be real valued • Concat followed with multiple layers to “mix” [1]
  11. Convolution and Recurrence • Exploit structure and prior knowledge ◦

    Parameter sharing is strong regularization • Convolution - exploit locality ◦ p(y | X_{i - n} … X_{i + n}) * p(y | X_{i + 1 - n} ... X_{i +1 + n})... ◦ A learned filter over a fixed 1D or 2D window ◦ Window slides over all input, updates filter • Recurrence - exploit sequential information ◦ p(y | X_1 … X_t) = p(y | X_<=t) can be seen as: ◦ ~ p(y | X_1) * p(y | X_2, X_1) * p(y | X_3, X_2, X_1) ... [1, 4, 5, 6, 7, 8, 9]
  12. More on Recurrence • Hidden state (s_t) encodes sequence info

    ◦ p(X_<=t) (in s_t) is compressed representation of X • Recurrence similar to ◦ Hidden Markov Model (HMM) ◦ Kalman Filter (KF, EKF, UKF) [1, 4, 15, 16]
  13. How-To MDN + RNN • Generating Sequences with Recurrent Neural

    Networks Alex Graves ◦ http://arxiv.org/abs/1308.0850 • Multi-level RNN, outputs GMM and bernoulli ◦ Handwriting ▪ Pen up/down and relative position per timestep ◦ Vocoder representation of speech ▪ Voiced/unvoiced and MFCC per timestep [3, 4]
  14. How-To Continued • Conditional model ◦ Adds input attention (more

    on this later) ◦ Gaussian per timestep over one hot text ◦ p(bernoulli, GMM | X_t, previous state, focused text) ◦ This gives control of the output via input text http://www.cs.toronto.edu/~graves/handwriting.html https://www.youtube.com/watch?v=-yX1SYeDHbg&t=43m30s [3, 4]
  15. Similar Approaches • RNN with sigmoid output ◦ ALICE •

    RNN with softmax ◦ RNN-LM • RNN-RBM, RNN-NADE [3, 1, 4, 5, 6, 7, 8, 9]
  16. Research Questions • Possible Issues ◦ Prosody/style are not smooth

    over time ◦ Deep network, but still shallow latent variables ◦ Vocoder is a highly engineered representation • How can we fix these problems? ◦ First, a bit about conditioning in RNNs
  17. Conditioning In Recurrent Networks • RNNs model p(X_t | X_<t)

    • Initial hidden state can condition ◦ p(X_t | X_<t, c) where c is init. hidden state (context) • Condition by concatenating in feedforward ◦ Before recurrence or after • Can do all of the above [1, 4, 15, 16, 17]
  18. Conditioning with a Sequence • RNN outputting Gaussian parameters over

    seq ◦ Seen in Generating Sequences • Use an RNN to compress ◦ Hidden state encodes p(X_<=t) ◦ Project into init hidden and ff ◦ Now have p(y_t | y_<t, X_<=t) ◦ Known as RNN Encode-Decode ◦ Cho et al [16, 17]
  19. Distributing The Representation • Distribute context, Bahdanau et al •

    Bidirectional RNN ◦ p(X_i | X_<i, X_>i) for i in t ◦ Needs whole sequence ◦ But sometimes this is fine • Soft attention over hiddens • Choose what is important [16, 17, 18]
  20. Previously, on FOX... • RNN-GMM Issues ◦ Prosody/style are not

    smooth over time ◦ Deep network, but still shallow latent variables ◦ Vocoder is a highly engineered representation • How can we try to fix these problems? ◦ Distributed latent representation for Z ◦ Use modified VAE to make latents deep ◦ Work on raw timeseries inputs ▪ Extreme approach, but proves a point
  21. Existing Approaches • VRAE, Z_t independent • STORN, Z_t independent

    • DRAW, Z_t loosely dependent via canvas • No large scale real-valued experiments ◦ VRAE, no real valued experiment ◦ STORN, real valued experiment was small ◦ DRAW, real values weren’t sequences [18, 19, 20]
  22. Variational RNN • Speech ◦ Complex but structured noise driven

    by mechanics ◦ Ideal latent factors include these mechanics • Z_<t should affect Z_t and h_t • Use a recurrent prior [15]
  23. Prior • Used for KL divergence • Fixed in VAE

    to N(0, 1) • Here it is learned • Instead of “be simple” (as in VAE), this says “be consistent” [15]
  24. Inference (encode) • Previous hidden state ◦ h_t-1 • Data

    ◦ X_t • Hidden state information ◦ z_<t ◦ X_<t [15]
  25. Generation (decode) • Generate based on ◦ Z_t, h_t-1 ◦

    h_t-1 has z_<t, X_<t ◦ Z_t has z_<t, X_<=t [15]
  26. Recurrence • Just a regular RNN • Input projection is

    a VAE • Can use LSTM, GRU, others [15]
  27. Final Thoughts on VRNN • Empirically, structured Z seems to

    help ◦ Keep style consistent ◦ Predict very correlated data, like raw timeseries ◦ Also works well for unconditional handwriting RNN-GMM VRNN-GMM [4, 15]
  28. Takeaways and Opinions • Can use deep learning like graphical

    modeling ◦ Different tools, same conceptual idea ◦ Conditional probability modeling is key • Put knowledge in model structure, not features • Let features be learned from data • Use conditioning to control or constrain
  29. References (1) [1] Y. Bengio, I. Goodfellow, A. Courville. “Deep

    Learning”, in preparation for MIT Press, 2015. http://www.iro.umontreal.ca/~bengioy/dlbook/ [2] D. Rumelhart, G. Hinton, R. Williams. "Learning representations by back-propagating errors", Nature 323 (6088): 533–536, 1986. http://www.iro. umontreal.ca/~vincentp/ift3395/lectures/backprop_old.pdf [3] C. Bishop. “Mixture Density Networks”, 1994. http://research.microsoft.com/en-us/um/people/cmbishop/downloads/Bishop-NCRG-94-004.ps [4] A. Graves. “Generating Sequences With Recurrent Neural Networks”, 2013. http://arxiv.org/abs/1308.0850 [5] D. Eck, J. Schmidhuber. “Finding Temporal Structure In Music: Blues Improvisation with LSTM Recurrent Networks”. Neural Networks for Signal Processing, 2002. ftp://ftp.idsia.ch/pub/juergen/2002_ieee.pdf [6] A. Brandmaier. “ALICE: An LSTM Inspired Composition Experiment”. 2008. [7] T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, S. Khudanpur. “Recurrent Neural Network Based Language Model”. Interspeech 2010. http://www.fit. vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf [9] N. Boulanger-Lewandowski, Y. Bengio, P. Vincent. “Modeling Temporal Dependencies in High-Dimensional Sequences: Application To Polyphonic Music Generation and Transcription”. ICML 2012. http://www-etud.iro.umontreal.ca/~boulanni/icml2012 [10] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learning applied to document recognition." Proceedings of the IEEE, 86(11):2278- 2324, 1998. http://yann.lecun.com/exdb/mnist/ [11] D. Kingma, M. Welling. “Auto-encoding Variational Bayes”. ICLR 2014. http://arxiv.org/abs/1312.6114 [12] D. Rezende, S. Mohamed, D. Wierstra. “Stochastic Backpropagation and Approximate Inference in Deep Generative Models”. ICML 2014. http: //arxiv.org/abs/1401.4082
  30. References (2) [13] A. Courville. “Course notes for Variational Autoencoders”.

    IFT6266H15. https://ift6266h15.files.wordpress.com/2015/04/20_vae.pdf [14] D. Kingma, D. Rezende, s. Mohamed, M. Welling. “Semi-supervised Learning With Deep Generative Models”. NIPS 2014. http://arxiv. org/abs/1406.5298 [15] J. Chung, K. Kastner, L. Dinh, K. Goel, A. Courville, Y. Bengio. “A Stochastic Latent Variable Model for Sequential Data”. http://arxiv.org/abs/1506. 02216 [16] K. Cho, B. Merrienboer, C. Gulchere, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio. “Learning Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation”. EMNLP 2014. http://arxiv.org/abs/1406.1078 [17] D. Bahdanau, K. Cho, Y. Bengio. “Neural Machine Translation By Jointly Learning To Align and Translate”. ICLR 2015. http://arxiv.org/abs/1409. 0473 [18] K. Gregor, I. Danihelka, A. Graves, D. Rezende, D. Wierstra. “DRAW: Directed Recurrent Attention Writer”. http://arxiv.org/abs/1502.04623 [19] J. Bayer, C. Osendorfer. “Learning Stochastic Recurrent Networks”. http://arxiv.org/abs/1411.7610 [20] O. Fabius, J. van Amersmoot. “Variational Recurrent Auto-Encoders”. http://arxiv.org/abs/1412.6581
  31. More on Convolution • Define size of feature map and

    how many ◦ Similar to output size of feedforward layer • Parameter sharing ◦ Small filter moves over entire input ◦ Believe local statistics consistent over regions ◦ Enforced by parameter sharing • Condition by concatenating ◦ Along “channel” axis ◦ http://arxiv.org/abs/1406.2283