Sum Product Networks

Transcript

1. Sum Product Networks: A New Deep Architecture
   ! Hoifung Poon and Pedro Domingos (2011) ! St. Louis Machine Learning & Data Science March 20, 2013 ! ! Steven Borrelli steve@borrelli.org @stevendborrelli {twitter, github}
4. http://deeplearningworkshopnips2010.files.wordpress.com/2010/09/nips10-workshop-tutorial-final.pdf Deep Learning
5. Sum Product Network
   Hoifung Poon and Pedro Domingos, 2011
   ! • Models with multiple layers of hidden variables allow for efficient inference in a much larger class of distributions. • However, deep architectures leads are difficult to learn and require approximate methods (MCMC, etc.) • Complex Networks have intractable Partition functions:
6. Sum Product Network
   Hoifung Poon and Pedro Domingos, 2011
   ! Mixture Feature Directed Acyclic Graph http://homes.cs.washington.edu/~pedrod/papers/nips12.pdf
7. Sum Product Network
   ⊗ ⊕ 0.4 ⊗ ⊗ ⊗ 0.2 0.1 0.3 ⎯ X1 X2 X1 ⎯ X2 X1 X2 P(X) 1 1 0.4 1 0 0.2 0 1 0.1 0 0 0.3 P(X) = 0.4 ⋅ X1 ⋅ X2 ! + 0.2 ⋅ X1 ⋅ X2 ! + 0.1 ⋅ X1 ⋅ X2 ! + 0.3 ⋅ X1 ⋅ X2 ⎯ ⎯ ⎯ ⎯
8. Computing SPNs
9. Mixture model vs. SPN N * 2N-1 2N-1 ~N
10. Partition Function https://class.coursera.org/neuralnets-2012-001
11. Image recognition: Google
12. U of Toronto DRBM http://www.cs.toronto.edu/~hinton/science.pdf
13. SPN Architecture http://homes.cs.washington.edu/~pedrod/papers/nips12.pdf
14. Caltech 101 Results (MSE)
15. Caltech 101 Image completion 64 SPN DBN Nearest Neighbor DBM PCA Original
16. Potential Benefits over other Deep Architectures
    ! • Less complex engineering, exact inference due to tractability of partition function. • Order of magnitude faster in learning and inference • Better Mean Square Error, Precision/recall and qualitative results.