Share
Embed
Transcript
61 slides
-
The State of Deep Learning in 2014 Deep Learning Paris
Meetup December 2014 in 30min -
Content Warnings This talk contains buzz-words and highly non-convex objective
functions that some attendees may find disturbing. -
Outline • Deep Learning for Computer Vision • Word Embeddings
for Natural Language Understanding & Machine Translation • Decoding visual embeddings as English sentences • Learning to Play, Execute, Program, Answer Queries -
Deep Learning in the 90’s • Yann LeCun invented Convolutional
Networks • First NN successfully trained with many layers -
ImageNet Challenge 2012 • 1.2M images labeled with 1000 object
categories • AlexNet from the deep learning team of U. of Toronto wins with 15% error rate vs 26% for the second (traditional CV pipeline) • Best NN was trained on GPUs for weeks -
ImageNet Challenge 2013 • Clarifai ConvNet model wins at 11%
error rate • Many other participants used ConvNets • OverFeat by Pierre Sermanet from NYU: shipped binary program to execute pre-trained models -
Transfer to other CV tasks • KTH CV team: CNN
Features off-the-shelf: an Astounding Baseline for Recognition “It can be concluded that from now on, deep learning with CNN has to be considered as the primary candidate in essentially any visual recognition task.” -
Jetpac: analysis of social media photos • Ratio of smiles
in faces: city happiness index • Ratio of mustaches on faces: hipster-ness index for coffee-shops • Ratio of lipstick on faces: glamour-ness index for night club and bars -
ImageNet Challenge 2014 • In the mean time Pierre Sermanet
had joined other people from Google Brain • Monster model: GoogLeNet now at 6.7% error rate -
Very Deep Nets • 2nd position on classification, 1st on
localization task • 16 to 19 weight layers (without max pool and ReLU) • Small 3x3 convolution kernels • Sequence of Conv + ReLU layers then max pool • Supervised pre-training: insert new conv layer to previously model before each max pool • Pre-trained models for Caffe a.k.a. VGGNet -
GoogLeNet vs Andrej • Andrej Karpathy evaluated human performance (himself):
~5% error rate • "It is clear that humans will soon only be able to outperform state of the art image classification models by use of significant effort, expertise, and time.” • “As for my personal take-away from this week-long exercise, I have to say that, qualitatively, I was very impressed with the ConvNet performance. Unless the image exhibits some irregularity or tricky parts, the ConvNet confidently and robustly predicts the correct label.” source: What I learned from competing against a ConvNet on ImageNet -
Neural Language Models • Each word is represented by a
fixed dimensional vector • Goal is to predict target word given ~5 words context from a random sentence in Wikipedia • Random substitutions of the target word to generate negative examples • Use NN-style training to optimize the vector coefficients -
Progress in 2013 / 2014 • Simpler linear models (word2vec)
benefit from larger training data (1B+ words) and dimensions (300+) • Some models (GloVe) now closer to matrix factorization than neural networks • Can successfully uncover semantic and syntactic word relationships, unsupervised! -
Analogies • [king] - [male] + [female] ~= [queen] •
[Berlin] - [Germany] + [France] ~= [Paris] • [eating] - [eat] + [fly] ~= [flying] -
RNN w/ LSTM for MT source: Learning Phrase Representations using
RNN Encoder- Decoder for Statistical Machine Translation -
Idea in the air • Baidu/UCLA: http://arxiv.org/pdf/1410.1090v1.pdf • Berkeley: http://arxiv.org/abs/1411.4389
• Google: http://googleresearch.blogspot.com/2014/11/ a-picture-is-worth-thousand-coherent.html • Stanford: http://cs.stanford.edu/people/karpathy/ deepimagesent/ • University of Toronto: http://arxiv.org/pdf/ 1411.2539v1.pdf -
Open Source implementation • code: https://github.com/karpathy/neuraltalk • models: http://cs.stanford.edu/people/karpathy/ neuraltalk/
• Use pre-tained VGGnet with Caffe to extract the vector representations of the images • Pure NumPy / OpenBLAS implementation of RNN + LSTM decoders. -
DeepMind: Learning to Play & win dozens of Atari games
• DeepMind startup demoed at NIPS 2013 a new Deep Reinforcement Learning algorithm • Raw pixel input from Atari games (state space) • Keyboard keys as action space • Scalar signal (score & game over) as reward • CNN trained with a Q-Learning variant -
Learning to Execute • Google Brain & NYU, October 2014
(very new) • RNN trained to map character representations of programs to outputs • Can learn to emulate a simplistic Python interpreter from examples programs & expected outputs • Limited to one-pass programs with O(n) complexity -
Neural Turing Machines • Google DeepMind, October 2014 (very new)
• Neural Network coupled to external memory (tape) • Analogue to a Turing Machine but differentiable • Can be used to learn to simple programs from example input / output pairs • copy, repeat copy, associative recall, • binary n-grams counts and sort -
Architecture source: Neural Turing Machines • Turing Machine: controller ==
FSM • Neural Turing Machine controller == RNN w/ LSTM -
Answering Queries • Memory Networks (Facebook AI Research) • http://arxiv.org/abs/1410.3916
• Explicitly addressable memory to manage a knowledge base • Able to use new words at query time (unseen at training time) -
Concluding remarks • Deep Learning now state of the art
at: • Several computer vision tasks • Speech recognition (partially NN-based in 2012, fully in 2013) • Machine Translation (English / French) and Q&A • Multi model tasks: caption generation • Recurrent Neural Network w/ LSTM units seems to be applicable to problems initially thought out of the scope of Machine Learning • Stay tuned for 2015! -
References • ConvNets in the 90’s by Yann LeCun: LeNet-5
http://yann.lecun.com/exdb/lenet/ • ImageNet Challenge 2012 winner: AlexNet (Toronto) http://papers.nips.cc/paper/4824-imagenet-classification-with-deep- convolutional-neural-networks • ImageNet Challenge 2013: OverFeat (NYU) http://cilvr.nyu.edu/doku.php?id=software:overfeat:start • ImageNet Challenge 2014 winner: GoogLeNet (Google Brain) http://googleresearch.blogspot.fr/2014/09/building-deeper-understanding-of- images.html -
References • Word embeddings First gen: http://metaoptimize.com/projects/wordreprs/ Word2Vec: https://code.google.com/p/word2vec/ GloVe:
http://nlp.stanford.edu/projects/glove/ • Neural Machine Translation Google Brain: http://arxiv.org/abs/1409.3215 U. of Montreal: http://arxiv.org/abs/1406.1078 https://github.com/lisa-groundhog/GroundHog -
References • Deep Reinforcement Learning: http://www.cs.toronto.edu/~vmnih/docs/dqn.pdf • Neural Turing Machines:
http://arxiv.org/abs/1410.5401 • Learning to Execute: http://arxiv.org/abs/1410.4615