Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let the AI Do the Talk: Adventures with Natural...

Let the AI Do the Talk: Adventures with Natural Language Generation

Slides for my presentation on Natural Language Generation at PyCon X (www.pycon.it)

Marco Bonzanini

May 05, 2019
Tweet

More Decks by Marco Bonzanini

Other Decks in Programming

Transcript

  1. Let the AI do the Talk Adventures with Natural Language

    Generation @MarcoBonzanini #PyConX
  2. Infinite Monkey Theorem from random import choice from string import

    printable def monkey_hits_keyboard(n): output = [choice(printable) for _ in range(n)] print("The monkey typed:") print(''.join(output)) 20
  3. Infinite Monkey Theorem >>> monkey_hits_keyboard(30) The monkey typed: % a9AK^YKx

    OkVG)u3.cQ,31("!ac% >>> monkey_hits_keyboard(30) The monkey typed: fWE,ou)cxmV2IZ l}jSV'XxQ**9'| 21
  4. n-grams >>> from nltk import ngrams >>> list(ngrams("pizza", 3)) [('p',

    'i', 'z'), ('i', 'z', 'z'), ('z', 'z', ‘a')] 25
  5. n-grams >>> from nltk import ngrams >>> list(ngrams("pizza", 3)) [('p',

    'i', 'z'), ('i', 'z', 'z'), ('z', 'z', ‘a')] character-based trigrams 26
  6. n-grams >>> s = "The quick brown fox".split() >>> list(ngrams(s,

    2)) [('The', 'quick'), ('quick', 'brown'), ('brown', 'fox')] 28
  7. n-grams >>> s = "The quick brown fox".split() >>> list(ngrams(s,

    2)) [('The', 'quick'), ('quick', 'brown'), ('brown', 'fox')] word-based bigrams 29
  8. From n-grams to Language Model • Given a large dataset

    of text • Find all the n-grams • Compute probabilities, e.g. count bigrams:
 
 
 31
  9. Marco is a good time to get the latest flash

    player is required for video playback is unavailable right now because this video is not sure if you have a great day. 36 Example: Predictive Text in Mobile
  10. Limitations of LM so far • P(word | full history)

    is too expensive • P(word | previous few words) is feasible • … Local context only! Lack of global context 38
  11. Neural Networks 41 x1 x2 h1 y1 h2 h3 Input

    layer Output layer Hidden layer(s)
  12. Training the Network 46 • Random weight init • Run

    input through the network • Compute error
 (loss function) • Use error to adjust weights
 (gradient descent + back-propagation)
  13. More on Training • Batch size • Iterations and Epochs

    • e.g. 1,000 data points, if batch size = 100 we need 10 iterations to complete 1 epoch 48
  14. Deep Learning in Python • Some NN support in scikit-learn

    • Many low-level frameworks: Theano, PyTorch, TensorFlow • … Keras! • Probably more 62
  15. Keras • Simple, high-level API • Uses TensorFlow, Theano or

    CNTK as backend • Runs seamlessly on GPU • Easier to start with 64
  16. LSTM Example model = Sequential() model.add( LSTM( 128, input_shape=(maxlen,len(chars)) )

    ) model.add(Dense(len(chars), activation='softmax')) 66 Define the network
  17. LSTM Example for i in range(output_size): ... preds = model.predict(x_pred,

    verbose=0)[0] next_index = sample(preds, diversity) next_char = indices_char[next_index] generated += next_char 69 Generate text
  18. LSTM Example for i in range(output_size): ... preds = model.predict(x_pred,

    verbose=0)[0] next_index = sample(preds, diversity) next_char = indices_char[next_index] generated += next_char 70 Seed text
  19. Sample Output are the glories it included. Now am I

    lrA to r ,d?ot praki ynhh kpHu ndst -h ahh umk,hrfheleuloluprffuamdaedospe aeooasak sh frxpaphrNumlpAryoaho (…) 72 Seed text After 1 epoch
  20. Sample Output I go from thee: Bear me forthwitht wh,

    t che f uf ld,hhorfAs c c ff.h scfylhle, rigrya p s lee rmoy, tofhryg dd?ofr hl t y ftrhoodfe- r Py (…) 73 After ~5 epochs
  21. Sample Output a wild-goose flies, Unclaim'd of any manwecddeelc uavekeMw

    gh whacelcwiiaeh xcacwiDac w fioarw ewoc h feicucra h,h, :ewh utiqitilweWy ha.h pc'hr, lagfh eIwislw ofiridete w laecheefb .ics,aicpaweteh fiw?egp t? (…) 74 After 20+ epochs
  22. Wyr feirm hat. meancucd kreukk? , foremee shiciarplle. My, Bnyivlaunef

    sough bus: Wad vomietlhas nteos thun. lore orain, Ty thee I Boe, I rue. niat 77 Tuning After 1 epoch
  23. to Dover, where inshipp'd Commit them to plean me than

    stand and the woul came the wife marn to the groat pery me Which that the senvose in the sen in the poor The death is and the calperits the should 78 Tuning Much later
  24. A Couple of Tips • You’ll need a GPU •

    Develop locally on very small dataset
 then run on cloud on real data • At least 1M characters in input,
 at least 20 epochs for training • model.save() !!!
  25. Summary • Natural Language Generation is fun • Simple models

    vs. Neural Networks • Keras makes your life easier • A lot of trial-and-error!
  26. • Brandon Rohrer on "Recurrent Neural Networks (RNN) and Long

    Short-Term Memory (LSTM)": https://www.youtube.com/watch?v=WCUNPb-5EYI • Chris Olah on Understanding LSTM Networks:
 http://colah.github.io/posts/2015-08-Understanding-LSTMs/ • Andrej Karpathy on "The Unreasonable Effectiveness of Recurrent Neural Networks":
 http://karpathy.github.io/2015/05/21/rnn-effectiveness/ Pics: • Weather forecast icon: https://commons.wikimedia.org/wiki/File:Newspaper_weather_forecast_-_today_and_tomorrow.svg • Stack of papers icon: https://commons.wikimedia.org/wiki/File:Stack_of_papers_tied.svg • Document icon: https://commons.wikimedia.org/wiki/File:Document_icon_(the_Noun_Project_27904).svg • News icon: https://commons.wikimedia.org/wiki/File:PICOL_icon_News.svg • Cortana icon: https://upload.wikimedia.org/wikipedia/commons/thumb/8/89/Microsoft_Cortana_light.svg/1024px- Microsoft_Cortana_light.svg.png • Siri icon: https://commons.wikimedia.org/wiki/File:Siri_icon.svg • Google assistant icon: https://commons.wikimedia.org/wiki/File:Google_mic.svg Readings & Credits