Deep NLP: 딥러닝을 이용한 자연어처리

머신러닝, 딥러닝 2학기 수업분량을 2시간으로 요약
VCNC 개발팀 워크샵 (2017/12/20)

Transcript

1. Deep NLP ٩۞׬ਸ ੉ਊೠ ੗োয୊ܻ ѐߊ੗ܳ ਤೠ ӝࣿ ইਓۄੋҗ ৘ઁ ઺बਵ۽ ӣ࢚਋ @ VCNC kevin@between.us
2. Why?? • ਋ܻо ೞҊ ੓ח ݽ߄ੌ জ ࠺ૉפझח അ੤о ੿੼ • 5֙, 10֙ ٍীח যڄө? • ݠन۞׬਷ ਋ܻ ࢤഝী ௾ ߸ചܳ оઉৢ ࣻ ੓ח ӝࣿ ઺ ೞա • 100֙ оח ӝস੉ غ۰ݶ ഥࢎب, ࢎۈب ߸ न੉ ೙ਃ • ׮ٜ प۱җ ҃೷ ੓ਵ޲۽ ޤٚ ٮۄ੟ইࢲ ੜ ೡ ࣻ ੓ѷ૑݅, ޷ܻ ળ࠺ೞҊ ҙब о૑ݶ જ ਸ Ѫ
3. ݠन۞׬ • ஹೊఠ җ೟੄ ೠ ࠙ঠ۽, যڃ ۽૒ਸ ೐۽Ӓې߁ ೞ૑ ঋҊب ஹೊ ఠо ੘স੉ա ૑धਸ ߓ਎ ࣻ ੓ѱ ೞח ࠙ঠ • ؀ࠗ࠙ Data-driven prediction ഑਷ decision ਸ ࣻ೯ೞח ঌҊ ્ܻٜ • ؘ੉ఠ ࠙ࢳ ࠙ঠীࢲب օܻ ॳ੉חؘ, ࠂ੟ೠ ݽ؛ਸ ࠙ࢳоо ૒੽ ٜ݅૑ ঋҊ ؘ੉ఠܳ ా೧ ೟णदெ যځೠ ࢎपਸ ৘ஏೞѢա, ऀѹ ૓ ੋࢎ੉౟ܳ ঌইղח ੌ ١ਸ ࣻ೯
4. ٩۞׬ (Deep Learning) • ݠन۞׬੄ ೞਤ ࠙ঠ ઺ ೞա, ౠ൤ Neural Network (ੋҕ न҃ ݎ) ҅ৌ ঌҊ્ܻҗ োҙ੓਺ • Deep Neural Network ੉ۄҊ ࠗܰӝب ೣ • ୭Ӕ ಩ߊ੸ਵ۽ ҙबਸ ߉Ҋ ੓ח ࠙ঠ۽, ӝઓ ݠन۞׬ ߑߨۿ੄ ೠ҅ܳ ӓࠂೞח ઺
5. ٩۞׬ (Deep Learning) • Rawೠ input (Ӗ੗, ܻࣗ, ױ য ١١) ਸ ੑ۱߉਺ • ઺р઺р ৈ۞ ҅கਵ۽ ؘ੉ఠ ٜਸ ઑ೤ೞৈ ಴അ • ҅கਸ ੼੼ ऺই ৢ۰оݴ ࢚ ਤ ѐ֛ਸ ಴അೡ ࣻ ੓ѱ ؽ
6. ӝઓ ݠन۞׬ ߑߨٜ੄ ೠ҅ • അ੤੄ ݠन۞׬ ߑߨٜ਷ ؀ࠗ࠙ Data-Driven ߑߨٜ • যڃ ੌਸ द೯ೞח ۽૒ਸ ೐۽Ӓې߁ೞחѪ੉ ইפۄ, ೟णೡ ࣻ ੓ ח ҳઑܳ ٜ݅য֬Ҋ ؘ੉ఠܳ ઁҕ, ஹೊఠо ঌইࢲ ೟ण • যڃ ؘ੉ఠܳ, যڃ ҳઑ۽ ೟णदఃוջо ઺ਃ! • ౠ൤ ؘ੉ఠ ࠗ࠙੄ ઺ਃࢿ੉ ௼Ҋ ؘ੉ఠܳ যڌѱ оҕ೧ࢲ ֍וջ о ݽ؛੄ ࢿמਸ ઝ਋ೣ -> Feature Engineering
7. Feature Engineering • ؘ੉ఠܳ оҕ೧ࢲ ஹೊఠܳ ೟णदఆ featureܳ ݅٘۰ݶ, ೧׼ ࠙ ঠী ؀ೠ ੹ޙ੸ੋ ૑ध (domain knowledge)੉ ೙ਃೞݴ दрҗ ࠺ਊ੉ ݆੉ ٘ח ੘স੐ • ਕ௼೒۽਋ - Feature engineering -> Machine training -> Evaluate result -> Idea -> Feature engineering -> …
8. ٩۞׬਷ ޤо ׮ܲо? • ٩۞׬ (Deep Neural Network) • ಴അ۱੉ જ਷ (ࠂ੟ೠ ղਊٜਸ ೟णೡ ࣻ ੓ח) ֎౟ਕ௼ܳ ৈ۞க ऺইࢲ, ֎౟ਕ௼о featureٜਸ झझ۽ ٜ݅যղҊ ೟णೡ ࣻ ੓ب ۾ ೞ੗!
9. ঱যܳ ਤೠ ٩۞׬ • ٩۞׬ ݽ؛ਸ ഝਊೠ ਺ࢿੋध • ӝઓ੄ ਺ࢿੋध ߑߨ ؀࠺ ࢿמ੉ ࠺ড੸ਵ۽ ೱ࢚ؽ
10. ੉޷૑ܳ ਤೠ ٩۞׬ • ୭Ӕ ٩۞׬ਸ ഝਊ೧ о੢ ݆੉ োҳо غҊ ੓ח ࠙ঠ • ImageNet ؀ഥীࢲ જ਷ ࢿ੸ਸ ղݶࢲ ݆਷ ҙबਸ ঳਺
11. ੗োয ୊ܻীࢲ੄ ٩۞׬ ਽ਊ • ఫझ౟੄ х੿࠙ࢳ (ӛ੿/ࠗ੿) • ੋҕ૑מ ࠺ࢲ • ૕੄਽׹ • ੗ز ߣ৉
12. ٩۞׬ - х੿࠙ࢳ • ੉੹੄ ߑߨ - ৈ۞ ױযٜ੉ ӛ੿/ࠗ੿੄ х੿ਸ о૑ח૑ DBҳ୷ - ޙ੢ী ೧׼ ױযٜ੉ ঴݃ա ١੢ೞח૑ ଺ӝ - ੤޷о ੓׮ - ੤޷о হ׮ - “੤޷” ۄח ױয݅ਵ۽ח ౸ױೡ ࣻ হ਺ • ٩۞׬ਸ ഝਊೠ ߑߨ - ޙߨ੉ա ੄޷੸ੋ Ѫө૑ Ҋ۰ೞৈ ౸߹ оמ
13. ٩۞׬ - ૕੄਽׹ • ੉੹੄ ߑߨ - ݆ࣻ਷ ૕ޙ-׹߸ DBܳ ҳ୷ೞҊ, ࢜۽ ٜযয়ח ૕ޙਸ ੉੹੄ ૕׹ ࣇ ীࢲ ଺਺ • ٩۞׬ਸ ഝਊೠ ߑߨ - ৈ۞ ૕ޙী ؀ೠ ׹੉ vector ഋधਵ۽ ݽ؛ী ֣ইٜয੓਺
14. ٩۞׬ - ੗زߣ৉ • ੉੹੄ ߑߨ - ই઱ ࠂ੟ೠ ৈ۞ ҅க੄ ߣ৉ ݽ؛ • ٩۞׬ਸ ഝਊೠ ߑߨ - ߣ৉ೡ ޙ੢ਸ vector۽ ಴അೞҊ, ੉ܳ ߸ജೞח ݽ؛ਸ ೟णदఇ
16. ݠन۞׬ ѐਃ
17. ݠन۞׬ (Machine Learning) • “The field of study that gives computers the ability to learn without being explicitly programmed.” - Arthur Samuel • “A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” - Tom Mitchell • ৘: ߄قѱ੐ਸ فח ஹೊఠ - E = ߄قѱ੐ਸ فח ҃೷஖ - T = ߄قѱ੐ਸ فח ೯ਤ - P = ஹೊఠо ׮਺౸੄ ߄قਸ ੉ӡ ഛܫ
18. ݠन۞׬ (Machine Learning) • Supervised Learning (૑ب ೟ण) - Regression - োࣘ੸ੋ чਸ ৘ஏ • ৘: ࠗز࢑ оѺ ৘ஏ - Classification - ࠛোࣘ੸ੋ ч ৘ஏ • ৘: ࢎ૓ਸࠁҊ ࢎޛ ݏ୶ӝ - Regressionޙઁب Classification ޙઁ۽ ߄Չࣻب ੓׮! • ৘: ࠗز࢑ оѺਸ ࠁҊ ࢓૑ উ࢓૑ Ѿ੿ • Unsupervised Learning (࠺૑ب ೟ण) - Clustering • Reinforcement Learning (ъച ೟ण) - ѥחߨਸ झझ۽ ߓ਋ח ۽ࠈ
19. Linear Regression
20. Linear Regression • ૘੄ և੉ ؘ੉ఠ۽ ࠗز࢑ оѺਸ ৘ஏ೧ ࠁ੗ • 30 ಣ = 3র 5000݅ਗ • 50ಣ = 5র 5000݅ਗ • 10ಣ = 1র 5000݅ਗ • 20ಣ = ?
21. Linear Regression • y = a * x + b - x: և੉(ಣ) - y: оѺ - h(x) ৬ э੉ ಴അೞӝب ೣ (hypothesis, оࢸ) - ৬ э੉ ಴അ • Training dataܳ learning algorithmী ੸ਊ, h ܳ ҳೣ! • h ܳ ੉ਊೞৈ x ী ؀ೠ Ѿҗч (y) ܳ ৘ஏ оמ
22. Cost Function • যڃ о੿ hܳ ࣁਛਸٸ, Ӓ о੿੉ ঴݃ա ౣ۷חо(error) ܳ ࠺ਊ (cost)۽ р઱ೞৈ ࣻधਸ ࣁ਑ • ੉ۧѱ ੿੄ೠ Costܳ ୭ࣗചೞח ߑೱਵ۽ ౵ۄݫఠ a,bٜਸ ઑ੿ ೞݶ ખ؊ ৢ߄ܲ о੿ h ܳ о૕ ࣻ ੓׮!
23. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function • पઁ ؘ੉ఠח h(x) = 1*x ੋؘ, h(x) = 0.5*x ۽ ৘ஏೠ ҃਋ • য়ରч (Error, Cost): 0 1 2 3 3 2 1 0
24. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function • ੄ 2ରߑ੿ध!!
25. Linear Regression(ױੌ ߸ࣻ)੄ Cost Function • , ী ؀೧ࢲח ইې৬ э਷ ੑ୓੸ੋ ݽন੉ ػ׮ • যڌѱ ୭੷੼ਸ ଺ਸѪੋо?
26. Gradient Descent • য়ରчਸ যڌѱ ୭ࣗചೡѪੋо? - Cost functionਸ ୭ࣗച - Cost function ١Ҋࢶ੄ Ҏ૞ӝ ଺ӝ • ੐੄੄ ૑੼ীࢲ द੘ೞৈ, ӝ਎ӝо ծই૑ח ૑੼ଃਵ۽ ҅ࣘ ੉زೞ੗ • ӝ਎ӝ = Cost function੄ ޷࠙ч
27. Gradient Descent • ੉ җ੿ਸ ࣻ۴ೡٸө૑ ߈ࠂ • a: learning rate
28. Cost Function੄ ޷࠙ч
29. Cost Function੄ ޷࠙ч
30. Linear Regression੄ Gradient Descent
31. Linear Regression੄ Gradient Descent • Bowl-shaped convex function
32. Linear Regression੄ Gradient Descent • Iterationਸ ా೧ ୭੸੼ਸ ଺ইх (for fixed , this is a function of x) (function of the parameters )
33. Linear Regression੄ Gradient Descent (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters )
34. Linear Regression੄ Gradient Descent (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters ) (for fixed , this is a function of x) (function of the parameters )
35. Multi Variable Linear Regression • ࠗز࢑ оѺਸ ৘ஏೞחؘח ಣࣻ ݈Ҋب ׮নೠ ੗ܐܳ ଵҊೡ ࣻ ੓ ਸѪ - ߑ੄ іࣻ, ળҕ֙ب, ߓۆ׮ іࣻ ١١.. • ݽ؛੉ ׮೦ध੄ ഋకо ؽ • Feature scaling, feature ઑ೤, ઺ࠂ feature ઁѢ ١ ݆਷ ੘স੉ ೙ਃೞѱ ؽ
36. Logistic Regression
37. Classification: Logistic Regression • োࣘ੸ੋ чਸ ৘ஏೞחѪ੉ ইצ, ࠛোࣘ੸ੋ чਸ ৘ஏ - झಅੋ૑ ইצ૑ - ঐੋ૑ ইצ૑ - ੉ ࢎ૓੉ Ҋন੉ੋ૑, ъই૑ੋ૑ Tumor Size Tumor Size (Yes) 1 (No) 0
38. Classification: Logistic Regression • Sigmoid (Logistic function) ਸ ࢎਊೞৈ և਷ ߧਤ੄ чਸ, 0~1 ࢎ੉੄ чਵ۽ ߸ജ • ݅ড 0.5ܳ ӝળਵ۽ ೠ׮ݶ, 0.5 ੉࢚:1, 0.5 ޷݅: 0
39. Logistic Cost Function • y = 0 ੌٸ৬ y = 1 ੌٸо ׮ܴ • Gradient Descentܳ ഝਊೞח ߑߨ਷ Linear Regressionҗ زੌ!
40. Regularization • Overfitting (җ୭੸ച) ޙઁ - അपࣁ҅੄ पઁ ݽ؛ࠁ׮, ೟णदఅ ݽ؛੉ ࠂ੟بо ؊ ֫ইઉࢲ ઱য૓ ೟णࣇী݅ җبೞѱ ୭੸ചػ ࢚ട - training set ੉ ইצ ࢜۽਍ ޙઁܳ ઁ؀۽ ৘ஏೡ ࣻ হѱ ػ׮ • Feature ੄ іࣻܳ ઴੉Ѣա, Regularizationਵ۽ ೧Ѿ • Regularization - Cost functionী ੄ب੸ਵ۽ ࢚ࣻ೦ਸ ୶оदெ ೟णਸ ߑ೧ೣ - җبೞѱ ೟णغחѪਸ ߑ૑ೞח ബҗ
41. Neural Network
42. Neural Network • ੋр੄ ֱ ҳઑܳ ٮۄೠ ҳઑ • Dendrite: ׮ܲ ׏۠ਵ۽ࠗఠ नഐܳ ߉਺ • Cell body: ݽٚ ੑ۱ਸ ઙ೤ೣ • Axon: ੑ۱੄ ೤੉ ੐҅੼ਸ ֈਵݶ, ׮ܲ ׏۠ਵ۽ नഐܳ ࠁն • Synapses: ׮ܲ ׏۠җ োѾؽ. োѾ ъبী ٮۄ नഐ੄ ࣁӝо ߄Պ - नഐо ъ೧૑Ѣա, ড೧૑חѪ = ೟ण
43. Neural Network • ੋр੄ ֱ ҳઑܳ ٮۄೠ ҳઑ • Dendrite: ׮ܲ ׏۠ਵ۽ࠗఠ नഐܳ ߉਺ • Cell body: ݽٚ ੑ۱ਸ ઙ೤ೣ • Axon: ੑ۱੄ ೤੉ ੐҅੼ਸ ֈਵݶ, ׮ܲ ׏۠ਵ۽ नഐܳ ࠁն • Synapses: ׮ܲ ׏۠җ োѾؽ. োѾ ъبী ٮۄ नഐ੄ ࣁӝо ߄Պ - नഐо ъ೧૑Ѣա, ড೧૑חѪ = ೟ण
44. Neural Network
45. Neural Network
46. Neural Network Playground • http://playground.tensorflow.org/ • 0 hidden layer, 2 neurons (Logistic Regression) - ױࣽೠ 1ର ഋక੄ ؘ੉ఠ߆ী ೟णदః૑ ޅೣ • 2 hidden layer, 2 neurons - Ѿҗо যڌѱ ׮ܲо? - ৈ۞க ऺইب Ѿҗо ࠺तೣ • neuronіࣻܳ טܻݶ ખ؊ ࠂ੟ೠ ഋకܳ ಴അೡ ࣻ ੓׮ • Featureܳ טܻݶ? • Learning rate੉ ௼ݶ? • Deep ೠ ֎౟ਖীࢲ activation function? • Regularization
47. Neural Network - Cost Function Layer 1 Layer 2 Layer 3 Layer 4 Logistic regression: Neural network: • Logistic Regression੄ ഛ੢౸!
48. Neural Network - Back Propagation • Forward propagation Layer 1 Layer 2 Layer 3 Layer 4
49. Neural Network - Back Propagation • Back propagation Layer 1 Layer 2 Layer 3 Layer 4 Intuition: “error” of node in layer . For each output unit (layer L = 4)
50. Neural Network - ੿ܻ • ֎౟ਕ௼ ҳઑܳ ੿ೠ׮ - Input feature੄ іࣻ, Output class੄ іࣻ, hidden layer੄ іࣻ ߂ ରਗ • weightܳ ےؒೞѱ ୡӝച • forward propagation, cost function, back propagation ਸ ҳ അ • ݽٚ ؘ੉ఠٜী ؀ೞৈ, back propagationਸ ా೧ ҳೠ gradient ٜਸ ݽ؛ী ੸ਊ • TensorFlowо ׮ ঌইࢲ ೧઻ਃ
51. (੤޷۽) ੋр੄ ֱ vs. ׏ۡ֎౟ਕ௼ • ੋр੄ ֱ - ড ୌর ѐ੄ ׏۠, ࣻߔ~ࣻୌઑ ѐ੄ दշझ۽ ੉ܖয૗ - ੋрҗ زޛ੄ ֱח ׮׮੊ࢶ • ׏ۡ֎౟ਕ௼ - ੋҕ੸ੋ ׏ۡ֎౟ਕ௼੄ ࢎ੉ૉח ࢚؀੸ਵ۽ ੘਺ - ࢎ੉ૉо ௿ࣻ۾ ೟ण੉ য۰ਕ૗ • AlphaGo - ৈ۞ѐ੄ ׏ۡ ֎౟ਕ௼੄ ઑ೤. (13 layer੄ Convolutional neural network ١) - Input layer: 48ѐ੄ 19x19 ߄ق౸ = 17328 ׏۠ - ࣻߔ݅ѐ੄ ׏۠
52. ঱য ݽ؛җ RNN (Recurrent Neural Networks)
53. RNN? • !?!?!? • RNN੉ ޤӡې, ੉۠Ѫਸ ೡࣻ੓חо • RNNਸ Ҿӓ੄ ੋҕ न҃ ݎ ҳઑۄҊ ઱੢ೞח ࢎ ۈٜب ੓਺
54. RNN (Recurrent Neural Networks) • ݆਷ NLP ੘সী ই઱ જ਷ ࢿמਸ ղષ • ѐ֛੉ য۵ӝب ೞҊ ઁ؀۽ ࢸݺೞҊ ੓ח ੗ܐо হ਺ - ઁ؀۽ ੉೧ೞח ࢎۈب ੜ হחѪ э਺ • য়טب ઁ؀۽ ࢸݺೡ ࣻ ੓ਸ૑ ഛप൤ ݽܰѷ਺
55. ঱য ݽ؛ (Language Model) • োࣘػ ױযٜ੄ ١੢ഛܫਸ ৘ஏೞח ݽ؛ • ߓо Ҋ౵ࢲ աח ߏਸ ____ - ___ী ٜযт ױযח? - ݡ঻׮, ݡח׮, ૑঻׮,… - p(ݡ঻׮) = 0.25, p(ݡח׮) = 0.15, p(૑঻׮) = 0.05, .. - p(੗زର) = 0.000001 • ޙ੢੄ ࣽࢲա ױযࢶఖ ١ਸ ৘ஏೡ ࣻ ੓ӝী, ੗زߣ৉ ١ ݆਷ NLP Taskী ਬਊೞ׮
56. ঱য ݽ؛ (Language Model) • ഛܫ੸ ঱য ݽ؛ਸ ݅٘۰ݶ, ੉੹ী աয়ח ױযܳ ݆੉ ଵઑೡࣻ۾ ੿ഛೞ׮! - 2-gram, 3-gram, 4-gram, .. n-gram • ߓо Ҋ౵ࢲ աח ߏਸ ____ - ߏਸ ____ <-> ߓо Ҋ౵ࢲ աח ߏਸ ____ • n-gramਸ טܾࣻ۾ ݫݽܻо ষ୒աѱ ೙ਃೞ׮! - ੉ ޙઁܳ ೧Ѿ೧઱חѪ੉ RNN(Recurrent Neural Networks)
57. RNN (Recurrent Neural Networks) • दрী ٮۄ (഑਷ ؘ੉ఠ੄ ૓೯ী ٮۄ) ҅ࣘ সؘ੉౟غח ࢚కчਸ р૒ೞҊ ੓ח ֎౟ਕ௼ • ױࣽೠ ࢚కо ইפۄ, ೟णਸ ૑ࣘೡࣻ۾ ੑ۱ਸ ୊ܻೞח ҭ੢൤ ࠂ੟ೠ ۽૒੉ ֣ ইٜѱؽ • ࢜۽ աৢ ױযܳ ৘ஏೞחؘী ੉੹੄ ݽٚ ױযٜ੄ ੿ࠁܳ ଵઑೞѱ ؽ
58. RNN (Recurrent Neural Networks) • ߓо Ҋ౵ࢲ աח ߏਸ ____ - 4ѐ੄ ױযܳ ଵઑ - RNNীࢲח 4 layer neural network੉ ࢤӣ
59. RNN (Recurrent Neural Networks) • RNNਸ ೟णदఃחѪ਷ য۵׮! • Vanishing Gradient ޙઁ - ցޖ য়ې੹੄ ੿ࠁө૑ ଵઑೞ۰׮ࠁפ, য٣ࢲ ੜޅ೮঻ח૑ ೟णೞӝ য۵ѱ ؽ • Exploding Gradient ޙઁ • ReLU / Clipping ١ ৈ۞о૑ ప௼ץਵ۽ ӓࠂ!
60. RNN (Recurrent Neural Networks) • ؊ ੗ࣁೠ ೟ण੗ܐ - http://www.wildml.com/2015/09/recurrent-neural-networks- tutorial-part-1-introduction-to-rnns/
61. LSTM (Long Short-Term Memory) • RNNҗ ਬࢎೠ ҳઑ੄ ֎౟ਕ௼ • RNN਷ ߈ࠂ ҳઑ۽ ੋ೧ ӝর۱ਸ(memory) о૑ѱ غ঻਺ • दр୷ਵ۽ ಽযࠁݶ, ৈ۞ѐ੄ ֎౟ਕ௼о ੓Ҋ п੗о ੗न੄ ׮਺ ֎౟ਕ௼ী ݫࣁ૑ܳ ࠁղחѪҗ э਷ ਗܻ • োࣘػ ؘ੉ఠ৬ Ө਷ োҙ - ঱য ݽ؛, ߣ৉, ਺ࢿੋध ١١ RNNҗ ੉ܳ दр୷ਵ۽ ಽ੉ೠ Ӓܿ
62. LSTM (Long Short-Term Memory) • RNN਷ ୭Ӕ ݻѐ ױয ੿ب੄ ӝর۱ ੿ب݅ਸ о૗ • LSTM: ؊਌ ӟ ӝর۱ਸ о૑ӝ ਤೠ ҳઑ
63. LSTM (Long Short-Term Memory) • Cell State: п ױ҅݃׮ ই઱ ੸਷ ࣻ੿ਸ Ѣ஖ݴ ੿ࠁܳ য়ۖزউ ࠁઓೠ׮.
64. LSTM (Long Short-Term Memory) • Forget gate: ੉੹ cell state઺ী ࠛ೙ਃೠ ੿ࠁܳ ઁѢ • Input gate: ࢜۽਍ ੿ࠁܳ cell stateী ӝর
65. LSTM (Long Short-Term Memory) • Forget gate৬ input gate੄ ઑ೤ਵ۽ cell state ܳ সؘ੉౟ • Output: cell state੄ ੌࠗ࠙ਵ۽ чਸ ୹۱
66. GRU (Gated Recurrent Unit) • LSTMҗ Ѣ੄ ਬࢎೠ ҳઑ, ؊ рױೠ ҳഅ - Forget gate ৬ input gate ܳ ೞա۽ ೤ஜ
67. Attention Mechanism • ੋр੄ ੋ૑җ੿ਸ ࢤп೧ࠁݶ, ݽٚ ࣽрী ݽٚࠗ࠙ী न҃ਸ ॳח Ѫ੉ ইפۄ, ౠ੿ ࣽрী ౠ੿ ࢎޛ੉ա ઱ઁী ૘઺ೞח ݽणਸ ࠁ ੐ • RNNэ਷ ݽ؛੄ ӡ੉о ӡয૕ࣻ۾ ೞա੄ Hidden Stateী ݽٚ ղਊਸ ӝরೞӝо য۰ਕ૗ • Hidden stateٜਸ ੷੢ೞҊ ੓׮о, ੉੹੄ stateٜਸ ഝਊ
68. ؊ ࠂ੟ೠ ֎౟ਕ௼
69. Dynamic Memory Networks • যڃ Ӗਸ ੍Ҋ, Ӓ Ӗী ؀ೠ ޛ਺ী ׹ਸ ೞחѪ - ࣻמद೷ ঱য৔৉җ ࠺तೣ - औ૑ ঋ਷ җઁ੉׮!
70. Dynamic Memory Networks • ӝמ߹۽ RNNਸ ઑ೤ • End-to-End ೟ण • ֤ޙਸ ੍੗! - https://arxiv.org/abs/1506.07285
71. Dynamic Memory Networks • Input Module - RNN (GRU), п ޙ੢ٜ੄ hidden stateܳ ੷੢ • Question Module - рױೠ RNN (GRU) • Episodic Memory Module - ੉ঠӝী ؀ೠ ੿ࠁܳ ҙ੢ೞח ݽٕ - ৈ۞ க੄ RNN (GRU) ۽ ҳࢿ
72. Dynamic Memory Networks • ׮ܲ ఋੑ੄ Input Moduleਸ ࠢ੉חѪب оמ!
73. Dynamic Memory Networks • ׮ܲ ఋੑ੄ Input Moduleਸ ࠢ੉חѪب оמ!
74. Dynamic Memory Networks
75. Dynamic Memory Networks
76. Dynamic Memory Networks
77. Dynamic Memory Networks • Attention + Memoryܳ ా೧ ૕ޙী ׹ೞח מ۱ਸ ഛ੢ • ӝמ߹۽ RNNਵ۽ ੉ܖয૓ ߹ب੄ Module, ੑ۱ ഋకо ׮ܲ ݽ ٕ۽ ߸҃ೞחѪب оמೣ • ੉޷૑ ੑ۱ ݽٕҗ Ѿ೤غݶ ֥ۄ਍ Ѿҗܳ ࠁৈષ
78. NMT (Neural Machine Translation) • ࠂ੟ೠ ֎౟ਕ௼ҳઑܳ ഝਊೠ ੗زߣ৉ ֎౟ਕ௼
79. Google NMT • https://translate.google.com • https://research.googleblog.com/2016/09/a-neural-network-for- machine.html
80. ٩۞׬ ੸ਊী ؀ೠ ࢤп • ٩۞׬ = ই੉٣য + ؘ੉ఠ + ݽ؛ ೟ण - ݽ؛ ೟ण - ੼੼ ੷۴೧૗ - ই੉٣য - ੼੼ ؊ ݆਷ ࢎۈٜ੉ ઙࢎೞݶࢲ ই੉٣যب ݆ই૕ Ѫ - ؘ੉ఠ - ݠन۞׬, ٩۞׬ द؀ীࢲ о੢ ൞ࣗೠ ੗ਗ • ؀ӏݽ ؘ੉ఠܳ ഝਊೠ ݠन۞׬, ؊ ࠂ੟ೠ ઑ೤੄ ݽ؛, ࠙ঠ߹ ௼ ۽झ ݽ؛ ١ ই૒ ߊ੹ оמࢿ੉ ޖҾޖ૓ೣ • ҕࠗೞҊ োҳ೤द׮!
81. More..? • Join VCNC and join #study_ml ੷൞ח ঱ઁա ࠺౟ਦ ࢲ࠺झܳ ೣԋ ٜ݅ݴ ӝࣿ੸ੋ ޙઁܳ ೣԋ ಽযաт מ۱੓ח ѐߊ੗ܳ ݽदҊ ੓णפ׮. ঱ઁٚ ࠗ׸হ੉ jobs@vcnc.co.kr۽ ੉ݫੌਸ ઱दӝ ߄ۉפ׮!
82. Thank you!