Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
城ヶ崎美嘉で学ぶRNNLM
Search
Kento Nozawa
June 05, 2016
Programming
2
3k
城ヶ崎美嘉で学ぶRNNLM
オタク機械学習勉強会#0 のLT
Kento Nozawa
June 05, 2016
Tweet
Share
More Decks by Kento Nozawa
See All by Kento Nozawa
Analysis on Negative Sample Size in Contrastive Unsupervised Representation Learning
nzw0301
0
140
[IJCAI-ECAI 2022] Evaluation Methods for Representation Learning: A Survey
nzw0301
0
590
[NeurIPS Japan meetup 2021 talk] Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
180
[IBIS2021] 対照的自己教師付き表現学習おける負例数の解析
nzw0301
0
170
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
470
Introduction of PAC-Bayes and its Application for Contrastive Unsupervised Representation Learning
nzw0301
2
790
NLP Tutorial; word representation learning
nzw0301
0
200
Analyzing Centralities of Embedded Nodes
nzw0301
0
150
Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
nzw0301
2
1.1k
Other Decks in Programming
See All in Programming
Exit 8 for SwiftUI
ojun9
0
140
Module Boundaries and Architecture with Forensic Analysis @NxSummit Amsterdam 2025
manfredsteyer
PRO
0
100
Contribute to Comunities | React Tokyo Meetup #4 LT
sasagar
0
500
State of Namespace
tagomoris
4
1.8k
DataStoreをテストする
mkeeda
0
290
設計の本質:コード、システム、そして組織へ / The Essence of Design: To Code, Systems, and Organizations
nrslib
4
620
プロダクト横断分析に役立つ、事前集計しないサマリーテーブル設計
hanon52_
2
450
Unlock the Potential of Swift Code Generation
rockname
0
260
Vibe Coding の話をしよう
schroneko
9
2.5k
海外のアプリで見かけたかっこいいTransitionを真似てみる
shogotakasaki
1
170
新しいPHP拡張モジュールインストール方法「PHP Installer for Extensions (PIE)」を使ってみよう!
cocoeyes02
0
400
プロフェッショナルとしての成長「問題の深掘り」が導く真のスキルアップ / issue-analysis-and-skill-up
minodriven
7
1.5k
Featured
See All Featured
Code Reviewing Like a Champion
maltzj
522
40k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
How to Think Like a Performance Engineer
csswizardry
23
1.5k
Making the Leap to Tech Lead
cromwellryan
133
9.2k
Scaling GitHub
holman
459
140k
Into the Great Unknown - MozCon
thekraken
37
1.7k
Building a Modern Day E-commerce SEO Strategy
aleyda
40
7.2k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
13
680
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Designing Experiences People Love
moore
141
24k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Transcript
ϲ࡚ඒՅ Λը૾ݕࡧ͓ͯͪ͠Լ͍͞
ϲ࡚ඒՅͰֶͿ RNNLM 2016/6/5 ΦλΫػցֶशษڧձ #0 @nzw0301
Ϟνϕʔγϣϯ ϲ࡚ඒՅͷηϦϑੜ
Recurrent Neural Network Language Model • ηϦϑੜ: લ·Ͱͷ୯ޠ͔Β࣍ͷ1୯ޠΛ༧ଌ͠ଓ͚Δ • ྫɿΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒
• ୯ޠׂ: <BOS> ΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒&04 • ֶश: Q ΊΔΊΔc#04 ͱ͔ Q ᣦՅc<BOS>, ΊΔΊΔ ʜ
RNNLMͷߏ ޠኮV࣍ݩͷϕΫτϧ softmax ؔ 1ͭલͷதؒͷϕΫτϧ RNNͷ༝ԑ h࣍ݩͷதؒ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿೖྗ w #04ͷPOFPG,දݱΛೖྗ w ࣍ݩͰີͳϕΫτϧʹม <BOS> ΊΔΊΔ 0 B
B B B B @ 0 1 0 . . . 0 1 C C C C C A
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿதؒ • ີͳϕΫτϧΛதؒʹ͢ • ଟύʔηϓτϩϯͱಉ͡ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛ͢ • ݱࡏͷதؒͷΛอ࣋ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ΊΔΊΔ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽ <BOS>
ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿೖྗ ૄΊΔΊΔϕΫτϧΛೖྗ͠ɼີͳΊΔΊΔϕΫτϧʹม p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ 0 B B
B B B B B B B B @ 0 . . . 0 1 0 . . . 0 1 C C C C C C C C C C A
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿதؒ ີͳΊΔΊΔϕΫτϧͱલʹܭࢉͨ͠தؒͷϕΫτϧΛதؒ p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛͯ͠ɼݱࡏͷதؒͷϕΫτϧΛอ࣋ p(ʜ|<BOS>, ΊΔΊΔ)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ʜ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽
ʜ ΊΔΊΔ
࣮ݧ
࣮ݧ֓ཁ • SCRNΛ༻ • LSTM GRU ΛΘͳ͍ • Keras
Ͱ࣮ • લॲཧ • ܗଶૉղੳͤͣʹจࣈ୯ҐͰֶश • /。|★|?|!|♪/ ͰηϦϑΛׂ • 900ηϦϑ (Վࢺ) Λ༻ • ϞόϚε • σϨες • TOKIMEKIΤεΧϨʔτ
݁Ռ
10epochޙɿϓϩσϡʔαʔͷҰ෦͕ͱΕͯΔ ϓϩσϩσϡʔͯͳͪʙʹෲΞλ γ΄ϡʔαʔΒతͳʔɺͨ͜ͳ
40epochޙɿΪϟϧޠʁ ϓϩσϡʔαʔʹ͍ͪΌΜɺ ݟ͘ͳ͍ʔ͘ͱԿߴͩ͠ʔͬ̇
80epochޙɿݺΕͨؾ͕ͨ͠ ϓϩσϡʔαʔ!
“<BOS> ϓ” ͔Β࠷ਪఆɿϧʔϓ ϓϩσϡʔαʔɺΞλγͷ͜ͱ͔Βɺ ϓϩσϡʔαʔɺΞλγͷ͜ͱ
ϥϯμϜʹηϦϑੜ
ॴײ • ηϦϑΛͲ͜ͰΔ͖͔ • ྫɿ͝Μʹ͢Δ?͓෩࿊ʹ͢Δ?…͜ΕͪΐͬͱϕλͬΆ͍ͳ͊ • ? Ͱ۠Δ͖͔൱͔ • …લޙͲͬͪͰ۠Δ͔൱͔ʁͦΕͱͳ͘͢ʁ
• ήʔϜը໘ͷͨΊ͔1ηϦϑܥྻ͕΄΅Ұఆʢֶͼʣ
ࢀߟจݙͳͲ • http://keras.io/ • DLͷϥΠϒϥϦ • ָ͍͢͝ʹॻ͚Δ • Mikolov at.el.
Recurrent neural network based language model. 2010. • RNNͷը૾͜ͷจͷͷΛ༻ • Mikolov at.el Learning Longer Memory in Recurrent Neural Networks. 2014. • ࠓճ༻ͨ͠Ϟσϧ