Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
城ヶ崎美嘉で学ぶRNNLM
Search
Kento Nozawa
June 05, 2016
Programming
2
2.9k
城ヶ崎美嘉で学ぶRNNLM
オタク機械学習勉強会#0 のLT
Kento Nozawa
June 05, 2016
Tweet
Share
More Decks by Kento Nozawa
See All by Kento Nozawa
Analysis on Negative Sample Size in Contrastive Unsupervised Representation Learning
nzw0301
0
97
[IJCAI-ECAI 2022] Evaluation Methods for Representation Learning: A Survey
nzw0301
0
520
[NeurIPS Japan meetup 2021 talk] Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
140
[IBIS2021] 対照的自己教師付き表現学習おける負例数の解析
nzw0301
0
130
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
410
Introduction of PAC-Bayes and its Application for Contrastive Unsupervised Representation Learning
nzw0301
2
740
NLP Tutorial; word representation learning
nzw0301
0
160
Analyzing Centralities of Embedded Nodes
nzw0301
0
120
Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
nzw0301
2
1k
Other Decks in Programming
See All in Programming
REXML改善のその後
naitoh
0
190
Pythonで改めて考える「クラス(class)」の使いどころ
os1ma
1
510
令和トラベルにおけるLLM活用事例:社内ツール開発から得た学びと実践
ippo012
0
210
Kotlin 2.0が与えるAndroid開発の進化
masayukisuda
1
410
いまから追い上げる、Jetpack Compose トレーニング
nyafunta9858
0
590
Rubyとクリエイティブコーディングの輪の広がり / The Growing Circle of Ruby and Creative Coding
chobishiba
1
270
エンジニア1年目で複雑なコードの改善に取り組んだ話
mtnmr
3
2k
全部見せます! クラシルリワードのSwiftTesting移行プロジェクト
uetyo
0
210
『ドメイン駆動設計をはじめよう』中核の業務領域
masuda220
PRO
5
1k
開発を加速する共有Swift Package実践
elmetal
PRO
0
420
Jakarta EE meets AI
ivargrimstad
0
390
Understand the mechanism! Let's do screenshots tests of Compose Previews with various variations / 仕組みから理解する!Composeプレビューを様々なバリエーションでスクリーンショットテストしよう
sumio
3
790
Featured
See All Featured
How GitHub (no longer) Works
holman
310
140k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
41
6.5k
The Language of Interfaces
destraynor
153
23k
Six Lessons from altMBA
skipperchong
26
3.4k
Docker and Python
trallard
39
3k
Done Done
chrislema
180
16k
Clear Off the Table
cherdarchuk
91
320k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
123
18k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
28
1.6k
For a Future-Friendly Web
brad_frost
174
9.3k
The Invisible Side of Design
smashingmag
296
50k
It's Worth the Effort
3n
182
27k
Transcript
ϲ࡚ඒՅ Λը૾ݕࡧ͓ͯͪ͠Լ͍͞
ϲ࡚ඒՅͰֶͿ RNNLM 2016/6/5 ΦλΫػցֶशษڧձ #0 @nzw0301
Ϟνϕʔγϣϯ ϲ࡚ඒՅͷηϦϑੜ
Recurrent Neural Network Language Model • ηϦϑੜ: લ·Ͱͷ୯ޠ͔Β࣍ͷ1୯ޠΛ༧ଌ͠ଓ͚Δ • ྫɿΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒
• ୯ޠׂ: <BOS> ΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒&04 • ֶश: Q ΊΔΊΔc#04 ͱ͔ Q ᣦՅc<BOS>, ΊΔΊΔ ʜ
RNNLMͷߏ ޠኮV࣍ݩͷϕΫτϧ softmax ؔ 1ͭલͷதؒͷϕΫτϧ RNNͷ༝ԑ h࣍ݩͷதؒ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿೖྗ w #04ͷPOFPG,දݱΛೖྗ w ࣍ݩͰີͳϕΫτϧʹม <BOS> ΊΔΊΔ 0 B
B B B B @ 0 1 0 . . . 0 1 C C C C C A
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿதؒ • ີͳϕΫτϧΛதؒʹ͢ • ଟύʔηϓτϩϯͱಉ͡ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛ͢ • ݱࡏͷதؒͷΛอ࣋ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ΊΔΊΔ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽ <BOS>
ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿೖྗ ૄΊΔΊΔϕΫτϧΛೖྗ͠ɼີͳΊΔΊΔϕΫτϧʹม p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ 0 B B
B B B B B B B B @ 0 . . . 0 1 0 . . . 0 1 C C C C C C C C C C A
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿதؒ ີͳΊΔΊΔϕΫτϧͱલʹܭࢉͨ͠தؒͷϕΫτϧΛதؒ p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛͯ͠ɼݱࡏͷதؒͷϕΫτϧΛอ࣋ p(ʜ|<BOS>, ΊΔΊΔ)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ʜ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽
ʜ ΊΔΊΔ
࣮ݧ
࣮ݧ֓ཁ • SCRNΛ༻ • LSTM GRU ΛΘͳ͍ • Keras
Ͱ࣮ • લॲཧ • ܗଶૉղੳͤͣʹจࣈ୯ҐͰֶश • /。|★|?|!|♪/ ͰηϦϑΛׂ • 900ηϦϑ (Վࢺ) Λ༻ • ϞόϚε • σϨες • TOKIMEKIΤεΧϨʔτ
݁Ռ
10epochޙɿϓϩσϡʔαʔͷҰ෦͕ͱΕͯΔ ϓϩσϩσϡʔͯͳͪʙʹෲΞλ γ΄ϡʔαʔΒతͳʔɺͨ͜ͳ
40epochޙɿΪϟϧޠʁ ϓϩσϡʔαʔʹ͍ͪΌΜɺ ݟ͘ͳ͍ʔ͘ͱԿߴͩ͠ʔͬ̇
80epochޙɿݺΕͨؾ͕ͨ͠ ϓϩσϡʔαʔ!
“<BOS> ϓ” ͔Β࠷ਪఆɿϧʔϓ ϓϩσϡʔαʔɺΞλγͷ͜ͱ͔Βɺ ϓϩσϡʔαʔɺΞλγͷ͜ͱ
ϥϯμϜʹηϦϑੜ
ॴײ • ηϦϑΛͲ͜ͰΔ͖͔ • ྫɿ͝Μʹ͢Δ?͓෩࿊ʹ͢Δ?…͜ΕͪΐͬͱϕλͬΆ͍ͳ͊ • ? Ͱ۠Δ͖͔൱͔ • …લޙͲͬͪͰ۠Δ͔൱͔ʁͦΕͱͳ͘͢ʁ
• ήʔϜը໘ͷͨΊ͔1ηϦϑܥྻ͕΄΅Ұఆʢֶͼʣ
ࢀߟจݙͳͲ • http://keras.io/ • DLͷϥΠϒϥϦ • ָ͍͢͝ʹॻ͚Δ • Mikolov at.el.
Recurrent neural network based language model. 2010. • RNNͷը૾͜ͷจͷͷΛ༻ • Mikolov at.el Learning Longer Memory in Recurrent Neural Networks. 2014. • ࠓճ༻ͨ͠Ϟσϧ