Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
城ヶ崎美嘉で学ぶRNNLM
Search
Kento Nozawa
June 05, 2016
Programming
2
3k
城ヶ崎美嘉で学ぶRNNLM
オタク機械学習勉強会#0 のLT
Kento Nozawa
June 05, 2016
Tweet
Share
More Decks by Kento Nozawa
See All by Kento Nozawa
Analysis on Negative Sample Size in Contrastive Unsupervised Representation Learning
nzw0301
0
150
[IJCAI-ECAI 2022] Evaluation Methods for Representation Learning: A Survey
nzw0301
0
600
[NeurIPS Japan meetup 2021 talk] Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
190
[IBIS2021] 対照的自己教師付き表現学習おける負例数の解析
nzw0301
0
180
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
480
Introduction of PAC-Bayes and its Application for Contrastive Unsupervised Representation Learning
nzw0301
2
800
NLP Tutorial; word representation learning
nzw0301
0
210
Analyzing Centralities of Embedded Nodes
nzw0301
0
160
Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
nzw0301
2
1.1k
Other Decks in Programming
See All in Programming
TVer iOSチームの共通認識の作り方 - Findy Job LT iOSアプリ開発の裏側 開発組織が向き合う課題とこれから
techtver
PRO
0
710
Perlで痩せる
yuukis
1
660
マテリアルって何者?RealityKitで扱うマテリアル入門
nao_randd
0
140
人には人それぞれのサービス層がある
shimabox
3
470
JSAI2025 RecSysChallenge2024 優勝報告
unonao
1
380
バリデーションライブラリ徹底比較
nayuta999999
1
450
"使いづらい" をリバースエンジニアリングする UI の読み解き方
rebase_engineering
0
110
抽象データ型について学んだ
ryounasso
0
210
從零到一:搭建你的第一個 Observability 平台
blueswen
0
220
TypeScript LSP の今までとこれから
quramy
0
130
イベントストーミングから始めるドメイン駆動設計
jgeem
3
450
MLOps Japan 勉強会 #52 - 特徴量を言語を越えて一貫して管理する, 『特徴量ドリブン』な MLOps の実現への試み
taniiicom
2
570
Featured
See All Featured
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.5k
How to train your dragon (web standard)
notwaldorf
92
6k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Site-Speed That Sticks
csswizardry
7
590
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Designing Experiences People Love
moore
142
24k
Scaling GitHub
holman
459
140k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Facilitating Awesome Meetings
lara
54
6.4k
Practical Orchestrator
shlominoach
188
11k
Transcript
ϲ࡚ඒՅ Λը૾ݕࡧ͓ͯͪ͠Լ͍͞
ϲ࡚ඒՅͰֶͿ RNNLM 2016/6/5 ΦλΫػցֶशษڧձ #0 @nzw0301
Ϟνϕʔγϣϯ ϲ࡚ඒՅͷηϦϑੜ
Recurrent Neural Network Language Model • ηϦϑੜ: લ·Ͱͷ୯ޠ͔Β࣍ͷ1୯ޠΛ༧ଌ͠ଓ͚Δ • ྫɿΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒
• ୯ޠׂ: <BOS> ΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒&04 • ֶश: Q ΊΔΊΔc#04 ͱ͔ Q ᣦՅc<BOS>, ΊΔΊΔ ʜ
RNNLMͷߏ ޠኮV࣍ݩͷϕΫτϧ softmax ؔ 1ͭલͷதؒͷϕΫτϧ RNNͷ༝ԑ h࣍ݩͷதؒ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿೖྗ w #04ͷPOFPG,දݱΛೖྗ w ࣍ݩͰີͳϕΫτϧʹม <BOS> ΊΔΊΔ 0 B
B B B B @ 0 1 0 . . . 0 1 C C C C C A
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿதؒ • ີͳϕΫτϧΛதؒʹ͢ • ଟύʔηϓτϩϯͱಉ͡ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛ͢ • ݱࡏͷதؒͷΛอ࣋ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ΊΔΊΔ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽ <BOS>
ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿೖྗ ૄΊΔΊΔϕΫτϧΛೖྗ͠ɼີͳΊΔΊΔϕΫτϧʹม p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ 0 B B
B B B B B B B B @ 0 . . . 0 1 0 . . . 0 1 C C C C C C C C C C A
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿதؒ ີͳΊΔΊΔϕΫτϧͱલʹܭࢉͨ͠தؒͷϕΫτϧΛதؒ p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛͯ͠ɼݱࡏͷதؒͷϕΫτϧΛอ࣋ p(ʜ|<BOS>, ΊΔΊΔ)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ʜ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽
ʜ ΊΔΊΔ
࣮ݧ
࣮ݧ֓ཁ • SCRNΛ༻ • LSTM GRU ΛΘͳ͍ • Keras
Ͱ࣮ • લॲཧ • ܗଶૉղੳͤͣʹจࣈ୯ҐͰֶश • /。|★|?|!|♪/ ͰηϦϑΛׂ • 900ηϦϑ (Վࢺ) Λ༻ • ϞόϚε • σϨες • TOKIMEKIΤεΧϨʔτ
݁Ռ
10epochޙɿϓϩσϡʔαʔͷҰ෦͕ͱΕͯΔ ϓϩσϩσϡʔͯͳͪʙʹෲΞλ γ΄ϡʔαʔΒతͳʔɺͨ͜ͳ
40epochޙɿΪϟϧޠʁ ϓϩσϡʔαʔʹ͍ͪΌΜɺ ݟ͘ͳ͍ʔ͘ͱԿߴͩ͠ʔͬ̇
80epochޙɿݺΕͨؾ͕ͨ͠ ϓϩσϡʔαʔ!
“<BOS> ϓ” ͔Β࠷ਪఆɿϧʔϓ ϓϩσϡʔαʔɺΞλγͷ͜ͱ͔Βɺ ϓϩσϡʔαʔɺΞλγͷ͜ͱ
ϥϯμϜʹηϦϑੜ
ॴײ • ηϦϑΛͲ͜ͰΔ͖͔ • ྫɿ͝Μʹ͢Δ?͓෩࿊ʹ͢Δ?…͜ΕͪΐͬͱϕλͬΆ͍ͳ͊ • ? Ͱ۠Δ͖͔൱͔ • …લޙͲͬͪͰ۠Δ͔൱͔ʁͦΕͱͳ͘͢ʁ
• ήʔϜը໘ͷͨΊ͔1ηϦϑܥྻ͕΄΅Ұఆʢֶͼʣ
ࢀߟจݙͳͲ • http://keras.io/ • DLͷϥΠϒϥϦ • ָ͍͢͝ʹॻ͚Δ • Mikolov at.el.
Recurrent neural network based language model. 2010. • RNNͷը૾͜ͷจͷͷΛ༻ • Mikolov at.el Learning Longer Memory in Recurrent Neural Networks. 2014. • ࠓճ༻ͨ͠Ϟσϧ