Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Problems of Neural Networks and its solutions
Search
izuna385
June 21, 2018
Technology
0
150
Problems of Neural Networks and its solutions
Residual Connections とBatch Normalizationがメイン
izuna385
June 21, 2018
Tweet
Share
More Decks by izuna385
See All by izuna385
jel: japanese entity linker
izuna385
0
400
Firebase-React-App
izuna385
0
250
React+FastAPIを用いた簡単なWebアプリ作製
izuna385
0
1.7k
UseCase of Entity Linking
izuna385
0
590
Unofficial slides: From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains (ACL 2020)
izuna385
1
670
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
izuna385
0
880
Zero-shot Entity Linking with Dense Entity Retrieval (Unofficial slides) and Entity Linking future directions
izuna385
3
1.1k
Entity representation with relational attention
izuna385
0
86
Zero-Shot Entity Linking by Reading Entity Descriptions
izuna385
0
570
Other Decks in Technology
See All in Technology
綺麗なデータマートをつくろう_データ整備を前向きに考える会 / Let's create clean data mart
brainpadpr
2
220
pprof vs runtime/trace (FlightRecorder)
task4233
0
170
How to achieve interoperable digital identity across Asian countries
fujie
0
120
o11yで育てる、強い内製開発組織
_awache
3
120
小学4年生夏休みの自由研究「ぼくと Copilot エージェント」
taichinakamura
0
400
組織観点からIAM Identity CenterとIAMの設計を考える
nrinetcom
PRO
1
180
Where will it converge?
ibknadedeji
0
190
Exadata Database Service on Dedicated Infrastructure(ExaDB-D) UI スクリーン・キャプチャ集
oracle4engineer
PRO
2
5.4k
自作LLM Native GORM Pluginで実現する AI Agentバックテスト基盤構築
po3rin
2
260
Optuna DashboardにおけるPLaMo2連携機能の紹介 / PFN LLM セミナー
pfn
PRO
1
890
M5製品で作るポン置きセルラー対応カメラ
sayacom
0
160
AI時代だからこそ考える、僕らが本当につくりたいスクラムチーム / A Scrum Team we really want to create in this AI era
takaking22
6
3.6k
Featured
See All Featured
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
127
53k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
The Pragmatic Product Professional
lauravandoore
36
6.9k
Navigating Team Friction
lara
189
15k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
132
19k
Making the Leap to Tech Lead
cromwellryan
135
9.5k
GraphQLとの向き合い方2022年版
quramy
49
14k
How STYLIGHT went responsive
nonsquared
100
5.8k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.2k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.7k
The Art of Programming - Codeland 2020
erikaheidi
56
14k
Transcript
1 / 18 Neural Networks
2 / 18 1. NN !
• Residual Network • Batch Normalization 2. 1. • •
3 / 18 Plain NNs(&) ' pros #%
" (ex. CNN, RNN, ...) cons ! $ $
4 / 18 RNN RNN [1] P. Razvan et
al ,"On the difficulty of training recurrent neural networks." International Conference on Machine Learning. 2013. !"#$ !" %"&$ %"#$ %" %"&$ '() '() '() '*+, '*+, -!"# = /(!!"# ) -! -!$# %! : input !! : hidden state '%&' : '() : input / !" = '*+, 2 !"#$ + '() %"
5 / 18 !" !# !$ %" %# %$ &'(
&'( &'( &)*+ &)*+ ,! = .(!! ) ," ,# RNN 3 1, 12 = 1," 12 + 1,# 12 + 1,$ 12 1,$ 12 = 4 "565$ 1,$ 1!$ 7 1!$ 1!6 7 18!6 12 1!$ 1!" = 1!$ 1!# 7 1!# 1!" = &)*+ 9 :;<= >? !# 7 &)*+ 9 :;<= >? !" @A!B @C : !" ~!6E" fix !6
6 / 18 RNN Vanishing/Exploding Gradient : !"#$ !%&
'( )( … … )* '* ………… ………… +( +* !"#$ (-) !%& (-) '% …… '/ )/ +/
7 / 18 ,$+ /' !"#$ !- !"#$ 2 %
× '()* + ×%,- → # !"#$ !"#$ . 2 % × '()*(+).,-×%,- 1%input or 1)* Loss( RNN ."0& Vanishing/Exploding Gradient
8 / 18 +$ DeepNN( ! +
" )*&!/#% ' (→ ! Loss func ! Loss func → Residual Connection, Batch No malization
9 / 18 0), : Residual Connection – -– F(x)
"/#2 → "/ F(x) + x → (4 '$"/ Identity Mapping +%*1&: 3 . ! 3 Identity – [1] He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer, Cham, 2016.
10 / 18 : Residual Connection –– ' Forward
$#& Backward !$"& Deep % & input
11 / 18 Residual Connection –– https://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
12 / 18 ResNet Batch Normalization ResNet Residual Block
• ImplementationBatch Normalization NN ! $# • Batch Normalization" ## http://torch.ch/blog/2016/02/04/resnets.html Plain
13 / 18 ( ) 1 2
( ) n … Batch Normalization –Revisit Gaussian-
14 / 18 Batch Normalization -Input Data distribution
- (Convergence) !! Input NN → input
15 / 18 Batch Normalization -distribution - !"#$% & '
= ) & ' ← ' − , - ~/(,, -2) input
16 / 18 Batch Normalization Data distribution •
=(!, ")fix • Batch Normalization Batch Normalization
17 / 18 Batch Normalization – [2]Ioffe, Sergey,
and Christian Szegedy. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." (2015). !, # !%$( → normalize scaling '"&# nomalize
18 / 18 DeepNN+ ! /
& -"#.#)%/'( *$ +!→ , Identity – normalize scaling implement Deep Net