Share
Embed
モデル高速化百選
画像センシングシンポジウム (SSII 2019) の企画セッション「深層学習の高速化 〜 高速チップ、分散学習、軽量モデル 〜」の講演資料です。
深層学習モデルを高速化する下記6種類の手法の解説です。
- 畳み込みの分解 (Factorization)
- 枝刈り (Pruning)
- アーキテクチャ探索 (Neural Architecture Search; NAS)
- 早期終了、動的計算グラフ
(Early Termination, Dynamic Computation Graph)
- 蒸留 (Distillation)
- 量子化 (Quantization)
yu4u
June 11, 2019
Transcript
64 slides
-
લఏ • ओʹԼهͷ݅Λຬͨ͢ख๏Λհ • ಛఆͷϋʔυΣΞʹґଘͤͣʹ࣮ݱՄೳ • ΈࠐΈχϡʔϥϧωοτϫʔΫ $// ͕ର •
ਪ࣌ͷߴԽ͕ର • ඦબ͠·͕ͨ͠Ұ෦͚ͩհ͠·͢ • "QQFOEJYʹϦετ͕͋Γ·͢ -
ߴԽʁ • Ϟσϧύϥϝʔλͷݮ • '-01T ."$T ͷݮ • ϞσϧϑΝΠϧαΠζͷݮ •
ਪ࣌ؒͷݮ • ܇࿅࣌ؒͷݮ ඍົʹҧ͏ͷͰɺ͏ͱ͖ԿΛॏࢹ͖͔͢ɺ จΛಡΉͱ͖Կ͕վળ͍ͯ͠Δͷ͔Λؾʹ͢Δ -
ϞσϧߴԽ • ΈࠐΈͷղ 'BDUPSJ[BUJPO • ࢬמΓ 1SVOJOH • ΞʔΩςΫνϟ୳ࡧ /FVSBM"SDIJUFDUVSF4FBSDI/"4
• ૣظऴྃɺಈతܭࢉάϥϑ &BSMZ5FSNJOBUJPO %ZOBNJD$PNQVUBUJPO(SBQI • ৠཹ %JTUJMMBUJPO • ྔࢠԽ 2VBOUJ[BUJPO -
ΈࠐΈͷܭࢉྔ • ೖྗϨΠϠαΠζɿ)Y8Y/ • ΈࠐΈΧʔωϧɿ,Y,Y/Y. DPOW,Y, .ͱදه FHDPOWY •
ग़ྗϨΠϠαΠζɿ)Y8Y. • ΈࠐΈͷܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ.ʢόΠΞε߲Λແࢹʣ 8 ) / . , , 8 ) ⼊⼒特徴マップ 畳み込み カーネル / 出⼒特徴マップ ˎ ཁૉੵ × . DPOW,º, . 畳み込み層の計算量は • 画像/特徴マップのサイズ(HW) • ⼊出⼒チャネル数(NM) • カーネルサイズ(K2) に⽐例 -
ۭؒํͷղ • େ͖ͳΈࠐΈΧʔωϧΛখ͞ͳΈࠐΈΧʔωϧʹղ • ྫ͑YͷΈࠐΈΛYͷΈࠐΈͭʹղ • ͜ΕΒಉ͡αΠζͷड༰Λ͕࣋ͭղ͢Δͱܭࢉྔ • *ODFQUJPOW<>Ͱ࠷ॳͷYΈࠐΈΛYΈࠐΈͭʹղ •
Ҏ߱ͷ4&/FU4IV⒐F/FU7ͷ࣮Ͱར༻͞Ε͍ͯΔ<> ಛϚοϓ conv5x5 conv3x3 - conv3x3 [4] C. Szegedy, et al., "Rethinking the Inception Architecture for Computer Vision," in Proc. of CVPR, 2016. [18] T. He, et al., "Bag of Tricks for Image Classification with Convolutional Neural Networks," in Proc. of CVPR, 2019. -
ۭؒํͷղ • OYOΛYOͱOYʹղ͢Δ͜ͱ [4] C. Szegedy, et al., "Rethinking
the Inception Architecture for Computer Vision," in Proc. of CVPR, 2016. -
ۭؒํͱνϟωϧํͷղ TFQBSBCMFDPOW • ۭؒํͱνϟωϧํͷΈࠐΈΛಠཱʹߦ͏ • %FQUIXJTFΈࠐΈʢۭؒํʣ • ಛϚοϓʹର͠νϟωϧຖʹΈࠐΈ • ܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ.
./ )ɾ8ɾ,ɾ/ • 1PJOUXJTFΈࠐΈʢνϟωϧํʣ • YͷΈࠐΈ • ܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ. , )ɾ8ɾ/ɾ. • %FQUIXJTF QPJOUXJTF TFQBSBCMF • ܭࢉྔɿ)ɾ8ɾ/ɾ , . 㲈)ɾ8ɾ/ɾ. ˞., • )ɾ8ɾ/ɾ,ɾ.͔Βେ෯ʹܭࢉྔΛݮ W H W H N 1 1 M W H W H N K K N W H W H N M K K 通常 depthwise pointwise -
9DFQUJPO<> • 4FQBSBCMFDPOWΛଟ༻ͨ͠Ϟσϧ [6] F. Chollet, "Xception: Deep learning
with depthwise separable convolutions," in Proc. of CVPR, 2017. -
.PCJMF/FU<> • EFQUIXJTFQPJOUXJTFDPOWΛଟ༻ • վྑ൛ͷ.PCJMF/FU7<>7<>͋Δ 通常の畳み込み MobileNetの1要素 [7] A.
Howard, et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," in arXiv:1704.04861, 2017. [13] M. Sandler, et al., "MobileNetV2: Inverted Residuals and Linear Bottlenecks," in Proc. of CVPR, 2018. [20] A. Howard, et al., "Searching for MobileNetV3," in arXiv:1905.02244, 2019. -
4IV⒐F/FU<> • .PCJMF/FUͷϘτϧωοΫͱͳ͍ͬͯΔDPOWYΛ HSPVQDPOWY DIBOOFMTIV⒐Fʹஔ • HSPVQDPOWೖྗͷಛϚοϓΛ(ݸʹάϧʔϓԽ͠ ֤άϧʔϓͰݸผʹΈࠐΈΛߦ͏ ʢܭࢉྔ)ɾ8ɾ/ɾ,ɾ.ˠ)ɾ8ɾ/ɾ,ɾ.(ʣ •
DIBOOFMTIV⒐FνϟωϧͷॱংΛೖΕସ͑Δ SFTIBQF USBOTQPTFͷૢ࡞Ͱ࣮ݱՄೳ DTIVGGMF EFQUIXJTFDPOW HDPOWY spatial channel HDPOWY [8] X. Zhang, et al., "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices," in arXiv:1707.01083, 2017. -
$IBOOFM/FU<> • νϟωϧํʹ࣍ݩͷΈࠐΈΛߦ͏ [11] H. Gao, Z. Wang, and
S. Ji, "ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions", in Proc. of NIPS, 2018. -
ࢬמΓ 1SVOJOH • ΈࠐΈશ݁߹ͷॏΈͷҰ෦Λʹ͢Δ͜ͱͰ ύϥϝʔλɾܭࢉྔΛݮ ωοτϫʔΫΛֶश ࢬמΓʢਫ਼Լʣ
ωοτϫʔΫΛ࠶ֶशʢਫ਼Λ͋Δఔճ෮ʣ ͱ͍͏ϑϩʔ͕Ұൠత -
6OTUSVDUVSFEWT4USVDUVSFE1SVOJOH • 1SVOJOHલͷΈࠐΈϑΟϧλ • 6OTUSVDUVSFEQSVOJOH • 4USVDUVSFEQSVOJOHʢϑΟϧλʢνϟωϧʣQSVOJOH͕Ұൠతʣ K K
… … … M(出⼒チャネル)個 計算量vs.精度のtrade-offは優れているが 専⽤のハードウェアでないと⾼速化できない 単にチャネル数が減少したネットワークに 再構築が可能で⾼速化の恩恵を受けやすい -
%FFQ$PNQSFTTJPO< > • 6OTUSVDUVSFEͳQSVOJOH • - ਖ਼ଇԽΛՃֶ͑ͯश͠ɺઈର͕খ͍͞XFJHIUΛʹ • ࣮ࡍʹߴʹಈ͔͢ʹઐ༻ϋʔυ͕ඞཁ<>
[23] S. Han, et al., "Learning both Weights and Connections for Efficient Neural Networks," in Proc. of NIPS, 2015. [25] S. Han, et al., "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding," in Proc. of ICLR, 2016. [26] S. Han, et al., "EIE: Efficient Inference Engine on Compressed Deep Neural Network," in Proc. of ISCA, 2016. -
1SVOJOH'JMUFSTGPS&⒏DJFOU$POW/FUT<> • 4USVDUVSFEQSVOJOHʢνϟωϧϨϕϧͷQSVOJOHʣ • ֤ϨΠϠʹ͍ͭͯɺϑΟϧλͷॏΈͷઈରͷ૯͕ খ͍͞ͷ͔ΒQSVOJOH • ֤ϨΠϠͷQSVOJOHQSVOJOHͷTFOTJUJWJUZ͔Β ਓखͰௐ •
1SVOJOHޙʹGJOFUVOF [30] H. Li, et al., "Pruning Filters for Efficient ConvNets," in Proc. of ICLR, 2017. -
/FUXPSL4MJNNJOH<> • #BUDIOPSNͷύϥϝʔλЍʹ- ϩεΛֶ͔͚ͯश • ֶशޙɺЍ͕খ͍͞νϟωϧΛআ͠ɺpOFUVOF νϟωϧຖʹೖྗΛฏۉ0ࢄ1ʹਖ਼نԽɺγͱβͰscale & shift
νϟωϧi … … Batch normalization [33] Z. Liu, et al., "Learning Efficient Convolutional Networks through Network Slimming," in Proc. of ICCV, 2017. -
L0 Ͱͳ͘Lasso ʹ؇ͯ͠ղ͘ $IBOOFM1SVOJOH<> • ͋ΔGFBUVSFNBQͷνϟωϧআͨ͠߹ʹ ࣍ͷGFBUVSFNBQͷޡ͕ࠩ࠷খͱͳΔΑ͏νϟωϧΛબ [34] Y.
He, et al., "Channel Pruning for Accelerating Very Deep Neural Networks," in Proc. of ICCV, 2017. -
5IJ/FU<> • લड़ͷख๏ͱಉ͡Α͏ʹɺ࣍ͷGFBUVSFNBQͷޡ͕ࠩ ࠷খͱͳΔϨΠϠΛHSFFEZআ • আޙʹɺΈࠐΈͷॏΈΛޡ͕ࠩ࠷খʹͳΔΑ͏ʹ ௐˠpOFUVOF [35] J.
Luo, et al., "ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression," in Proc. of ICCV, 2017. -
"VUP.-GPS.PEFM$PNQSFTTJPOBOE"DDFMFSBUJPO ".$ <> • ڧԽֶशʢP⒎QPMJDZBDUPSDSJUJDʣʹΑΓ ֤ϨΠϠຖͷ࠷దͳQSVOJOHΛֶश • ೖྗରϨΠϠͷใͱͦΕ·ͰͷQSVOJOH݁Ռɺ ใुrΤϥʔºMPH '-01T
PSMPH 1BSBNT [41] Y. He, et al., "AMC - AutoML for Model Compression and Acceleration on Mobile Devices," in Proc. of ECCV, 2018. -
-PUUFSZ5JDLFU)ZQPUIFTJT *$-3`#FTU1BQFS <> • //ʹɺʮ෦ωοτϫʔΫߏʯͱʮॳظʯͷ Έ߹ΘͤʹʮͨΓʯ͕ଘࡏ͠ɺͦΕΛҾ͖ͯΔͱ ޮతʹֶश͕Մೳͱ͍͏Ծઆ • 6OTUSVDUVSFEͳQSVOJOHͰͦͷߏͱॳظΛݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ
https://www.slideshare.net/YosukeShinya/the-lottery-ticket-hypothesis-finding-small-trainable-neural-networks [44] Jonathan Frankle, Michael Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks," in Proc. of ICLR, 2019. -
/FUXPSL1SVOJOHBT"SDIJUFDUVSF4FBSDI<> • 4USVDUVSFEͳQSVOJOHޙͷωοτϫʔΫΛTDSBUDI͔Βֶशͤͯ͞ pOFUVOFͱಉ͔ͦΕΑΓྑ͍݁Ռ͕ಘΒΕΔͱ͍͏ओு • ͭ·ΓQSVOJOHɺॏཁͳॏΈΛ୳ࡧ͍ͯ͠ΔͷͰͳ͘ ֤ϨΠϠʹͲͷఔͷνϟωϧΛׂΓͯΔ͔ͱ͍͏ /FVSBM"SDIJUFDUVSF4FBSDI /"4 Λ͍ͯ͠ΔͱΈͳͤΔ
• -PUUFSZ5JDLFU)ZQPUIFTJTͰVOTUSVDUVSFEͰɺ-3ͷΈɺ ࣮ݧখنωοτϫʔΫͷΈ [45] Z. Liu, et al., "Rethinking the Value of Network Pruning," in Proc. of ICLR, 2019. -
ΞʔΩςΫνϟ୳ࡧ /"4 • //ͷΞʔΩςΫνϟΛࣗಈઃܭ͢Δख๏ • ୳ࡧۭؒɺ୳ࡧख๏ɺਫ਼ධՁख๏Ͱେ·͔ʹྨ͞ΕΔ • ୳ࡧۭؒ • (MPCBM
DFMMCBTFE • ୳ࡧख๏ • ڧԽֶशɺਐԽతΞϧΰϦζϜɺHSBEJFOUϕʔεɺSBOEPN • ਫ਼ଌఆख๏ • શֶशɺ෦ֶशɺXFJHIUTIBSFɺࢬמΓ୳ࡧ T. Elsken, J. Metzen, and F. Hutter, "Neural Architecture Search: A Survey," in JMLR, 2019. M. Wistuba, A. Rawat, and T. Pedapati, "A Survey on Neural Architecture Search," in arXiv:1905.01392, 2019. https://github.com/D-X-Y/awesome-NAS -
/"4/FU<> • ୳ࡧۭؒɿDFMMɺ ୳ࡧख๏ɿڧԽֶश 1SPYJNBM1PMJDZ0QUJNJ[BUJPO • (MPCBMͳઃܭʹυϝΠϯࣝΛ׆༻ɺ ߏ͢ΔDFMMͷΈΛࣗಈઃܭ ˠ୳ࡧۭؒΛେ෯ʹݮ •
/PSNBMDFMMY/ͱSFEVDUJPODFMMͷελοΫ • 3FEVDUJPODFMM࠷ॳʹTUSJEF͖ͷ01Ͱ ಛϚοϓΛμϯαϯϓϧ • 3FEVDUJPODFMMҎ߱ͰνϟωϧΛഒʹ [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018. -
/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ ͦΕΒͷ01TΛબ˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ͠৽ͨͳIJEEFOTUBUFͱ͢Δ
˞)JEEFOTUBUFͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUFͷ01ީิ [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018. -
/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ ͦΕΒͷ01TΛબ˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ͠৽ͨͳIJEEFOTUBUFͱ͢Δ
˞)JEEFOTUBUFͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUFͷ01ީิ [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018. -
/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ ͦΕΒͷ01TΛબ˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ͠৽ͨͳIJEEFOTUBUFͱ͢Δ
˞)JEEFOTUBUFͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUFͷ01ީิ sep 3x3 avg 3x3 [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018. -
/"4/FUͷίϯτϩʔϥͷಈ࡞ )JEEFOTUBUF˞ Λબ ͦΕΒͷ01TΛબ˞ ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ͠৽ͨͳIJEEFOTUBUFͱ͢Δ
˞)JEEFOTUBUFͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUFͷ01ީิ concat sep 3x3 avg 3x3 [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018. -
&/"4<> • ୳ࡧۭؒɿDFMMɺ୳ࡧख๏ɿڧԽֶश 3&*/'03$& • $FMMͷߏΛग़ྗ͢Δ3//ίϯτϩʔϥͱɺ ίϯτϩʔϥʔ͕ग़ྗ͢ΔશͯͷωοτϫʔΫΛαϒάϥϑͱͯ͠อ ࣋Ͱ͖ΔڊେͳܭࢉάϥϑʢωοτϫʔΫʣΛಉ࣌ʹֶश ˠੜͨ͠ωοτϫʔΫͷֶश͕ෆཁʹʢ(16GPSEBZTʣ •
4JOHMFTIPU XFJHIUTIBSF • ৄࡉਆࢿྉ Λࢀর [54] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and Jeff Dean, "Efficient Neural Architecture Search via Parameter Sharing," in Proc. of ICML, 2018. * https://www.slideshare.net/tkatojp/efficient-neural-architecture-search-via-parameters- sharing-icml2018 -
&/"4ͷֶश • ίϯτϩʔϥʔͷύϥϝʔλВͱ ڊେͳωοτϫʔΫͷύϥϝʔλXΛަޓʹֶश • Xͷֶश • ВΛݻఆ͠ɺαϒάϥϑΛαϯϓϦϯά • αϒάϥϑΛGPSXBSECBDLXBSE͠XΛߋ৽
• Вͷֶश • XΛݻఆ͠ɺαϒάϥϑΛαϯϓϦϯά • WBMJEBUJPOσʔλͰਫ਼Λଌఆ͠ใुΛऔಘɺ3&*/'03$&ͰВΛߋ৽ -
%"354<> • ୳ࡧۭؒɿDFMMɺ୳ࡧख๏ɿHSBEJFOU • άϥϑͷଓ01ͷબΛTPGUNBYͰ࣮ݱ͢Δ͜ͱͰɺ ߏ୳ࡧGPSXBSECBDLXBSEͰ࣮ݱ • &/"4ͱಉ͘͡TIBSFEQBSBNɺXͱߏΛަޓʹ࠷దԽ [57]
H. Liu, K. Simonyan, and Y. Yang, "DARTS: Differentiable Architecture Search," in Proc. of ICLR, 2019. -
'#/FU<> • %"354ͱಉ͘͡HSBEJFOUCBTFE • ֤01ͷ࣮σόΠε্Ͱͷॲཧ࣌ؒΛMPPLVQUBCMFʹอ࣋ • ॲཧ࣌ؒΛߟྀͨ͠ϩεΛ͔͚Δ [61] B.
Wu, et al., "FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search", in Proc. of CVPR, 2019. ΫϩεΤϯτϩϐʔ ॲཧ࣌ؒ -
ଞʹ <>)$BJ -;IV BOE4)BO 1SPYZMFTT/"4%JSFDU/FVSBM"SDIJUFDUVSF 4FBSDIPO5BSHFU5BTLBOE)BSEXBSF JO1SPDPG*$-3 <>.5BO #$IFO
31BOH 77BTVEFWBO .4BOEMFS ")PXBSE BOE27 -F .OBT/FU1MBUGPSN"XBSF/FVSBM"SDIJUFDUVSF4FBSDIGPS.PCJMF JO 1SPDPG$713 <>9%BJ FUBM $IBN/FU5PXBSET&⒏DJFOU/FUXPSL%FTJHOUISPVHI 1MBUGPSN"XBSF.PEFM"EBQUBUJPO JO1SPDPG$713 <>%4UBNPVMJT FUBM 4JOHMF1BUI/"4%FWJDF"XBSF&⒏DJFOU$POW/FU %FTJHO JO1SPDPG*$.-8 -
#SBODIZ/FU<> • ωοτϫʔΫͷ్தʹ݁Ռͷग़ྗΛՃ • ֶश࣌ʹͯ͢ͷग़ྗʹదͳXFJHIUΛֶ͔͚ͯश • ͦͷTPGUNBYͷΤϯτϩϐʔ͕ᮢҎԼͷ߹ʹ&YJU [65] S.
Teerapittayanon, et al., "BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks," in Proc. of ICPR, 2016. -
4QBUJBMMZ"EBQUJWF$PNQVUBUJPO5JNF 4"$5 <> • "$5֤3FT#MPDL͕IBMUJOHTDPSFΛग़ྗɺ߹ܭ͕Λ͑Δͱ Ҏ߱ͷॲཧΛεΩοϓʢۭؒྖҬͰߦ͏ͱ4"$5ʣ ܭࢉྔʹؔ͢ΔޯΛՃ [66] M.
Figurnov, et al., "Spatially Adaptive Computation Time for Residual Networks," in Proc. of CVPR, 2017. -
3VOUJNF/FVSBM1SVOJOH<> • ֤ϨΠϠຖʹɺલ·ͰͷಛϚοϓΛೖྗͱ͢Δ3//͕ ར༻͢ΔΈࠐΈϑΟϧλू߹Λܾఆ • ,FFQͨ͠ΈࠐΈϑΟϧλͱݩλεΫͷଛࣦؔʢ࠷ऴͷ߹ ʣΛෛͷใुͱͯ͠2ֶशͰ3//Λֶश [68] J.
Lin, et al., "Runtime Neural Pruning," in Proc. of NIPS, 2017. -
#MPDL%SPQ<> • 1PMJDZOFUXPSLʹը૾ΛೖྗɺͲͷ#MPDLΛεΩοϓ͢Δ͔Λग़ྗ • ,FFQͱͳͬͨ3FT#MPDLͷΈΛGPSXBSE • ೝ͕ࣦࣝഊͨ͠߹ෛͷใुΛɺޭͨ͠߹ʹεΩοϓʹԠ ͨ͡ਖ਼ͷใुΛ༩͑Δ͜ͱͰQPMJDZOFUXPSLΛֶश [73]
Z. Wu, et al., "BlockDrop: Dynamic Inference Paths in Residual Networks," in Proc. of CVPR, 2018. -
ৠཹ %JTUJMMBUJPO • େ͖ͳϞσϧɺෳͷωοτϫʔΫͷΞϯαϯϒϧΛ ʮڭࢣϞσϧʯͱ͠ɺখ͞ͳʮੜెϞσϧʯΛֶश • ڭࢣͷग़ྗதؒಛΛੜె͕ٖ͢ΔΑ͏ͳϩεΛ͔͚Δ 1. アンサンブルモデルや⼤
きなモデルを学習 2. 学習済みモデルを利⽤して ⼩さなモデルを学習 -
%JTUJMMJOHUIF,OPXMFEHFJOB/FVSBM/FUXPSL<> … … 学習画像 学習済みモデル 学習するモデル … 正解ラベル (ハード
ターゲット) ௨ৗT = 1ͷsoftmaxͷTΛେ͖ͨ͘͠ ιϑτλʔήοτΛར༻ … ソフトターゲット ソフト ターゲット ハード ターゲット ਖ਼ղϥϕϧͱ ֶशϞσϧग़ྗͷ ྆ํΛར༻ [77] G. Hinton, et al., "Distilling the Knowledge in a Neural Network," in Proc. of NIPS Workshop, 2014. -
'JU/FU<> • ڭࢣΑΓEFFQ͔ͭUIJOͳੜెΛֶश͢Δ • ੜెͷHVJEFEMBZFS͕ɺڭࢣͷIJUMBZFSͷग़ྗΛ ਖ਼֬ʹٖ͢Δ SFHSFTTJPO ϩεΛՃ [79]
A. Romero, et al., "FitNets: Hints for Thin Deep Nets," in Proc. of ICLR, 2015. -
ྔࢠԽ • ωοτϫʔΫͷύϥϝʔλΛྔࢠԽ͢Δ͜ͱͰ ϞσϧαΠζΛݮɺֶशਪΛߴԽ • ྔࢠԽର • ॏΈɺΞΫςΟϕʔγϣϯʢಛϚοϓʣɺޯɺΤϥʔ • ྔࢠԽख๏
• ઢܗɺMPHɺඇઢܗεΧϥɺϕΫτϧɺੵྔࢠԽ • ྔࢠԽϏοτ • CJUʢόΠφϦʣɺ ɺCJUɺCJUɺҙCJU • ઐ༻ϋʔυ͕ͳ͍ͱԸܙΛड͚ΒΕͳ͍ࣄ͕ଟ͍ • ਫ਼ࠞ߹ਫ਼ ൚༻ϋʔυˍϑϨʔϜϫʔΫͰαϙʔτ * https://github.com/NVIDIA/apex -
8"(&<> • XFJHIUT 8 BDUJWBUJPOT " HSBEJFOUT ( FSSPST &
ͷશͯΛྔࢠԽ [96] S. Wu, et al., "Training and Inference with Integers in Deep Neural Networks," in Proc. of ICLR, 2018. -
8"(&<> • XFJHIUT 8 BDUJWBUJPOT " HSBEJFOUT ( FSSPST &
バイナリ [96] S. Wu, et al., "Training and Inference with Integers in Deep Neural Networks," in Proc. of ICLR, 2018. -
2VBOUJ[BUJPOBOE5SBJOJOHPG/FVSBM/FUXPSLTGPS&⒏DJFOU*OUFHFS"SJUINFUJD0OMZ *OGFSFODF<> • ਪ࣌ʹVJOUͷԋࢉ͕ϝΠϯͱͳΔΑ͏ʹ ֶश࣌ʹྔࢠԽΛγϛϡϨʔγϣϯ͠ͳ͕Βֶश • 5FOTPS'MPXެࣜʹ࣮͕ଘࡏ [97] B.
Jacob, et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference," in Proc. of CVPR, 2018. * https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/quantize/README.md -
൚༻తͳߴԽख๏Λհ • ΈࠐΈͷղ 'BDUPSJ[BUJPO • ࢬמΓ 1SVOJOH • ΞʔΩςΫνϟ୳ࡧ /FVSBM"SDIJUFDUVSF4FBSDI/"4
• ૣظऴྃɺಈతܭࢉάϥϑ &BSMZ5FSNJOBUJPO %ZOBNJD$PNQVUBUJPO(SBQI • ৠཹ %JTUJMMBUJPO • ྔࢠԽ 2VBOUJ[BUJPO -
5BLFIPNF.FTTBHF • ܰྔͳϞσϧʢΈࠐΈͷղʣΛ1SVOJOH͢Δͷ͕ खͬऔΓૣ͍ • /"4͕ॸຽͷखʹ • ΞʔΩςΫνϟͱϞσϧͷಉֶ࣌शʢ4JOHMFTIPUԽʣ • '-01TͰͳ࣮͘σόΠεͰͷΛϑΟʔυόοΫ
• ࠓޙ • 1SVOJOHͱ/"4ͷҰମԽ • ࢀߟ • ୈճεςΞϥϘਓೳηϛφʔ ʮΈࠐΈχϡʔϥϧωοτϫʔΫͷߴਫ਼ԽͱߴԽʯ -
ΈࠐΈͷղ <>-4JGSFBOE4.BMMBU 3PUBUJPO 4DBMJOHBOE%FGPSNBUJPO*OWBSJBOU4DBUUFSJOHGPS5FYUVSF%JTDSJNJOBUJPO JO1SPDPG$713 <>-4JGSF 3JHJENPUJPO4DBUUFSJOHGPS*NBHF$MBTTJpDBUJPO JO1I%UIFTJT
<>.-JO 2$IFO BOE4:BO /FUXPSLJO/FUXPSL JO1SPDPG*$-3 <>$4[FHFEZ FUBM 3FUIJOLJOHUIF*ODFQUJPO"SDIJUFDUVSFGPS$PNQVUFS7JTJPO JO1SPDPG$713 <>'/*BOEPMB 4)BO .8.PTLFXJD[ ,"TISBG 8+%BMMZ BOE,,FVU[FS 4RVFF[F/FU"MFY/FUMFWFMBDDVSBDZXJUIYGFXFSQBSBNFUFSTBOE.# NPEFMTJ[F JOBS9JW <>'$IPMMFU 9DFQUJPO%FFQMFBSOJOHXJUIEFQUIXJTFTFQBSBCMFDPOWPMVUJPOT JO1SPDPG$713 <>")PXBSE FUBM .PCJMF/FUT&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLTGPS.PCJMF7JTJPO"QQMJDBUJPOT JOBS9JW <>9;IBOH FUBM 4IV⒐F/FU"O&YUSFNFMZ&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLGPS.PCJMF%FWJDFT JOBS9JW <>#8V FUBM 4IJGU";FSP'-01 ;FSP1BSBNFUFS JOBS9JW <>/.B 9;IBOH );IFOH BOE+4VO 4IV⒐F/FU71SBDUJDBM(VJEFMJOFTGPS&⒏DJFOU$//"SDIJUFDUVSF%FTJHO JO1SPDPG&$$7 <>)(BP ;8BOH BOE4+J $IBOOFM/FUT$PNQBDUBOE&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLTWJB$IBOOFM8JTF$POWPMVUJPOT JO1SPDPG/*14 <>()VBOH 4-JV -.BBUFO BOE,8FJOCFSHFS $POEFOTF/FU"O&⒏DJFOU%FOTF/FUVTJOH-FBSOFE(SPVQ$POWPMVUJPOT JO1SPDPG$713 <>.4BOEMFS FUBM .PCJMF/FU7*OWFSUFE3FTJEVBMTBOE-JOFBS#PUUMFOFDLT JO1SPDPG$713 <>(9JF +8BOH 5;IBOH +-BJ 3)POH BOE(2J *($7*OUFSMFBWFE4USVDUVSFE4QBSTF$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG$713 -
ΈࠐΈͷղ <>5;IBOH (2J #9JBP BOE+8BOH *OUFSMFBWFEHSPVQDPOWPMVUJPOTGPSEFFQOFVSBMOFUXPSLT JO1SPDPG*$$7 <>;2JO ;;IBOH
9$IFO BOE:1FOH '%.PCJMF/FU*NQSPWFE.PCJMF/FUXJUIB'BTU%PXOTBNQMJOH4USBUFHZ JO1SPDPG*$*1 <>,4VO .-J %-JV BOE+8BOH *($7*OUFSMFBWFE-PX3BOL(SPVQ$POWPMVUJPOTGPS&⒏DJFOU%FFQ/FVSBM/FUXPSLT JO#.7$ <>5)F FUBM #BHPG5SJDLTGPS*NBHF$MBTTJpDBUJPOXJUI$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG$713 <>:$IFO FUBM %SPQBO0DUBWF3FEVDJOH4QBUJBM3FEVOEBODZJO$POWPMVUJPOBM/FVSBM/FUXPSLTXJUI0DUBWF$POWPMVUJPO JOBS9JW <>")PXBSE FUBM 4FBSDIJOHGPS.PCJMF/FU7 JOBS9JW <>+;IBOH 4FFTBX/FU$POWPMVUJPO/FVSBM/FUXPSL8JUI6OFWFO(SPVQ$POWPMVUJPO JOBS9JW -
ࢬמΓ <>:-F$VO +%FOLFS BOE44PMMB 0QUJNBM#SBJO%BNBHF JO1SPDPG/*14 <>4)BO +1PPM +5SBO
BOE8%BMMZ -FBSOJOHCPUI8FJHIUTBOE$POOFDUJPOTGPS&⒏DJFOU/FVSBM/FUXPSLT JO1SPDPG/*14 <>88FO $8V :8BOH :$IFO BOE)-J -FBSOJOH4USVDUVSFE4QBSTJUZJO%FFQ/FVSBM/FUXPSLT JO1SPDPG/*14 <>4)BO FUBM %FFQ$PNQSFTTJPO$PNQSFTTJOH%FFQ/FVSBM/FUXPSLTXJUI1SVOJOH 5SBJOFE2VBOUJ[BUJPOBOE)V⒎NBO$PEJOH JO1SPDPG*$-3 <>4)BO +1PPM +5SBO BOE8%BMMZ &*&&⒏DJFOU*OGFSFODF&OHJOFPO$PNQSFTTFE%FFQ/FVSBM/FUXPSL JO1SPDPG*4$" <>4"OXBS ,)XBOH BOE84VOH 4USVDUVSFE1SVOJOHPG%FFQ$POWPMVUJPOBM/FVSBM/FUXPSLT JO+&5$ <>4$IBOHQJOZP .4BOEMFS BOE";INPHJOPW 5IF1PXFSPG4QBSTJUZJO$POWPMVUJPOBM/FVSBM/FUXPSLT JOBS9JW <>44DBSEBQBOF %$PNNJOJFMMP ")VTTBJO BOE"6ODJOJ (SPVQ4QBSTF3FHVMBSJ[BUJPOGPS%FFQ/FVSBM/FUXPSLT JO/FVSPDPNQVUJOH <>)-J FUBM 1SVOJOH'JMUFSTGPS&⒏DJFOU$POW/FUT JO1SPDPG*$-3 <>1.PMDIBOPW 45ZSFF 5,BSSBT 5"JMB BOE+,BVU[ 1SVOJOH$POWPMVUJPOBM/FVSBM/FUXPSLTGPS3FTPVSDF&⒏DJFOU*OGFSFODF JO1SPDPG*$-3 <>%.PMDIBOPW ""TIVLIB BOE%7FUSPW 7BSJBUJPOBM%SPQPVU4QBSTJpFT%FFQ/FVSBM/FUXPSLT JO1SPDPG*$.- <>;-JV FUBM -FBSOJOH&⒏DJFOU$POWPMVUJPOBM/FUXPSLTUISPVHI/FUXPSL4MJNNJOH JO1SPDPG*$$7 <>:)F FUBM $IBOOFM1SVOJOHGPS"DDFMFSBUJOH7FSZ%FFQ/FVSBM/FUXPSLT JO1SPDPG*$$7 <>+-VP FUBM 5IJ/FU"'JMUFS-FWFM1SVOJOH.FUIPEGPS%FFQ/FVSBM/FUXPSL$PNQSFTTJPO JO1SPDPG*$$7 <>$-PVJ[PT ,6MMSJDI BOE.8FMMJOH #BZFTJBO$PNQSFTTJPOGPS%FFQ-FBSOJOH JO1SPDPG/*14 -
ࢬמΓ <>,JSJMM/FLMZVEPW %NJUSZ.PMDIBOPW "STFOJJ"TIVLIB %NJUSZ7FUSPW 4USVDUVSFE#BZFTJBO1SVOJOHWJB-PH/PSNBM.VMUJQMJDBUJWF/PJTF JO1SPDPG/*14 <>.;IVBOE4(VQUB 5PQSVOF
PSOPUUPQSVOFFYQMPSJOHUIFF⒏DBDZPGQSVOJOHGPSNPEFMDPNQSFTTJPO JO1SPDPG*$-38 <>5:BOH :$IFO BOE74[F %FTJHOJOH&OFSHZ&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLT6TJOH&OFSHZ"XBSF1SVOJOH JO1SPDPG$713 <>:)F (,BOH 9%POH :'V BOE::BOH 4PGU'JMUFS1SVOJOHGPS"DDFMFSBUJOH%FFQ$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG*+$"* <>:)F FUBM ".$"VUP.-GPS.PEFM$PNQSFTTJPOBOE"DDFMFSBUJPOPO.PCJMF%FWJDFT JO1SPDPG&$$7 <>5:BOH ")PXBSE #$IFO 9;IBOH "(P .4BOEMFS 74[F BOE)"EBN /FU"EBQU1MBUGPSN"XBSF/FVSBM/FUXPSL"EBQUBUJPOGPS.PCJMF "QQMJDBUJPOT JO1SPDPG&$$7 <>+-VPBOE+8V "VUP1SVOFS"O&OEUP&OE5SBJOBCMF'JMUFS1SVOJOH.FUIPEGPS&⒏DJFOU%FFQ.PEFM*OGFSFODF JOBS9JW <>+'SBOLMFBOE.$BSCJO 5IF-PUUFSZ5JDLFU)ZQPUIFTJT'JOEJOH4QBSTF 5SBJOBCMF/FVSBM/FUXPSLT JO1SPDPG*$-3 <>;-JV FUBM 3FUIJOLJOHUIF7BMVFPG/FUXPSL1SVOJOH JO1SPDPG*$-3 <>+:V -:BOH /9V +:BOH BOE5)VBOH 4MJNNBCMF/FVSBM/FUXPSLT JO1SPDPG*$-3 <>4-JO 3+J $:BO #;IBOH -$BP 2:F ')VBOH BOE%%PFSNBOO 5PXBSET0QUJNBM4USVDUVSFE$//1SVOJOHWJB(FOFSBUJWF"EWFSTBSJBM-FBSOJOH JO 1SPDPG$713 ("/ <>+:VBOE 5)VBOH 6OJWFSTBMMZ4MJNNBCMF/FUXPSLTBOE*NQSPWFE5SBJOJOH5FDIOJRVFT JOBS9JW <>+:VBOE 5)VBOH /FUXPSL4MJNNJOHCZ4MJNNBCMF/FUXPSLT5PXBSET0OF4IPU"SDIJUFDUVSF4FBSDIGPS$IBOOFM/VNCFST JOBS9JW <>;-JV ).V 9;IBOH ;(VP 9:BOH 5$IFOH BOE+4VO .FUB1SVOJOH.FUB-FBSOJOHGPS"VUPNBUJD/FVSBM/FUXPSL$IBOOFM1SVOJOH JO BS9JW -
ΞʔΩςΫνϟ୳ࡧ <>#;PQIBOE27-F /FVSBMBSDIJUFDUVSFTFBSDIXJUISFJOGPSDFNFOUMFBSOJOH JO1SPDPG*$-3 <>#;PQI 77BTVEFWBO +4IMFOT BOE27-F -FBSOJOHUSBOTGFSBCMFBSDIJUFDUVSFTGPSTDBMBCMFJNBHFSFDPHOJUJPO
JO1SPDPG$713 <>$-JV FUBM 1SPHSFTTJWF/FVSBM"SDIJUFDUVSF4FBSDI JO1SPDPG&$$7 <>)1IBN .:(VBO #;PQI 27-F BOE+FGG%FBO &GGJDJFOU/FVSBM"SDIJUFDUVSF4FBSDIWJB1BSBNFUFS4IBSJOH JO1SPDPG*$.- <>)-JV ,4JNPOZBO 07JOZBMT $'FSOBOEP BOE,,BWVLDVPHMV )JFSBSDIJDBM3FQSFTFOUBUJPOTGPS&GGJDJFOU"SDIJUFDUVSF4FBSDI JO1SPDPG*$-3 <>&3FBM ""HHBSXBM :)VBOH 27-F 3FHVMBSJ[FE&WPMVUJPOGPS*NBHF$MBTTJGJFS"SDIJUFDUVSF4FBSDI JO1SPDPG"""* <>)-JV ,4JNPOZBO BOE::BOH %"354%JGGFSFOUJBCMF"SDIJUFDUVSF4FBSDI JO1SPDPG*$-3 <>)$BJ -;IV BOE4)BO 1SPYZMFTT/"4%JSFDU/FVSBM"SDIJUFDUVSF4FBSDIPO5BSHFU5BTLBOE)BSEXBSF JO1SPDPG*$-3 <>.5BO #$IFO 31BOH 77BTVEFWBO .4BOEMFS ")PXBSE BOE27-F .OBT/FU1MBUGPSN"XBSF/FVSBM"SDIJUFDUVSF4FBSDIGPS.PCJMF JO 1SPDPG$713 <>9%BJ FUBM $IBN/FU5PXBSET&GGJDJFOU/FUXPSL%FTJHOUISPVHI1MBUGPSN"XBSF.PEFM"EBQUBUJPO JO1SPDPG$713 <>#8V FUBM '#/FU)BSEXBSF"XBSF&GGJDJFOU$POW/FU%FTJHOWJB%JGGFSFOUJBCMF/FVSBM"SDIJUFDUVSF4FBSDI JO1SPDPG$713 <>%4UBNPVMJT FUBM 4JOHMF1BUI/"4%FWJDF"XBSF&GGJDJFOU$POW/FU%FTJHO JO1SPDPG*$.-8 <>--JBOE"5BMXBMLBS 3BOEPNTFBSDIBOESFQSPEVDJCJMJUZGPSOFVSBMBSDIJUFDUVSFTFBSDI JOBS9JW -
ૣظऴྃɺಈతܭࢉάϥϑ <>:(VP ":BP BOE:$IFO %ZOBNJD/FUXPSL4VSHFSZGPS&⒏DJFOU%//T JO1SPDPG/*14 <>45FFSBQJUUBZBOPO FUBM #SBODIZ/FU'BTU*OGFSFODFWJB&BSMZ&YJUJOHGSPN%FFQ/FVSBM/FUXPSLT
JO1SPDPG*$13 <>.'JHVSOPW FUBM 4QBUJBMMZ"EBQUJWF$PNQVUBUJPO5JNFGPS3FTJEVBM/FUXPSLT JO1SPDPG$713 <>5#PMVLCBTJ +8BOH 0%FLFM BOE74BMJHSBNB "EBQUJWF/FVSBM/FUXPSLTGPS&⒏DJFOU*OGFSFODF JO1SPDPG*$.- <>+-JO FUBM 3VOUJNF/FVSBM1SVOJOH JO1SPDPG/*14 <>()VBOH %$IFO 5-J '8V -.BBUFO BOE,8FJOCFSHFS .VMUJ4DBMF%FOTF/FUXPSLTGPS3FTPVSDF&⒏DJFOU*NBHF$MBTTJpDBUJPO JO1SPDPG *$-3 <>98BOH ':V ;%PV 5%BSSFMM BOE+(PO[BMF[ 4LJQ/FU-FBSOJOH%ZOBNJD3PVUJOHJO$POWPMVUJPOBM/FUXPSLT JO1SPDPG&$$7 <>"7FJUBOE 4#FMPOHJF $POWPMVUJPOBM/FUXPSLTXJUI"EBQUJWF*OGFSFODF(SBQIT JO1SPDPG&$$7 <>--JVBOE+%FOH %ZOBNJD%FFQ/FVSBM/FUXPSLT0QUJNJ[JOH"DDVSBDZ&⒏DJFODZ5SBEF0⒎TCZ4FMFDUJWF&YFDVUJPO JO1SPDPG"""* <>;8V FUBM #MPDL%SPQ%ZOBNJD*OGFSFODF1BUITJO3FTJEVBM/FUXPSLT JO1SPDPG$713 <>3 :V FUBM /*411SVOJOH/FUXPSLTVTJOH/FVSPO*NQPSUBODF4DPSF1SPQBHBUJPO JO1SPDPG$713 <>+,VFO 9,POH ;-JO (8BOH +:JO 44FF BOE:5BO 4UPDIBTUJD%PXOTBNQMJOHGPS$PTU"EKVTUBCMF*OGFSFODFBOE*NQSPWFE3FHVMBSJ[BUJPOJO $POWPMVUJPOBM/FUXPSLT JO1SPDPG$713 <>9(BP :;IBP -%VE[JBL 3.VMMJOT BOE$9V %ZOBNJD$IBOOFM1SVOJOH'FBUVSF#PPTUJOHBOE4VQQSFTTJPO JO1SPDPG*$-3 -
ৠཹ <>()JOUPO FUBM %JTUJMMJOHUIF,OPXMFEHFJOB/FVSBM/FUXPSL JO1SPDPG/*148PSLTIPQ <>+#BBOE3$BSVBOB %P%FFQ/FUT3FBMMZ/FFEUPCF%FFQ JO1SPDPG/*14
<>"3PNFSP FUBM 'JU/FUT)JOUTGPS5IJO%FFQ/FUT JO1SPDPG*$-3 <>5$IFO *(PPEGFMMPX BOE+4IMFOT /FU/FU"DDFMFSBUJOH-FBSOJOHWJB,OPXMFEHF5SBOTGFS JO1SPDPG*$-3 <>(6SCBO FUBM %P%FFQ$POWPMVUJPOBM/FUT3FBMMZ/FFEUPCF%FFQBOE$POWPMVUJPOBM JO1SPDPG*$-3 <>+:JN %+PP +#BF BOE+,JN "(JGUGSPN,OPXMFEHF%JTUJMMBUJPO'BTU0QUJNJ[BUJPO /FUXPSL.JOJNJ[BUJPOBOE5SBOTGFS-FBSOJOH JO1SPDPG $713 <>".JTISBBOE%.BSS "QQSFOUJDF6TJOH,OPXMFEHF%JTUJMMBUJPO5FDIOJRVFT5P*NQSPWF-PX1SFDJTJPO/FUXPSL"DDVSBDZ JO1SPDPG*$-3 <>5'VSMBOFMMP ;-JQUPO .5TDIBOOFO -*UUJ BOE""OBOELVNBS #PSO"HBJO/FVSBM/FUXPSLT JO1SPDPG*$.- <>:;IBOH 59JBOH 5)PTQFEBMFT BOE)-V %FFQ.VUVBM-FBSOJOH JO1SPDPG$713 <>9-BO 9;IV BOE4(POH ,OPXMFEHF%JTUJMMBUJPOCZ0OUIF'MZ/BUJWF&OTFNCMF JO1SPDPG/*14 <>81BSL %,JN :-V BOE.$IP 3FMBUJPOBM,OPXMFEHF%JTUJMMBUJPO JO1SPDPG$713 -
ྔࢠԽ <>.$PVSCBSJBVY :#FOHJP BOE+%BWJE #JOBSZ$POOFDU5SBJOJOH%FFQ/FVSBM/FUXPSLTXJUICJOBSZXFJHIUTEVSJOHQSPQBHBUJPOT JO1SPDPG/*14 <>*)VCBSB .$PVSCBSJBVY
%4PVESZ 3&M:BOJW BOE:#FOHJP #JOBSJ[FE/FVSBM/FUXPSLT JO1SPDPG/*14 <>.3BTUFHBSJ 70SEPOF[+PTFQI BOE3'BSIBEJ 9/03/FU*NBHF/FU$MBTTJpDBUJPO6TJOH#JOBSZ$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG&$$7 <>+8V $-FOH :8BOH 2)V BOE+$IFOH 2VBOUJ[FE$POWPMVUJPOBM/FVSBM/FUXPSLTGPS.PCJMF%FWJDFT JO1SPDPG$713 <>'-J #;IBOH BOE#-JV 5FSOBSZ8FJHIU/FUXPSLT JOBS9JW <>4;IPV :8V ;/J 9;IPV )8FO BOE:;PV %P3F'B/FU5SBJOJOH-PX#JUXJEUI$POWPMVUJPOBM/FVSBM/FUXPSLTXJUI-PX#JUXJEUI(SBEJFOUT JO BS9JW <>$;IV 4)BO ).BP BOE8%BMMZ 5SBJOFE5FSOBSZ2VBOUJ[BUJPO JO1SPDPG*$-3 <>";IPV ":BP :(VP -9V BOE:$IFO *ODSFNFOUBM/FUXPSL2VBOUJ[BUJPO5PXBSET-PTTMFTT$//TXJUI-PX1SFDJTJPO8FJHIUT JO1SPDPG*$-3 <>48V (-J '$IFO BOE-4IJ 5SBJOJOHBOE*OGFSFODFXJUI*OUFHFSTJO%FFQ/FVSBM/FUXPSLT JO1SPDPG*$-3 <>#+BDPC FUBM 2VBOUJ[BUJPOBOE5SBJOJOHPG/FVSBM/FUXPSLTGPS&⒏DJFOU*OUFHFS"SJUINFUJD0OMZ*OGFSFODF JO1SPDPG$713 <>;-JV #8V 8-VP 9:BOH 8-JV BOE,$IFOH #J3FBM/FU&OIBODJOHUIF1FSGPSNBODFPGCJU$//T8JUI*NQSPWFE3FQSFTFOUBUJPOBM$BQBCJMJUZ BOE"EWBODFE5SBJOJOH"MHPSJUIN JO1SPDPG&$$7 <>/8BOH +$IPJ %#SBOE $$IFO BOE,(PQBMBLSJTIOBO 5SBJOJOH%FFQ/FVSBM/FUXPSLTXJUICJU'MPBUJOH1PJOU/VNCFST JO1SPDPG/*14 <>(:BOH FUBM 48"-14UPDIBTUJD8FJHIU"WFSBHJOHJO-PX1SFDJTJPO5SBJOJOH JO1SPDPG*$.-