$30 off During Our Annual Pro Sale. View Details »

モデル高速化百選

yu4u
June 11, 2019

 モデル高速化百選

画像センシングシンポジウム (SSII 2019) の企画セッション「深層学習の高速化 〜 高速チップ、分散学習、軽量モデル 〜​」の講演資料です。
深層学習モデルを高速化する下記6種類の手法の解説です。
- 畳み込みの分解 (Factorization)
- 枝刈り (Pruning)
- アーキテクチャ探索 (Neural Architecture Search; NAS)
- 早期終了、動的計算グラフ
(Early Termination, Dynamic Computation Graph)
- 蒸留 (Distillation)
- 量子化 (Quantization)

yu4u

June 11, 2019
Tweet

More Decks by yu4u

Other Decks in Technology

Transcript

  1. લఏ • ओʹԼهͷ৚݅Λຬͨ͢ख๏Λ঺հ • ಛఆͷϋʔυ΢ΣΞʹґଘͤͣʹ࣮ݱՄೳ • ৞ΈࠐΈχϡʔϥϧωοτϫʔΫ $// ͕ର৅ •

    ਪ࿦࣌ͷߴ଎Խ͕ର৅ • ඦબ͠·͕ͨ͠Ұ෦͚ͩ঺հ͠·͢ • "QQFOEJYʹϦετ͕͋Γ·͢ 
  2. ߴ଎Խʁ • Ϟσϧύϥϝʔλ਺ͷ࡟ݮ • '-01T ."$T ਺ͷ࡟ݮ • ϞσϧϑΝΠϧαΠζͷ࡟ݮ •

    ਪ࿦࣌ؒͷ࡟ݮ • ܇࿅࣌ؒͷ࡟ݮ ඍົʹҧ͏ͷͰɺ࢖͏ͱ͖͸ԿΛॏࢹ͢΂͖͔ɺ ࿦จΛಡΉͱ͖͸Կ͕վળ͍ͯ͠Δͷ͔Λؾʹ͢Δ 
  3. Ϟσϧߴ଎Խ • ৞ΈࠐΈͷ෼ղ 'BDUPSJ[BUJPO • ࢬמΓ 1SVOJOH • ΞʔΩςΫνϟ୳ࡧ /FVSBM"SDIJUFDUVSF4FBSDI/"4

    • ૣظऴྃɺಈతܭࢉάϥϑ &BSMZ5FSNJOBUJPO %ZOBNJD$PNQVUBUJPO(SBQI • ৠཹ %JTUJMMBUJPO • ྔࢠԽ 2VBOUJ[BUJPO 
  4. ৞ΈࠐΈ૚ͷܭࢉྔ • ೖྗϨΠϠαΠζɿ)Y8Y/ • ৞ΈࠐΈΧʔωϧɿ,Y,Y/Y. DPOW,Y, .ͱදه FHDPOWY  •

    ग़ྗϨΠϠαΠζɿ)Y8Y. • ৞ΈࠐΈͷܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ.ʢόΠΞε߲Λແࢹʣ  8 ) / . , , 8 ) ⼊⼒特徴マップ 畳み込み カーネル / 出⼒特徴マップ ˎ ࿨ ཁૉੵ × . DPOW,º, . 畳み込み層の計算量は • 画像/特徴マップのサイズ(HW) • ⼊出⼒チャネル数(NM) • カーネルサイズ(K2) に⽐例
  5. ۭؒํ޲ͷ෼ղ • େ͖ͳ৞ΈࠐΈΧʔωϧΛখ͞ͳ৞ΈࠐΈΧʔωϧʹ෼ղ • ྫ͑͹Yͷ৞ΈࠐΈΛYͷ৞ΈࠐΈͭʹ෼ղ • ͜ΕΒ͸ಉ͡αΠζͷड༰໺Λ͕࣋ͭ෼ղ͢Δͱܭࢉྔ͸ • *ODFQUJPOW<>Ͱ͸࠷ॳͷY৞ΈࠐΈΛY৞ΈࠐΈͭʹ෼ղ •

    Ҏ߱ͷ4&/FU΍4IV⒐F/FU7౳ͷ࣮૷Ͱ΋ར༻͞Ε͍ͯΔ<>  ಛ௃Ϛοϓ conv5x5 conv3x3 - conv3x3 [4] C. Szegedy, et al., "Rethinking the Inception Architecture for Computer Vision," in Proc. of CVPR, 2016. [18] T. He, et al., "Bag of Tricks for Image Classification with Convolutional Neural Networks," in Proc. of CVPR, 2019.
  6. ۭؒํ޲ͷ෼ղ • OYOΛYOͱOYʹ෼ղ͢Δ͜ͱ΋  [4] C. Szegedy, et al., "Rethinking

    the Inception Architecture for Computer Vision," in Proc. of CVPR, 2016.
  7. ۭؒํ޲ͱνϟωϧํ޲ͷ෼ղ TFQBSBCMFDPOW • ۭؒํ޲ͱνϟωϧํ޲ͷ৞ΈࠐΈΛಠཱʹߦ͏ • %FQUIXJTF৞ΈࠐΈʢۭؒํ޲ʣ • ಛ௃Ϛοϓʹର͠νϟωϧຖʹ৞ΈࠐΈ • ܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ.

    ./ )ɾ8ɾ,ɾ/ • 1PJOUXJTF৞ΈࠐΈʢνϟωϧํ޲ʣ • Yͷ৞ΈࠐΈ • ܭࢉྔɿ)ɾ8ɾ/ɾ,ɾ. , )ɾ8ɾ/ɾ. • %FQUIXJTF QPJOUXJTF TFQBSBCMF • ܭࢉྔɿ)ɾ8ɾ/ɾ , . 㲈)ɾ8ɾ/ɾ. ˞., • )ɾ8ɾ/ɾ,ɾ.͔Βେ෯ʹܭࢉྔΛ࡟ݮ  W H W H N 1 1 M W H W H N K K N W H W H N M K K 通常 depthwise pointwise
  8. 9DFQUJPO<> • 4FQBSBCMFDPOWΛଟ༻ͨ͠Ϟσϧ  [6] F. Chollet, "Xception: Deep learning

    with depthwise separable convolutions," in Proc. of CVPR, 2017.
  9. .PCJMF/FU<> • EFQUIXJTFQPJOUXJTFDPOWΛଟ༻ • վྑ൛ͷ.PCJMF/FU7<>7<>΋͋Δ  通常の畳み込み MobileNetの1要素 [7] A.

    Howard, et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," in arXiv:1704.04861, 2017. [13] M. Sandler, et al., "MobileNetV2: Inverted Residuals and Linear Bottlenecks," in Proc. of CVPR, 2018. [20] A. Howard, et al., "Searching for MobileNetV3," in arXiv:1905.02244, 2019.
  10. 4IV⒐F/FU<> • .PCJMF/FUͷϘτϧωοΫͱͳ͍ͬͯΔDPOWYΛ HSPVQDPOWY DIBOOFMTIV⒐Fʹஔ׵ • HSPVQDPOWೖྗͷಛ௃ϚοϓΛ(ݸʹάϧʔϓԽ͠ ֤άϧʔϓ಺Ͱݸผʹ৞ΈࠐΈΛߦ͏ ʢܭࢉྔ)ɾ8ɾ/ɾ,ɾ.ˠ)ɾ8ɾ/ɾ,ɾ.(ʣ •

    DIBOOFMTIV⒐FνϟωϧͷॱংΛೖΕସ͑Δ SFTIBQF USBOTQPTFͷૢ࡞Ͱ࣮ݱՄೳ DTIVGGMF EFQUIXJTFDPOW HDPOWY spatial channel HDPOWY [8] X. Zhang, et al., "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices," in arXiv:1707.01083, 2017.
  11. $IBOOFM/FU<> • νϟωϧํ޲ʹ࣍ݩͷ৞ΈࠐΈΛߦ͏  [11] H. Gao, Z. Wang, and

    S. Ji, "ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions", in Proc. of NIPS, 2018.
  12. 6OTUSVDUVSFEWT4USVDUVSFE1SVOJOH • 1SVOJOHલͷ৞ΈࠐΈϑΟϧλ • 6OTUSVDUVSFEQSVOJOH • 4USVDUVSFEQSVOJOHʢϑΟϧλʢνϟωϧʣQSVOJOH͕Ұൠతʣ  K K

    … … … M(出⼒チャネル)個 計算量vs.精度のtrade-offは優れているが 専⽤のハードウェアでないと⾼速化できない 単にチャネル数が減少したネットワークに 再構築が可能で⾼速化の恩恵を受けやすい
  13. %FFQ$PNQSFTTJPO<  > • 6OTUSVDUVSFEͳQSVOJOH • - ਖ਼ଇԽΛՃֶ͑ͯश͠ɺઈର஋͕খ͍͞XFJHIUΛʹ • ࣮ࡍʹߴ଎ʹಈ͔͢ʹ͸ઐ༻ϋʔυ͕ඞཁ<>

     [23] S. Han, et al., "Learning both Weights and Connections for Efficient Neural Networks," in Proc. of NIPS, 2015. [25] S. Han, et al., "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding," in Proc. of ICLR, 2016. [26] S. Han, et al., "EIE: Efficient Inference Engine on Compressed Deep Neural Network," in Proc. of ISCA, 2016.
  14. /FUXPSL4MJNNJOH<> • #BUDIOPSNͷύϥϝʔλЍʹ- ϩεΛֶ͔͚ͯश • ֶशޙɺЍ͕খ͍͞νϟωϧΛ࡟আ͠ɺpOFUVOF  νϟωϧຖʹೖྗΛฏۉ0෼ࢄ1ʹਖ਼نԽɺγͱβͰscale & shift

    νϟωϧi … … Batch normalization [33] Z. Liu, et al., "Learning Efficient Convolutional Networks through Network Slimming," in Proc. of ICCV, 2017.
  15. -PUUFSZ5JDLFU)ZQPUIFTJT *$-3`#FTU1BQFS <> • //ʹ͸ɺʮ෦෼ωοτϫʔΫߏ଄ʯͱʮॳظ஋ʯͷ ૊Έ߹Θͤʹʮ౰ͨΓʯ͕ଘࡏ͠ɺͦΕΛҾ͖౰ͯΔͱ ޮ཰తʹֶश͕Մೳͱ͍͏Ծઆ • 6OTUSVDUVSFEͳQSVOJOHͰͦͷߏ଄ͱॳظ஋Λݟ͚ͭΔ͜ͱ͕Ͱ͖ͨ 

    https://www.slideshare.net/YosukeShinya/the-lottery-ticket-hypothesis-finding-small-trainable-neural-networks [44] Jonathan Frankle, Michael Carbin, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks," in Proc. of ICLR, 2019.
  16. ΞʔΩςΫνϟ୳ࡧ /"4 • //ͷΞʔΩςΫνϟΛࣗಈઃܭ͢Δख๏ • ୳ࡧۭؒɺ୳ࡧख๏ɺਫ਼౓ධՁख๏Ͱେ·͔ʹ෼ྨ͞ΕΔ • ୳ࡧۭؒ • (MPCBM

    DFMMCBTFE • ୳ࡧख๏ • ڧԽֶशɺਐԽతΞϧΰϦζϜɺHSBEJFOUϕʔεɺSBOEPN • ਫ਼౓ଌఆख๏ • શֶशɺ෦෼ֶशɺXFJHIUTIBSFɺࢬמΓ୳ࡧ  T. Elsken, J. Metzen, and F. Hutter, "Neural Architecture Search: A Survey," in JMLR, 2019. M. Wistuba, A. Rawat, and T. Pedapati, "A Survey on Neural Architecture Search," in arXiv:1905.01392, 2019. https://github.com/D-X-Y/awesome-NAS
  17. /"4/FU<> • ୳ࡧۭؒɿDFMMɺ ୳ࡧख๏ɿڧԽֶश 1SPYJNBM1PMJDZ0QUJNJ[BUJPO • (MPCBMͳઃܭʹυϝΠϯ஌ࣝΛ׆༻ɺ ߏ੒͢ΔDFMMͷΈΛࣗಈઃܭ ˠ୳ࡧۭؒΛେ෯ʹ࡟ݮ •

    /PSNBMDFMMY/ͱSFEVDUJPODFMMͷελοΫ • 3FEVDUJPODFMM͸࠷ॳʹTUSJEF෇͖ͷ01Ͱ ಛ௃ϚοϓΛμ΢ϯαϯϓϧ • 3FEVDUJPODFMMҎ߱ͰνϟωϧΛഒʹ  [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.
  18. /"4/FUͷίϯτϩʔϥͷಈ࡞  )JEEFOTUBUF˞ Λબ୒  ͦΕΒ΁ͷ01TΛબ୒˞  ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ

    ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ  [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.
  19. /"4/FUͷίϯτϩʔϥͷಈ࡞  )JEEFOTUBUF˞ Λબ୒  ͦΕΒ΁ͷ01TΛબ୒˞  ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ

    ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ  [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.
  20. /"4/FUͷίϯτϩʔϥͷಈ࡞  )JEEFOTUBUF˞ Λબ୒  ͦΕΒ΁ͷ01TΛબ୒˞  ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ

    ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ  sep 3x3 avg 3x3 [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.
  21. /"4/FUͷίϯτϩʔϥͷಈ࡞  )JEEFOTUBUF˞ Λબ୒  ͦΕΒ΁ͷ01TΛબ୒˞  ͦΕΒΛ݁߹͢Δ01 BEEPSDPODBU Λબ୒͠৽ͨͳIJEEFOTUBUFͱ͢Δ

    ˞)JEEFOTUBUF྘ͷϒϩοΫͱIJ IJ* ˞)JEEFOTUBUF΁ͷ01ީิ  concat sep 3x3 avg 3x3 [52] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proc. of CVPR, 2018.
  22. &/"4<> • ୳ࡧۭؒɿDFMMɺ୳ࡧख๏ɿڧԽֶश 3&*/'03$& • $FMMͷߏ଄Λग़ྗ͢Δ3//ίϯτϩʔϥͱɺ ίϯτϩʔϥʔ͕ग़ྗ͢ΔશͯͷωοτϫʔΫΛαϒάϥϑͱͯ͠อ ࣋Ͱ͖ΔڊେͳܭࢉάϥϑʢωοτϫʔΫʣΛಉ࣌ʹֶश ˠੜ੒ͨ͠ωοτϫʔΫͷֶश͕ෆཁʹʢ(16GPSEBZTʣ •

    4JOHMFTIPU XFJHIUTIBSF • ৄࡉ͸ਆࢿྉ Λࢀর  [54] H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and Jeff Dean, "Efficient Neural Architecture Search via Parameter Sharing," in Proc. of ICML, 2018. * https://www.slideshare.net/tkatojp/efficient-neural-architecture-search-via-parameters- sharing-icml2018
  23. &/"4ͷֶश • ίϯτϩʔϥʔͷύϥϝʔλВͱ ڊେͳωοτϫʔΫͷύϥϝʔλXΛަޓʹֶश • Xͷֶश • ВΛݻఆ͠ɺαϒάϥϑΛαϯϓϦϯά • αϒάϥϑΛGPSXBSECBDLXBSE͠XΛߋ৽

    • Вͷֶश • XΛݻఆ͠ɺαϒάϥϑΛαϯϓϦϯά • WBMJEBUJPOσʔλͰਫ਼౓Λଌఆ͠ใुΛऔಘɺ3&*/'03$&ͰВΛߋ৽ 
  24. '#/FU<> • %"354ͱಉ͘͡HSBEJFOUCBTFE • ֤01ͷ࣮σόΠε্Ͱͷॲཧ࣌ؒΛMPPLVQUBCMFʹอ࣋ • ॲཧ࣌ؒΛߟྀͨ͠ϩεΛ͔͚Δ  [61] B.

    Wu, et al., "FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search", in Proc. of CVPR, 2019. ΫϩεΤϯτϩϐʔ ॲཧ࣌ؒ
  25. ଞʹ΋ <>)$BJ -;IV BOE4)BO 1SPYZMFTT/"4%JSFDU/FVSBM"SDIJUFDUVSF 4FBSDIPO5BSHFU5BTLBOE)BSEXBSF JO1SPDPG*$-3  <>.5BO #$IFO

    31BOH 77BTVEFWBO .4BOEMFS ")PXBSE BOE27 -F .OBT/FU1MBUGPSN"XBSF/FVSBM"SDIJUFDUVSF4FBSDIGPS.PCJMF JO 1SPDPG$713  <>9%BJ FUBM $IBN/FU5PXBSET&⒏DJFOU/FUXPSL%FTJHOUISPVHI 1MBUGPSN"XBSF.PEFM"EBQUBUJPO JO1SPDPG$713  <>%4UBNPVMJT FUBM 4JOHMF1BUI/"4%FWJDF"XBSF&⒏DJFOU$POW/FU %FTJHO JO1SPDPG*$.-8  
  26. %JTUJMMJOHUIF,OPXMFEHFJOB/FVSBM/FUXPSL<>  … … 学習画像 学習済みモデル 学習するモデル … 正解ラベル (ハード

    ターゲット) ௨ৗT = 1ͷsoftmaxͷTΛେ͖ͨ͘͠ ιϑτλʔήοτΛར༻ … ソフトターゲット ソフト ターゲット ハード ターゲット ਖ਼ղϥϕϧͱ ֶशϞσϧग़ྗͷ ྆ํΛར༻ [77] G. Hinton, et al., "Distilling the Knowledge in a Neural Network," in Proc. of NIPS Workshop, 2014.
  27. ྔࢠԽ • ωοτϫʔΫͷύϥϝʔλ౳ΛྔࢠԽ͢Δ͜ͱͰ ϞσϧαΠζΛ࡟ݮɺֶश΍ਪ࿦Λߴ଎Խ • ྔࢠԽର৅ • ॏΈɺΞΫςΟϕʔγϣϯʢಛ௃Ϛοϓʣɺޯ഑ɺΤϥʔ • ྔࢠԽख๏

    • ઢܗɺMPHɺඇઢܗεΧϥɺϕΫτϧɺ௚ੵྔࢠԽ • ྔࢠԽϏοτ • CJUʢόΠφϦʣɺ஋    ɺCJUɺCJUɺ೚ҙCJU • ઐ༻ϋʔυ͕ͳ͍ͱԸܙΛड͚ΒΕͳ͍ࣄ͕ଟ͍ • ൒ਫ਼౓ࠞ߹ਫ਼౓ ͸൚༻ϋʔυˍϑϨʔϜϫʔΫͰ΋αϙʔτ  * https://github.com/NVIDIA/apex
  28. 8"(&<> • XFJHIUT 8 BDUJWBUJPOT " HSBEJFOUT ( FSSPST &

    ͷશͯΛྔࢠԽ  [96] S. Wu, et al., "Training and Inference with Integers in Deep Neural Networks," in Proc. of ICLR, 2018.
  29. 8"(&<> • XFJHIUT 8 BDUJWBUJPOT " HSBEJFOUT ( FSSPST &

     バイナリ [96] S. Wu, et al., "Training and Inference with Integers in Deep Neural Networks," in Proc. of ICLR, 2018.
  30. 2VBOUJ[BUJPOBOE5SBJOJOHPG/FVSBM/FUXPSLTGPS&⒏DJFOU*OUFHFS"SJUINFUJD0OMZ *OGFSFODF<> • ਪ࿦࣌ʹVJOUͷԋࢉ͕ϝΠϯͱͳΔΑ͏ʹ ֶश࣌ʹྔࢠԽΛγϛϡϨʔγϣϯ͠ͳ͕Βֶश • 5FOTPS'MPXެࣜʹ࣮૷͕ଘࡏ  [97] B.

    Jacob, et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference," in Proc. of CVPR, 2018. * https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/quantize/README.md
  31. ൚༻తͳߴ଎Խख๏Λ঺հ • ৞ΈࠐΈͷ෼ղ 'BDUPSJ[BUJPO • ࢬמΓ 1SVOJOH • ΞʔΩςΫνϟ୳ࡧ /FVSBM"SDIJUFDUVSF4FBSDI/"4

    • ૣظऴྃɺಈతܭࢉάϥϑ &BSMZ5FSNJOBUJPO %ZOBNJD$PNQVUBUJPO(SBQI • ৠཹ %JTUJMMBUJPO • ྔࢠԽ 2VBOUJ[BUJPO 
  32. 5BLFIPNF.FTTBHF • ܰྔͳϞσϧʢ৞ΈࠐΈͷ෼ղʣΛ1SVOJOH͢Δͷ͕ खͬऔΓૣ͍ • /"4͕ॸຽͷखʹ • ΞʔΩςΫνϟͱϞσϧͷಉֶ࣌शʢ4JOHMFTIPUԽʣ • '-01TͰ͸ͳ࣮͘σόΠεͰͷ଎౓ΛϑΟʔυόοΫ

    • ࠓޙ • 1SVOJOHͱ/"4ͷҰମԽ • ࢀߟ • ୈճεςΞϥϘਓ޻஌ೳηϛφʔ ʮ৞ΈࠐΈχϡʔϥϧωοτϫʔΫͷߴਫ਼౓Խͱߴ଎Խʯ 
  33. ৞ΈࠐΈͷ෼ղ <>-4JGSFBOE4.BMMBU 3PUBUJPO 4DBMJOHBOE%FGPSNBUJPO*OWBSJBOU4DBUUFSJOHGPS5FYUVSF%JTDSJNJOBUJPO JO1SPDPG$713  <>-4JGSF 3JHJENPUJPO4DBUUFSJOHGPS*NBHF$MBTTJpDBUJPO JO1I%UIFTJT 

    <>.-JO 2$IFO BOE4:BO /FUXPSLJO/FUXPSL JO1SPDPG*$-3  <>$4[FHFEZ FUBM 3FUIJOLJOHUIF*ODFQUJPO"SDIJUFDUVSFGPS$PNQVUFS7JTJPO JO1SPDPG$713  <>'/*BOEPMB 4)BO .8.PTLFXJD[ ,"TISBG 8+%BMMZ BOE,,FVU[FS 4RVFF[F/FU"MFY/FUMFWFMBDDVSBDZXJUIYGFXFSQBSBNFUFSTBOE.# NPEFMTJ[F JOBS9JW  <>'$IPMMFU 9DFQUJPO%FFQMFBSOJOHXJUIEFQUIXJTFTFQBSBCMFDPOWPMVUJPOT JO1SPDPG$713  <>")PXBSE FUBM .PCJMF/FUT&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLTGPS.PCJMF7JTJPO"QQMJDBUJPOT JOBS9JW  <>9;IBOH FUBM 4IV⒐F/FU"O&YUSFNFMZ&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLGPS.PCJMF%FWJDFT JOBS9JW  <>#8V FUBM 4IJGU";FSP'-01 ;FSP1BSBNFUFS JOBS9JW  <>/.B 9;IBOH );IFOH BOE+4VO 4IV⒐F/FU71SBDUJDBM(VJEFMJOFTGPS&⒏DJFOU$//"SDIJUFDUVSF%FTJHO JO1SPDPG&$$7  <>)(BP ;8BOH BOE4+J $IBOOFM/FUT$PNQBDUBOE&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLTWJB$IBOOFM8JTF$POWPMVUJPOT JO1SPDPG/*14  <>()VBOH 4-JV -.BBUFO BOE,8FJOCFSHFS $POEFOTF/FU"O&⒏DJFOU%FOTF/FUVTJOH-FBSOFE(SPVQ$POWPMVUJPOT JO1SPDPG$713  <>.4BOEMFS FUBM .PCJMF/FU7*OWFSUFE3FTJEVBMTBOE-JOFBS#PUUMFOFDLT JO1SPDPG$713  <>(9JF +8BOH 5;IBOH +-BJ 3)POH BOE(2J *($7*OUFSMFBWFE4USVDUVSFE4QBSTF$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG$713  
  34. ৞ΈࠐΈͷ෼ղ <>5;IBOH (2J #9JBP BOE+8BOH *OUFSMFBWFEHSPVQDPOWPMVUJPOTGPSEFFQOFVSBMOFUXPSLT JO1SPDPG*$$7  <>;2JO ;;IBOH

    9$IFO BOE:1FOH '%.PCJMF/FU*NQSPWFE.PCJMF/FUXJUIB'BTU%PXOTBNQMJOH4USBUFHZ JO1SPDPG*$*1  <>,4VO .-J %-JV BOE+8BOH *($7*OUFSMFBWFE-PX3BOL(SPVQ$POWPMVUJPOTGPS&⒏DJFOU%FFQ/FVSBM/FUXPSLT JO#.7$  <>5)F FUBM #BHPG5SJDLTGPS*NBHF$MBTTJpDBUJPOXJUI$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG$713  <>:$IFO FUBM %SPQBO0DUBWF3FEVDJOH4QBUJBM3FEVOEBODZJO$POWPMVUJPOBM/FVSBM/FUXPSLTXJUI0DUBWF$POWPMVUJPO JOBS9JW   <>")PXBSE FUBM 4FBSDIJOHGPS.PCJMF/FU7 JOBS9JW  <>+;IBOH 4FFTBX/FU$POWPMVUJPO/FVSBM/FUXPSL8JUI6OFWFO(SPVQ$POWPMVUJPO JOBS9JW  
  35. ࢬמΓ <>:-F$VO +%FOLFS BOE44PMMB 0QUJNBM#SBJO%BNBHF JO1SPDPG/*14  <>4)BO +1PPM +5SBO

    BOE8%BMMZ -FBSOJOHCPUI8FJHIUTBOE$POOFDUJPOTGPS&⒏DJFOU/FVSBM/FUXPSLT JO1SPDPG/*14  <>88FO $8V :8BOH :$IFO BOE)-J -FBSOJOH4USVDUVSFE4QBSTJUZJO%FFQ/FVSBM/FUXPSLT JO1SPDPG/*14  <>4)BO FUBM %FFQ$PNQSFTTJPO$PNQSFTTJOH%FFQ/FVSBM/FUXPSLTXJUI1SVOJOH 5SBJOFE2VBOUJ[BUJPOBOE)V⒎NBO$PEJOH JO1SPDPG*$-3  <>4)BO +1PPM +5SBO BOE8%BMMZ &*&&⒏DJFOU*OGFSFODF&OHJOFPO$PNQSFTTFE%FFQ/FVSBM/FUXPSL JO1SPDPG*4$"  <>4"OXBS ,)XBOH BOE84VOH 4USVDUVSFE1SVOJOHPG%FFQ$POWPMVUJPOBM/FVSBM/FUXPSLT JO+&5$  <>4$IBOHQJOZP .4BOEMFS BOE";INPHJOPW 5IF1PXFSPG4QBSTJUZJO$POWPMVUJPOBM/FVSBM/FUXPSLT JOBS9JW  <>44DBSEBQBOF %$PNNJOJFMMP ")VTTBJO BOE"6ODJOJ (SPVQ4QBSTF3FHVMBSJ[BUJPOGPS%FFQ/FVSBM/FUXPSLT JO/FVSPDPNQVUJOH  <>)-J FUBM 1SVOJOH'JMUFSTGPS&⒏DJFOU$POW/FUT JO1SPDPG*$-3  <>1.PMDIBOPW 45ZSFF 5,BSSBT 5"JMB BOE+,BVU[ 1SVOJOH$POWPMVUJPOBM/FVSBM/FUXPSLTGPS3FTPVSDF&⒏DJFOU*OGFSFODF JO1SPDPG*$-3  <>%.PMDIBOPW ""TIVLIB BOE%7FUSPW 7BSJBUJPOBM%SPQPVU4QBSTJpFT%FFQ/FVSBM/FUXPSLT JO1SPDPG*$.-  <>;-JV FUBM -FBSOJOH&⒏DJFOU$POWPMVUJPOBM/FUXPSLTUISPVHI/FUXPSL4MJNNJOH JO1SPDPG*$$7  <>:)F FUBM $IBOOFM1SVOJOHGPS"DDFMFSBUJOH7FSZ%FFQ/FVSBM/FUXPSLT JO1SPDPG*$$7  <>+-VP FUBM 5IJ/FU"'JMUFS-FWFM1SVOJOH.FUIPEGPS%FFQ/FVSBM/FUXPSL$PNQSFTTJPO JO1SPDPG*$$7  <>$-PVJ[PT ,6MMSJDI BOE.8FMMJOH #BZFTJBO$PNQSFTTJPOGPS%FFQ-FBSOJOH JO1SPDPG/*14  
  36. ࢬמΓ <>,JSJMM/FLMZVEPW %NJUSZ.PMDIBOPW "STFOJJ"TIVLIB %NJUSZ7FUSPW 4USVDUVSFE#BZFTJBO1SVOJOHWJB-PH/PSNBM.VMUJQMJDBUJWF/PJTF JO1SPDPG/*14  <>.;IVBOE4(VQUB 5PQSVOF

    PSOPUUPQSVOFFYQMPSJOHUIFF⒏DBDZPGQSVOJOHGPSNPEFMDPNQSFTTJPO JO1SPDPG*$-38  <>5:BOH :$IFO BOE74[F %FTJHOJOH&OFSHZ&⒏DJFOU$POWPMVUJPOBM/FVSBM/FUXPSLT6TJOH&OFSHZ"XBSF1SVOJOH JO1SPDPG$713  <>:)F (,BOH 9%POH :'V BOE::BOH 4PGU'JMUFS1SVOJOHGPS"DDFMFSBUJOH%FFQ$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG*+$"*  <>:)F FUBM ".$"VUP.-GPS.PEFM$PNQSFTTJPOBOE"DDFMFSBUJPOPO.PCJMF%FWJDFT JO1SPDPG&$$7  <>5:BOH ")PXBSE #$IFO 9;IBOH "(P .4BOEMFS 74[F BOE)"EBN /FU"EBQU1MBUGPSN"XBSF/FVSBM/FUXPSL"EBQUBUJPOGPS.PCJMF "QQMJDBUJPOT JO1SPDPG&$$7  <>+-VPBOE+8V "VUP1SVOFS"O&OEUP&OE5SBJOBCMF'JMUFS1SVOJOH.FUIPEGPS&⒏DJFOU%FFQ.PEFM*OGFSFODF JOBS9JW  <>+'SBOLMFBOE.$BSCJO 5IF-PUUFSZ5JDLFU)ZQPUIFTJT'JOEJOH4QBSTF 5SBJOBCMF/FVSBM/FUXPSLT JO1SPDPG*$-3  <>;-JV FUBM 3FUIJOLJOHUIF7BMVFPG/FUXPSL1SVOJOH JO1SPDPG*$-3  <>+:V -:BOH /9V +:BOH BOE5)VBOH 4MJNNBCMF/FVSBM/FUXPSLT JO1SPDPG*$-3  <>4-JO 3+J $:BO #;IBOH -$BP 2:F ')VBOH BOE%%PFSNBOO 5PXBSET0QUJNBM4USVDUVSFE$//1SVOJOHWJB(FOFSBUJWF"EWFSTBSJBM-FBSOJOH JO 1SPDPG$713 ("/ <>+:VBOE 5)VBOH 6OJWFSTBMMZ4MJNNBCMF/FUXPSLTBOE*NQSPWFE5SBJOJOH5FDIOJRVFT JOBS9JW  <>+:VBOE 5)VBOH /FUXPSL4MJNNJOHCZ4MJNNBCMF/FUXPSLT5PXBSET0OF4IPU"SDIJUFDUVSF4FBSDIGPS$IBOOFM/VNCFST JOBS9JW  <>;-JV ).V 9;IBOH ;(VP 9:BOH 5$IFOH BOE+4VO .FUB1SVOJOH.FUB-FBSOJOHGPS"VUPNBUJD/FVSBM/FUXPSL$IBOOFM1SVOJOH JO BS9JW  
  37. ΞʔΩςΫνϟ୳ࡧ <>#;PQIBOE27-F /FVSBMBSDIJUFDUVSFTFBSDIXJUISFJOGPSDFNFOUMFBSOJOH JO1SPDPG*$-3  <>#;PQI 77BTVEFWBO +4IMFOT BOE27-F -FBSOJOHUSBOTGFSBCMFBSDIJUFDUVSFTGPSTDBMBCMFJNBHFSFDPHOJUJPO

    JO1SPDPG$713  <>$-JV FUBM 1SPHSFTTJWF/FVSBM"SDIJUFDUVSF4FBSDI JO1SPDPG&$$7  <>)1IBN .:(VBO #;PQI 27-F BOE+FGG%FBO &GGJDJFOU/FVSBM"SDIJUFDUVSF4FBSDIWJB1BSBNFUFS4IBSJOH JO1SPDPG*$.-  <>)-JV ,4JNPOZBO 07JOZBMT $'FSOBOEP BOE,,BWVLDVPHMV )JFSBSDIJDBM3FQSFTFOUBUJPOTGPS&GGJDJFOU"SDIJUFDUVSF4FBSDI JO1SPDPG*$-3   <>&3FBM ""HHBSXBM :)VBOH 27-F 3FHVMBSJ[FE&WPMVUJPOGPS*NBHF$MBTTJGJFS"SDIJUFDUVSF4FBSDI JO1SPDPG"""*  <>)-JV ,4JNPOZBO BOE::BOH %"354%JGGFSFOUJBCMF"SDIJUFDUVSF4FBSDI JO1SPDPG*$-3  <>)$BJ -;IV BOE4)BO 1SPYZMFTT/"4%JSFDU/FVSBM"SDIJUFDUVSF4FBSDIPO5BSHFU5BTLBOE)BSEXBSF JO1SPDPG*$-3  <>.5BO #$IFO 31BOH 77BTVEFWBO .4BOEMFS ")PXBSE BOE27-F .OBT/FU1MBUGPSN"XBSF/FVSBM"SDIJUFDUVSF4FBSDIGPS.PCJMF JO 1SPDPG$713  <>9%BJ FUBM $IBN/FU5PXBSET&GGJDJFOU/FUXPSL%FTJHOUISPVHI1MBUGPSN"XBSF.PEFM"EBQUBUJPO JO1SPDPG$713  <>#8V FUBM '#/FU)BSEXBSF"XBSF&GGJDJFOU$POW/FU%FTJHOWJB%JGGFSFOUJBCMF/FVSBM"SDIJUFDUVSF4FBSDI JO1SPDPG$713  <>%4UBNPVMJT FUBM 4JOHMF1BUI/"4%FWJDF"XBSF&GGJDJFOU$POW/FU%FTJHO JO1SPDPG*$.-8  <>--JBOE"5BMXBMLBS 3BOEPNTFBSDIBOESFQSPEVDJCJMJUZGPSOFVSBMBSDIJUFDUVSFTFBSDI JOBS9JW  
  38. ૣظऴྃɺಈతܭࢉάϥϑ <>:(VP ":BP BOE:$IFO %ZOBNJD/FUXPSL4VSHFSZGPS&⒏DJFOU%//T JO1SPDPG/*14  <>45FFSBQJUUBZBOPO FUBM #SBODIZ/FU'BTU*OGFSFODFWJB&BSMZ&YJUJOHGSPN%FFQ/FVSBM/FUXPSLT

    JO1SPDPG*$13  <>.'JHVSOPW FUBM 4QBUJBMMZ"EBQUJWF$PNQVUBUJPO5JNFGPS3FTJEVBM/FUXPSLT JO1SPDPG$713  <>5#PMVLCBTJ +8BOH 0%FLFM BOE74BMJHSBNB "EBQUJWF/FVSBM/FUXPSLTGPS&⒏DJFOU*OGFSFODF JO1SPDPG*$.-  <>+-JO FUBM 3VOUJNF/FVSBM1SVOJOH JO1SPDPG/*14  <>()VBOH %$IFO 5-J '8V -.BBUFO BOE,8FJOCFSHFS .VMUJ4DBMF%FOTF/FUXPSLTGPS3FTPVSDF&⒏DJFOU*NBHF$MBTTJpDBUJPO JO1SPDPG *$-3  <>98BOH ':V ;%PV 5%BSSFMM BOE+(PO[BMF[ 4LJQ/FU-FBSOJOH%ZOBNJD3PVUJOHJO$POWPMVUJPOBM/FUXPSLT JO1SPDPG&$$7  <>"7FJUBOE 4#FMPOHJF $POWPMVUJPOBM/FUXPSLTXJUI"EBQUJWF*OGFSFODF(SBQIT JO1SPDPG&$$7  <>--JVBOE+%FOH %ZOBNJD%FFQ/FVSBM/FUXPSLT0QUJNJ[JOH"DDVSBDZ&⒏DJFODZ5SBEF0⒎TCZ4FMFDUJWF&YFDVUJPO JO1SPDPG"""*  <>;8V FUBM #MPDL%SPQ%ZOBNJD*OGFSFODF1BUITJO3FTJEVBM/FUXPSLT JO1SPDPG$713  <>3 :V FUBM /*411SVOJOH/FUXPSLTVTJOH/FVSPO*NQPSUBODF4DPSF1SPQBHBUJPO JO1SPDPG$713  <>+,VFO 9,POH ;-JO (8BOH +:JO 44FF BOE:5BO 4UPDIBTUJD%PXOTBNQMJOHGPS$PTU"EKVTUBCMF*OGFSFODFBOE*NQSPWFE3FHVMBSJ[BUJPOJO $POWPMVUJPOBM/FUXPSLT JO1SPDPG$713  <>9(BP :;IBP -%VE[JBL 3.VMMJOT BOE$9V %ZOBNJD$IBOOFM1SVOJOH'FBUVSF#PPTUJOHBOE4VQQSFTTJPO JO1SPDPG*$-3  
  39. ৠཹ <>()JOUPO FUBM %JTUJMMJOHUIF,OPXMFEHFJOB/FVSBM/FUXPSL JO1SPDPG/*148PSLTIPQ  <>+#BBOE3$BSVBOB %P%FFQ/FUT3FBMMZ/FFEUPCF%FFQ JO1SPDPG/*14 

    <>"3PNFSP FUBM 'JU/FUT)JOUTGPS5IJO%FFQ/FUT JO1SPDPG*$-3  <>5$IFO *(PPEGFMMPX BOE+4IMFOT /FU/FU"DDFMFSBUJOH-FBSOJOHWJB,OPXMFEHF5SBOTGFS JO1SPDPG*$-3  <>(6SCBO FUBM %P%FFQ$POWPMVUJPOBM/FUT3FBMMZ/FFEUPCF%FFQBOE$POWPMVUJPOBM JO1SPDPG*$-3  <>+:JN %+PP +#BF BOE+,JN "(JGUGSPN,OPXMFEHF%JTUJMMBUJPO'BTU0QUJNJ[BUJPO /FUXPSL.JOJNJ[BUJPOBOE5SBOTGFS-FBSOJOH JO1SPDPG $713  <>".JTISBBOE%.BSS "QQSFOUJDF6TJOH,OPXMFEHF%JTUJMMBUJPO5FDIOJRVFT5P*NQSPWF-PX1SFDJTJPO/FUXPSL"DDVSBDZ JO1SPDPG*$-3  <>5'VSMBOFMMP ;-JQUPO .5TDIBOOFO -*UUJ BOE""OBOELVNBS #PSO"HBJO/FVSBM/FUXPSLT JO1SPDPG*$.-  <>:;IBOH 59JBOH 5)PTQFEBMFT BOE)-V %FFQ.VUVBM-FBSOJOH JO1SPDPG$713  <>9-BO 9;IV BOE4(POH ,OPXMFEHF%JTUJMMBUJPOCZ0OUIF'MZ/BUJWF&OTFNCMF JO1SPDPG/*14  <>81BSL %,JN :-V BOE.$IP 3FMBUJPOBM,OPXMFEHF%JTUJMMBUJPO JO1SPDPG$713  
  40. ྔࢠԽ <>.$PVSCBSJBVY :#FOHJP BOE+%BWJE #JOBSZ$POOFDU5SBJOJOH%FFQ/FVSBM/FUXPSLTXJUICJOBSZXFJHIUTEVSJOHQSPQBHBUJPOT JO1SPDPG/*14   <>*)VCBSB .$PVSCBSJBVY

    %4PVESZ 3&M:BOJW BOE:#FOHJP #JOBSJ[FE/FVSBM/FUXPSLT JO1SPDPG/*14  <>.3BTUFHBSJ 70SEPOF[+PTFQI BOE3'BSIBEJ 9/03/FU*NBHF/FU$MBTTJpDBUJPO6TJOH#JOBSZ$POWPMVUJPOBM/FVSBM/FUXPSLT JO1SPDPG&$$7   <>+8V $-FOH :8BOH 2)V BOE+$IFOH 2VBOUJ[FE$POWPMVUJPOBM/FVSBM/FUXPSLTGPS.PCJMF%FWJDFT JO1SPDPG$713  <>'-J #;IBOH BOE#-JV 5FSOBSZ8FJHIU/FUXPSLT JOBS9JW  <>4;IPV :8V ;/J 9;IPV )8FO BOE:;PV %P3F'B/FU5SBJOJOH-PX#JUXJEUI$POWPMVUJPOBM/FVSBM/FUXPSLTXJUI-PX#JUXJEUI(SBEJFOUT JO BS9JW  <>$;IV 4)BO ).BP BOE8%BMMZ 5SBJOFE5FSOBSZ2VBOUJ[BUJPO JO1SPDPG*$-3  <>";IPV ":BP :(VP -9V BOE:$IFO *ODSFNFOUBM/FUXPSL2VBOUJ[BUJPO5PXBSET-PTTMFTT$//TXJUI-PX1SFDJTJPO8FJHIUT JO1SPDPG*$-3   <>48V (-J '$IFO BOE-4IJ 5SBJOJOHBOE*OGFSFODFXJUI*OUFHFSTJO%FFQ/FVSBM/FUXPSLT JO1SPDPG*$-3  <>#+BDPC FUBM 2VBOUJ[BUJPOBOE5SBJOJOHPG/FVSBM/FUXPSLTGPS&⒏DJFOU*OUFHFS"SJUINFUJD0OMZ*OGFSFODF JO1SPDPG$713  <>;-JV #8V 8-VP 9:BOH 8-JV BOE,$IFOH #J3FBM/FU&OIBODJOHUIF1FSGPSNBODFPGCJU$//T8JUI*NQSPWFE3FQSFTFOUBUJPOBM$BQBCJMJUZ BOE"EWBODFE5SBJOJOH"MHPSJUIN JO1SPDPG&$$7  <>/8BOH +$IPJ %#SBOE $$IFO BOE,(PQBMBLSJTIOBO 5SBJOJOH%FFQ/FVSBM/FUXPSLTXJUICJU'MPBUJOH1PJOU/VNCFST JO1SPDPG/*14  <>(:BOH FUBM 48"-14UPDIBTUJD8FJHIU"WFSBHJOHJO-PX1SFDJTJPO5SBJOJOH JO1SPDPG*$.-  
  41.