Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ligtweight Deep Learning Roadmap with Mobile-BERT

jeongah.arie
December 07, 2019

Ligtweight Deep Learning Roadmap with Mobile-BERT

2019 MODUCON - 기술, 교육, 예술의 컨볼루션(convolution)
http://moducon.kr/

On-Device ML(모바일·임베디드 장치에 딥러닝 모델 이식)을 위한 모델 경량화 가이드 로드맵을 소개합니다. 실제 Mobile-BERT 모델의 Application Case를 기반으로 실제 개발 과정에서 발생할 수 있는 시행착오와 노하우들을 함께 살펴봅니다.

jeongah.arie

December 07, 2019
Tweet

More Decks by jeongah.arie

Other Decks in Technology

Transcript

  1.  %PZPVOH (XBL %POH4FPL :BOH 5BFLNJO ,JN +BFXPPL ,BOH :POHHFVO

    -FF +FPOHBI 4IJO :PVOHDIBF +J .BDIJOF-FBSOJOHPG5IJOHT झ݃౟ಪ ࣘ੄ ٩۞׬ .PEFM 0QUJNJ[BUJPO )BSEXBSF .PCJMF न੿ই KFPOHBISJ উ٘۽੉٘ ੹׸ োҳਗ • 5FOTPSGMPX -JUF .-,JU 1Z5PSDI .PCJMF "OESPJE//"1* 'FEFSBUFE-FBSOJOH • ૒੽ ݽ؛ਸ ࢸ҅ೞҊ ӫҊ ୭੸ച ೟णदఃҊ ࢲࡂ 4FSWJOH ೞח оղࣻҕসਸ ૌѹਃ • ૑಴ ஏ੿җ పझ౟ ࢤഝച • ઱۽ 7JTJPO 1PTF&TUJNBUJPO 0CKFDU%FUFDUJPO 'BDF%FUFDUJPO 5FYU%FUFDUJPOj • ਃ્੄ ҙबࢎח 3// җ 5SBOTGPSNFS m য়ט੄ ߊ಴ب #&35 ✨
  2. -JHIUXFJHIU%FFQ-FBSOJOH • ݽ߄ੌ੐߬٣٘ ੢஖੄ ੷੢ ਊ۝ %JTL4J[F ী Ѧݏח ੸਷

    ݽ؛ ࢎ੉ૉ • ݽ߄ੌ "1੄ यӖ ௏য ജ҃ীࢲ ਗഝ൤ ز੘ೡ ࣻ ੓ب۾ ੸਷ ো࢑۝ • ؀׮ࣻ੄ ݽ߄ੌ জٜ੉ ݣ౭௏যܳ ૑ਗೞ૑ ঋҊ ੓਺ • וܽ 3". 3BOEPN"DDFTT.FNPSZ ࣘبী ؀਽ೞח ੸਷ ݫݽܻ ࢎਊ • ݽ߄ੌ੐߬٣٘ ӝӝ m ಕ੉૚ 1BHJOH ੄ ࠺ਊ੉ ࠺श • উ٘۽੉٘ জীࢲ ࢎਊೞח ݽٚ ё୓ח 3".ী ੷੢ 2VBMDPNN4OBQESBHPO4%. • ௪٘௏য ૑ਗ • $16੄ ࢿמ  ()[ ()[  • YGBTUFSUIBOJ6 • 3".TQFFE.)[ • .)[MPXFSUIBOJ6 • ˯.FNPSZTJ[F (PPHMF1JYFM9- 5FOTPSGMPX -JUF .-,JU 0O%FWJDF 1Z5PSDI .PCJMF
  3. -JHIUXFJHIU%FFQ-FBSOJOH • য়ߡ೻٘ 0WFSIFBE о ઴যٜݶ ࢲߡ ਬ૑ ࠺ਊب хࣗ

    • $16WT (16 • $16ীࢲ উ੿੸ਵ۽ ݽ؛ਸ ࢲࡂೣਵ۽ॄ ঳ਸ ࣻ ੓ח ࠺ਊ ੺х ബҗ • ױ (16୭੸ച 5FOTPS35 ഝਊ о ੜػ ݽ؛੄ ҃਋ ৘৻о ੓ਸ ࣻ ੓਺ • .JDSPTFSWJDF"SDIJUFDUVSFm ݽ؛ ࢎ੉ૉ ੉ग • ੸੤ೡ ࣻ ੓ח ݽ؛ ࢎ੉ૉ੄ ઁড੉ ੓ח 4FWFSMFTT 'VODUJPOٜ ($1 .JDSPTPGU"[VSF $MPVE 4FSWFS "[VSF'VODUJPO
  4. -JHIUXFJHIU%FFQ-FBSOJOH ৵ ਋ܻ੄ ݽ؛਷ о߶ਕঠ ೡө ! 0O%FWJDF]]$MPVE 4FSWFS ࢎ੉ૉо

    ௼Ҋ ࠂ੟ೠ ݽ؛਷ • ݆਷ ੷੢ ҕр ٣झ௼ .FNPSZ ١ ਸ ର૑ೠ׮ • ҅࢑ ࠺ਊ੉ ֫׮ • ীց૑ ߓఠܻ ߂ ੹ӝ ܳ ݆੉ ࣗ࠺ೠ׮ #VU য়ט ࣗѐೡ ۽٘ݗ਷ 0O%FWJDF.-ਸ ਤೠ ҃۝ചী ؊ ૘઺
  5. 148JLJ XJUI.PCJMF#&35 #&35 #JEJSFDUJPOBM&ODPEFS3FQSFTFOUBUJPOTGSPN5SBOTGPSNFST IUUQTBSYJWPSHBCT ױߑೱ ঩਷ নߑೱ Ө਷ নߑೱ

    EFFQMZCJEJSFDUJPOBM • ؀ਊ۝ ఫझ౟ ௏ಌझܳ ੉ਊೞৈ ߧਊ ݾ੸੄ -BOHVBHF6OEFSTUBOEJOHਸ ࣻ೯ೞח ݽ؛ ള۲ • Ө਷ নߑೱࢿਸ ૑פҊ ੓਺ • ࣻরѐ . . ੄ QBSBNFUFSm Ѣ؀ೠ ݽ؛ ࢎ੉ૉ !
  6. 148JLJ XJUI.PCJMF#&35 #&35XJUI4UBOGPSE2VFTUJPO"OTXFSJOH%BUBTFU 42V"% • ਤఃೖ٣ই BSUJDMF BSUJDMFী ؀ೠ 2VFTUJPO

     "OTXFSTFU • ੿ࠁо ࣻ۾ػ ޙױ 8JLJQFEJB Ѩ࢝ Ѿҗ җ ૕ޙ ޙ੢ਸ JOQVUਵ۽ ߉ইࢲ ޙױ ղীࢲ о੢ ਬ۱ೠ ੿׹ TFHNFOUܳ SFUVSO೧઱ח ߑध যڌѱ ٜ݅঻ਸөਃ #&35 .PCJMF
  7. 4UFQTGPS-JHIUXFJHIU 45&1ݾ಴ ࢸ੿ ୶ۿ ࣘب ੿ഛب ݽ؛ ௼ӝ ݫݽܻ ࢎਊ۝

    ߓఠܻ ࢎਊ۝ • पदр ࢿמ ࢎਊ੗ ୓х _NT • NC੉ೞ੄ ݽ؛ ࢎ੉ૉ • ੸਷ ߓఠܻ ࢎਊ۝ 5BTL݃׮ ׮ܲ ਋ࢶࣽਤ $1.#BTFE1PTF&TUJNBUJPO !
  8. 4UFQTGPS-JHIUXFJHIU 45&1ݾ಴ ࢸ੿ ୶ۿ ࣘب ੿ഛب ݽ؛ ௼ӝ ݫݽܻ ࢎਊ۝

    ߓఠܻ ࢎਊ۝ 148JLJ • ࢎਊ੗ ૑োਸ וՙ૑ ޅೡ݅ೠ ࢿמ _NT • (PPHMF1MBZ4UPSFী ١੤ೡ ࣻ ੓ח ݽ؛ ࢎ੉ૉ • NC੉ೞ੄ জ ࢎ੉ૉ ݽ؛ ࢎ੉ૉ NC੉ೞ • ੿ഛب •  ੉࢚ 2"ݽ؛੉޲۽ ੿ഛبо ઺ਃ !
  9. 4UFQTGPS-JHIUXFJHIU #&35 .PCJMF 45&1҃۝ ঌҊ્ܻ ଺ӝ • ޖࣻೠ प೷җ োҳо

    ೙ਃೠ ৔৉ • োҳ ࢶ۹ܳ ࢓ಝࠁݶ  .PEFM"SDIJUFDUVSF • 3FT/FU %FOTF/FU 4RVFF[F/FU  0QUJNJ[JOH$POWPMVUJPO'JMUFS • .PCJMF/FU 4IVGGMF/FU  /FVSBM"SDIJUFDUVSF4FBSDI • /FU"EBQU ./BT/FU
  10. 4UFQTGPS-JHIUXFJHIU #&35 .PCJMF 45&1҃۝ ঌҊ્ܻ ଺ӝ "-#&35"-JUF#&35GPS4FMGTVQFSWJTFE-FBSOJOHPG -BOHVBHF3FQSFTFOUBUJPOT https://arxiv.org/abs/1909.11942 •

    ౵ۄ޷ఠ хࣗ • GBDUPSJ[FEFNCFEEJOHQBSBNFUFSJ[BUJPO • DSPTTMBZFSQBSBNFUFSTIBSJOH • *OUFSDPIFSFODFܳ ਤೠ ࢜۽਍ MPTTࢎਊ • TFOUFODFPSEFSQSFEJDUJPO 401 • ഴܯೠ UG ҳഅ୓ ଵҊ • IUUQTHJUIVCDPNLQFCFSUGPSUG
  11. 4UFQTGPS-JHIUXFJHIU 45&1ݽ؛ ঑୷ .PEFM$PNQSFTTJPO • 8FJHIU1SVOJOH о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO

    ন੗ച • #JOBSJ[BUJPO ੉૓ച • 8FJHIU4IBSJOH о઺஖ ҕਬ • ,OPXMFEHF%JTUJMMBUJPO ૑ध ૐܨ • 5SBOTGFS-FBSOJOH ੹੉ ೟ण j
  12. 4UFQTGPS-JHIUXFJHIU 45&1ݽ؛ ঑୷ .PEFM$PNQSFTTJPO • 8FJHIU1SVOJOH о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO

    ন੗ച • #JOBSJ[BUJPO ੉૓ച • 8FJHIU4IBSJOH о઺஖ ҕਬ • ,OPXMFEHF%JTUJMMBUJPO ૑ध ૐܨ • 5SBOTGFS-FBSOJOH ੹੉ ೟ण j                
  13. 4UFQTGPS-JHIUXFJHIU 45&1ݽ؛ ঑୷ .PEFM$PNQSFTTJPO • 8FJHIU1SVOJOH о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO

    ন੗ച • #JOBSJ[BUJPO ੉૓ച • 8FJHIU4IBSJOH о઺஖ ҕਬ • ,OPXMFEHF%JTUJMMBUJPO ૑ध ૐܨ • 5SBOTGFS-FBSOJOH ੹੉ ೟ण j 5BTLౠࢿ࢚ ࠗ೤ೞ૑ 9 "-#&35$PODFQU • 1SFUSBJOFEػ -BSHFNPEFMਸ ೟णೞৈ ੘਷ ݽ؛ী EJTUJMMBUJPOೞח җ੿ হ੉ ੘਷ ݽ؛ਸ ߄۽ ೟ण दр੉ য়ې Ѧܻח प೷ /0 !
  14. 4UFQTGPS-JHIUXFJHIU 45&1ݽ؛ ঑୷ .PEFM$PNQSFTTJPO • 8FJHIU1SVOJOH о઺஖ о૑஖ӝ • 2VBOUJ[BUJPO

    ন੗ച )PXUP 5FOTPSGMPX ীࢲ ૑ਗೞח ҃۝ച୭੸ച GFBUVSFܳ ࢎਊ
  15. 4UFQTGPS-JHIUXFJHIU 45&1ݽ؛ ঑୷ .PEFM$PNQSFTTJPO m 8FJHIU1SVOJOH 8FJHIU1SVOJOH о઺஖ о૑஖ӝ ۆ

    ӝઓ न҃ݎ੉ о૑Ҋ ੓ח о઺஖ 8FJHIUT ઺ पઁ ୶ۿਸ ਤ೧ ೙ਃೠ ч਷ ࠺Ү੸ ੘਷ чٜী ؀ೠ ղࢿਸ о૑޲۽  ੘਷ о઺஖ чਸ ݽف ਵ ۽ ೞৈ ֎౟ਕ௼੄ ݽ؛ ௼ӝܳ ઴੉ח ӝࣿ import tensorflow_model_optimization as tfmot # Sparsity 0% to 50% while training pruning_schedule = tfmot.sparsity.keras.PolynomialDecay( initial_sparsity=0.0, final_sparsity=0.5, begin_step=2000, end_step=4000) # From 2000 to 4000 step, runs every 100 steps # If it runs every steps, the accuracy would be fallen ! model_for_pruning = tfmot.sparsity.keras.prune_low_magnitude( model, pruning_schedule=pruning_schedule) model_for_pruning.fit(...)
  16. 4UFQTGPS-JHIUXFJHIU 45&1ݽ؛ ঑୷ .PEFM$PNQSFTTJPO m 2VBOUJ[BUJPO 2VBOUJ[BUJPO ন੗ച ۆ न҃ݎ੄

    ࠗز ࣗࣻ੼ ࣻܳ ౠ੿ࣻ۽ ઴੉ח ੘স઱۽ ݽ߄ੌীࢲח GMPBU! VJOU • %VSJOHUSBJOJOHRVBOUJ[BUJPO • "-#&35QSFUSBJOFEXFJHIUܳ ߉ই RVBOUJ[BUJPOਸ ऺইоח ߑೱਵ۽ ࣻ೯ • 1PTUUSBJOJOHRVBOUJ[BUJPO • 8FJHIU0/-: • ݽ؛ ࢎ੉ૉ хࣗ ೞ૑݅ पઁ ୶ۿद੄ ো࢑਷ GMPBU୓҅۽ SFDPOTUSVDUJPOغয ੉ܖয૑ӝ ٸޙী ࣘ ب ߸ച 9 • 8FJHIU"DUJWBUJPO 'VMMZ • 'VMMZ۽ QPTURVBOUࣻ೯ೞӝ ਤ೧ࢲח 5FOTPSGMPX -JUF࢚ীࢲ VJOU୶ۿ ੹ਊ 2VBOUJ[FE0QTܳ ૑ਗ೧ঠೣ " "
  17. 4UFQTGPS-JHIUXFJHIU 45&1ࢲࡂ 4FSWJOH ୭੸ച m (16%FMFHBUJPO"1*ࢎਊ (16 (SBQIJDT1SPDFTTJOH6OJU ݆਷ ୊ܻܳ

    ਃೞ૑݅ ࡈܻ ୊ܻ೧ঠೞח ݽ؛ $16 $FOUSBM1SPDFTTJOH6OJU рױೠ .-ݽ؛ী੄ ୭ࢶ੄ ২࣌ %41 %JHJUBM4JHOBM1SPDFTTPS ੸਷ ୊ܻ۝ਸ ਃೞݶࢲب ࠂ੟ೠ ݽ؛ ষ୒աѱ ࡈܻ ୊ܻ೧ঠೞח ݽ؛
  18. (16%FMFHBUJPO"1* IUUQTXXXUFOTPSGMPXPSHMJUFQFSGPSNBODFHQV@BEWBODFE ✓ .PCJMF(16ܳ оࣘೡ ࣻ ੓ח 5FOTPSGMPX0QFSBUPSח ೠ ੿غয

    ੓਺ ✓ (16%FMFHBUFীࢲ ૑ਗೞ૑ ঋח 0QFSBUPSо ࢎਊػ ࠗ ࠙ী ೠীࢲ $16৬੄ ߽೯੸ੋ ࢎਊ੉ оמೞ૑݅ $16݅ ࢎਊ ೡ ٸ ࠁ׮ غ۰ ࢿמ੉ ೞۅؼ ࣻ ੓਺ ✓ .PCJMF#&35ݽ؛ীࢲח (16%FMFHBUJPOਸ ࢎਊೞח Ѫ ࠁ׮ $16ജ҃ীࢲ .VMUJUISFBEJOHೞחѱ ؊ ബҗ੸੉঻਺
  19. 4VNNBSZ Ѿҗ • *OGFSFODFUJNF ಣӐ NT • .PEFM4J[F  .#

    • "DDVSBDZ • UGFTUJNBUPS XJUI4BWFE.PEFM 'SP[FO(SBQI m  • UG*OUFQSFUFS XJUI5'-JUF m 
  20. 4VNNBSZ 45&1 45&1 45&1 45&1 ݾ಴ ࢸ੿ೞӝ m ୶ۿ ࣘب

    ݽ؛ ࢎ੉ૉ ݫݽܻ ࢎਊ۝ ߓఠܻ ࢎਊ۝ j ҃۝ ঌҊ્ܻ ଺ӝ m թ੉ ݅ٚ պ੢Ҋ ഝਊೞӝ ݽ؛ ঑୷ .PEFM$PNQSFTTJPO ࢲࡂ 4FSWJOH ୭੸ച