Tensorコアを使った PyTorch の高速化について

Transcript

1. 'BTUFS
   1ZUIPO
   .FFU
   VQ
   -5ձ  5FOTPS
   ίΞΛ࢖ͬͨ 1Z5PSDI
   ͷߴ଎Խ  !GBN@UBSP
2. "HFOEB  5FOTPSίΞ JT
   Կ  1Z5PSDIͰ 5FOTPS
   ίΞ࢖͏ʹ͸  ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU  ֶशͰ 5FOTPSίΞΛ࢖͍͍ͨ৔߹ 
3. 
   5FOTPSίΞ JT
   Կ – CJUුಈখ਺఺ '1
   ͷߦྻԋࢉʹಛԽͨ͠ܭࢉ༻ίΞ – 1Z5PSDI ౳ͷਂ૚ֶशϑϨʔϜϫʔΫΛ࢖͏ͱɺσϑΥϧτͰ͸ CJUුಈখ਺఺ ʢ'1 Ͱॾʑͷܭࢉ͕ߦΘΕΔ – /7*%*"
   (16
   ͷ͏ͪ 7PMUB
   ੈ୅Ҏ߱ͷҰ෦ʹ৐͍ͬͯΔ – 5VSJOH
   Ͱ΋ 359
   
   ͡Όͳ͍ͱ৐ͬͯͳ͍ (59
   
   5J
   ʹ͸৐͍ͬͯͳ͍ – 1BTDBM
   .BYXFMM
   ,FQMFS
   ʹ͸ແ͍ ແ࣊൵ – (16
   ྫ – 5*5"/
   7 – (F'PSDF
   359
   γϦʔζ 
   
   
   5J – 5*5"/
   359 – 5FTMB
   7
    ($1
   ͰࢼͤΔ – ࠓޙΑΓීٴ͢Δͱݸਓతʹࢥͬͯ·͢ – ࢀߟϦϯΫ – IUUQTXJLJXJLJKQOWJEJBWHB(16&##&"&#&""&"" – IUUQTKBXJLJQFEJBPSHXJLJ/7*%*"@(F'PSDF  https://www.nvidia.com/content/apac/gtc/ja/pdf/2017/1040.pdf
4. 
   5FOTPSίΞ JT
   Կ – 5FOTPS
   ίΞ࢖Θͳ͍ͱ '1
   ʹͯ͠΋଎౓͸ ഒఔ౓ – 5FOTPS
   ίΞ࢖͏ͱ '1
   ʹͨ͠Β ഒ Ҏ্ʂ – ͨͩ͠࢖͏ͨΊʹ͸੍໿͕ଟ͍ʜ – $6%"
   Ҏ߱ – DV%// 
   Ҏ߱ – '1
   Ͱѻ͏͜ͱΛ໌ه ϥΠϒϥϦͷίϚϯυ౳Ͱ – $POW
   ૚ͷೖྗɾग़ྗνϟϯωϧ਺͕ ͷഒ਺ – શͯͷ૚͕ରԠ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ Ή͠ΖҰ෦͔͠ – ͋͘·Ͱݱ࣌఺ͳͷͰࠓޙ͸؇࿨͞ΕΔ͔΋ʜ  https://www.slideshare.net/NVIDIAJapan/chainer-tensor-fp16
5. 
   1Z5PSDIͰ 5FOTPS
   ίΞ࢖͏ʹ͸ – '1
   Λ࢖͏͜ͱΛ໌ه͢Ε͹ϑϨʔϜϫʔΫ͕উखʹ ࢖ͬͯ͘ΕΔ ͜ͱ͕ଟ͍ – ࠷ۙͷόʔδϣϯʹ͠ͳ͍ͱ͍͚ͳ͍͕ʜ – 1Z5PSDI
   Ͱ͸ʜ – .PEFM
   ͱ *OQVU
   ʹର͠ lIBMG z
   Λ෇͚Δ – ൒ਫ਼౓ʹ͢Δͱ͍͏ҙຯ 
   '1
   ʹ͢Δ – 0VUQVU
   ͸ '1
   ͱ͸ݶΒͳ͍ͷͰ஫ҙ – $POW
   ૚ͷೖྗͱग़ྗνϟϯωϧ਺Λ 
   ͷഒ਺ʹ͢Δ – ஍ຯʹ੍͖͍ͭ໿ – 1SFUSBJO
   NPEFM
   ͩͱ͏·͘࢖͑ͳ͍έʔε͕ग़ͯ͘Δ – ໌ࣔతʹ 5FOTPSίΞΛ࢖͏Α͏ʹ͸Ͱ͖ͳ͍ Ά͍ʁ 
6. 
   ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – .%FU
   ͱ͸ – ࿦จ
   IUUQTRJKJF[IBPHJUIVCJPJNHTNEFUQEG – ࣮૷
   IUUQTHJUIVCDPNRJKJF[IBP.%FU – ࠷ۙग़͖ͯͨݕग़Ϟσϧ – :0-0W
   ΑΓ଎ͯ͘ਫ਼౓͕ߴ͍ Β͍͠ – ࣗ෼ͷσʔλͰ͸ֶशશ͘Ͱ͖ͯͳ͍Ͱ͕͢! – ࣮ݧ৚݅ – $0$0
   EBUBTFU
   %FUFDUJPO
   Ͱ͸ϝδϟʔͳ΍ͭ – ֶशࡁΈϞσϧ 
   Y
    Λ࢖༻ – ެࣜϦϙδτϦʹ͋Δ UFTUQZ Λ࢖༻ – NPEFM JOQVU
   ͷ࣌ؒͱޙॲཧ /.4 ͷ࣌ؒΛܭଌ – ଍ͨ࣌ؒ͠Λ࢖ͬͯ '14
   Λग़͢ ߴ͍΄Ͳྑ͍ –   ຕͷσʔλʹରͯ͠ਪ࿦ ༧ଌ
   Λͯ͠ฏۉ࣌ؒΛͱΔ – ݩ࿦จ͸  
   ຕͰऔ͍ͬͯͨ – Ϟσϧͷਫ਼౓͸ N"1 
7. 
   ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – ࣮ߦ࣌ؒ΍ 5FOTPS
   ίΞΛ࢖͍ͬͯΔ͔֬ೝ͢ΔͨΊʹ࢖ͬͨίϚϯυ – UJNFUJNF
   ؔ਺ – (16བྷΉͨΊԼهͷํ͕ྑͦ͞͏Ͱ͕͢ࠓճ͸ UJNF
   ؔ਺Ͱ࣌ؒͱΓ·ͨ͠! – UPSDIDVEB&WFOU FOBCMF@UJNJOH5SVF – QSPGJMFS – 
   QZUIPO
   N
   UPSDIVUJMTCPUUMFOFDL UFTUQZ ddd – 1Z5PSDI ͷఏڙ͢Δ QSPGJMFS – ݕग़ͷޙॲཧ ࠓճ͸ /.4
   ͕ϘτϧωοΫʹͳ͍ͬͯͳ͍͔֬ೝ͢ΔͨΊʹ࢖༻ – 
   OWQSPG QZUIPO
   ddd – /7*%*"
   ͷఏڙ͢Δ QSPGJMFS
   ίϚϯυɻ5FOTPS
   ίΞ࢖͍ͬͯΔ͔ݟΕΔ – 
   OWDD ddd
   ະ࢖༻ – OWQSPG ͷίϚϯυΛ (6*
   ͰϦονʹݟΕΔΒ͍͠ɻͨͩ͠ΞϓϦαΠζσΧ͍ ਺(# 
8. 
   ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – ࣮ߦ݁Ռ  (16 N"1 ਫ਼౓ %FUFDU
   UJNF
   QFS
   JNBHF
   <NT> /NT UJNF 1FS JNBHF
   <NT> 5PUBM
   <NT> '14 ݩ࿦จ ϦϙδτϦ্ͷ஋ 5JUBO
   9
   1"4$"-    1Z5PSDI
    ެࣜࢦఆόʔδϣϯ 5JUBO
   7      1Z5PSDI
   
   
   '1 5JUBO
   7      1Z5PSDI
   QPTU 5JUBO
   7      1Z5PSDI
   QPTU
   '1 5JUBO
   7     
9. 
   ਪ࿦ ༧ଌ Ͱ࢖ͬͯΈͨ .%FU – ࣮ߦ݁Ռ  (16 N"1 ਫ਼౓ %FUFDU
   UJNF
   QFS
   JNBHF
   <NT> /NT UJNF 1FS JNBHF
   <NT> 5PUBM
   <NT> '14 ݩ࿦จ ϦϙδτϦ্ͷ஋ 5JUBO
   9
   1"4$"-    1Z5PSDI
    ެࣜࢦఆόʔδϣϯ 5JUBO
   7      1Z5PSDI
   
   
   '1 5JUBO
   7      1Z5PSDI
   QPTU 5JUBO
   7      1Z5PSDI
   QPTU
   '1 5JUBO
   7      l1Z5PSDI
   ʹͯ͠z
   ͔ͭ l'1
   ࢖͏z
   ͱ࠷΋଎͘ͳΔ ˠ ࠷ॳͷ ഒ ͔Ζ͏ͯ͡ݩ࿦จͷ '14
   Λ௒͑ͨ ༧ଌ࣌ͷΈͳΒ '1ʹͯ͠΋ ਫ਼౓͸มΘΒͳ͍ 1Z5PSDI
   ͷόʔδϣϯ্͛ͨΒ଎͘ ͳΔ͠ɺ'1ʹͯ͠΋଎͘ͳΔ
10. 
    ֶशͰ 5FOTPSίΞΛ࢖͍͍ͨ৔߹ – ͱΓ͋͑ͣશ෦ '1
    ʹ͢Ε͹͍͍ΜͰ͠ΐʁˠ /Pʂ – ޯ഑ܭࢉ࣌ʹ͔ͳΓӨڹ͢Δ ޯ഑͕ফ͑Δ৔߹΋͋Δ – ˠ ਫ਼౓͕େ͖͘Լ͕Δ৔߹͕͋Δ – .JYFE
    1SFDJTJPO
    5SBJOJOH – ࢀߟϦϯΫ
    IUUQTXXXTMJEFTIBSFOFU/7*%*"+BQBODIBJOFSUFOTPSGQ – '1
    ͱ '1
    ʹΑΔܭࢉΛ NJY
    ֶͨ͠शํ๏ – ۩ମతʹ͸ҎԼͷΑ͏ͳςΫχοΫ͕ඞཁ – ϩεεέʔϦϯά 
    ϩε஋ΛཁॴཁॴͰεέʔϧ͋ͬ΀ͯ͠ޯ഑ফࣦΛ؇࿨ – '1
    ΢ΣΠτߋ৽ 
    'PSXBSE
    ͱ #BDLXBSE
    ͸ '1
    6QEBUF
    Ͱ '1Λ࢖༻ – ্هΛશͯࣗ෼Ͱ΍Δͱେมʂ ֶͼ͸͋Δͱࢥ͍·͕͢ʜ – ˠ BQFY
    Λ࢖͏ͱൺֱతखܰʹͰ͖Δ IUUQTHJUIVCDPN/7*%*"BQFY – /7*%*"
    ͕ఏڙ͢Δ 1Z5PSDI
    ༻ "VUPNBUJD
    .JYFE
    1SFDJTJPO ".1
    πʔϧ – ݩͷίʔυʹର͠਺ߦ଍͚ͩ͢Ͱ .JYFE
    1SFDJTJPO
    5SBJOJOH
    Ͱ͖Δͱͷ͜ͱ – ͨͩ͠ JOTUBMM
    ࣌͸ $6%"
    ΍ 1Z5PSDI
    ͷόʔδϣϯʹؾΛ͚ͭͳ͍ͱ͍͚ͳ͍ 
11. 3FGFSFODFT – <೥൛ ػցֶशϋʔυ΢ΣΞͷ4UBUF
    PG
    UIF
    "SUΛߟ͑Δ d$16
    (16
    516Λఴ͑ͯd
     2JJUB> IUUQTRJJUBDPNBSVUFNBJUFNTGCB – <5SBJOJOH
    /FVSBM
    /FUXPSLT
    XJUI
    .JYFE
    1SFDJTJPO
     /7*%*"> IUUQPOEFNBOEHQVUFDIDPOGDPNHUDUBJXBOQEG @*OUFSOBM4QFBLFS@.JDIBFM$BSJMMJ@1%''PS4IBSJOHQEG – <$IBJOFS Ͱ 5FOTPS
    ίΞ GQ
    Λ࢖͍͜ͳ͢> IUUQTXXXTMJEFTIBSFOFU/7*%*"+BQBODIBJOFSUFOTPSGQ – <$IBJOFS
    ʹ͓͚Δਂ૚ֶशͷߴ଎Խ> IUUQTXXXOWJEJBDPNDPOUFOUBQBDHUDKBQEGQEG – <70-5"
    "/%
    563*/(
    "3$)*5&$563&
    "/%
    1&3'03."/$&
    015*.*;"5*0/
    /7*%*"> IUUQTXXXOWJEJBDPNDPOUFOUBQBDHUDKBQEGQEG – <5SBJOJOH
    XJUI
    .JYFE
    1SFDJTJPO
    
    %FFQ
    -FBSOJOH
    4%,
    %PDVNFOUBUJPO> IUUQTEPDTOWJEJBDPNEFFQMFBSOJOHTELNJYFE QSFDJTJPOUSBJOJOHJOEFYIUNMQZUPSDI – <(16࢓༷Ұཡද  /7*%*"
    (F'PSDF
    8JLJ > IUUQTXJLJXJLJKQOWJEJBWHB(16&##&"&#&""&"" 
12. ͓ΘΓ  ͋Γ͕ͱ͏͍͟͝·ͨ͠
13. ิ଍
    OWQSPG ͷ݁Ռ 1Z5PSDI
     
14. ิ଍
    OWQSPG ͷ݁Ռ 1Z5PSDI