Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding distributed processing in Python

Avatar for chie8842 chie8842
June 16, 2019
Technology
2
2k

Understanding distributed processing in Python

#pyconth2019
https://th.pycon.org/talks/#row-47
Slides of my talk at PyConTh2019
Avatar for chie8842

chie8842

June 16, 2019
Tweet

More Decks by chie8842

See All by chie8842
MongoDB Atlas：モダンなアプリ開発を支えるデータプラットフォームのご紹介
Avatar for chie8842 chie8842
0
9
MongoDB Vectorsearchではじめるカスタマイズ可能な生成AIアプリ開発
Avatar for chie8842 chie8842
0
8
MongoDB Atlas Search のご紹介
Avatar for chie8842 chie8842
2
1.7k
MongoDB Atlas Vectorsearchではじめる生成AIアプリ開発
Avatar for chie8842 chie8842
3
1.7k
AWS GlueとAWS Lake Formationではじめるデータマネジメント
Avatar for chie8842 chie8842
0
1.1k
Distributed Processing in Python
Avatar for chie8842 chie8842
2
730
クックパッドにおける推薦（と検索）の取り組み
Avatar for chie8842 chie8842
20
8.1k
Performance Tuning Tips of TensorFlow Inference
Avatar for chie8842 chie8842
1
750
クックパッドにおけるCloud AutoML事例
Avatar for chie8842 chie8842
9
7.9k

Other Decks in Technology

See All in Technology
マルチテナント+マルチプロダクト SaaS への AI Agent の組み込み方
Avatar for KNOWLEDGE WORK / 株式会社ナレッジワーク kworkdev
PRO
2
360
自分を理解するAI時代の準備 〜マイプロフィールMCPの実装〜
Avatar for edo edo_m18
0
110
Devin(Deep) Wiki/Searchの活用で変わる開発の世界観/devin-wiki-search-impact
Avatar for tomoki10 tomoki10
0
350
「実体」で築く共通認識: 開発現場のコミュニケーション最適化 / Let's Get on the Same Page with Concrete Artifacts: Optimization of Communication in dev teams
Avatar for kazizi kazizi55
0
150
評価の納得感を2段階高める「構造化フィードバック」
Avatar for あろえ aloerina
1
210
Tenstorrent 開発者プログラム
Avatar for Tenstorrent Japan tenstorrent_japan
0
310
比起獨自升級 我更喜歡 DevOps 文化 <3
Avatar for LINE Developers Taiwan line_developers_tw
PRO
0
240
Clineを含めたAIエージェントを 大規模組織に導入し、投資対効果を考える / Introducing AI agents into your organization
Avatar for Masato Ishigaki / 石垣雅人 i35_267
1
340
成立するElixirの再束縛（再代入）可という選択
Avatar for kubell kubell_hr
0
360
"SaaS is Dead" は本当か！？ 生成AI時代の医療 Vertical SaaS のリアル
Avatar for KAKEHASHI kakehashi
PRO
3
230
開発効率と信頼性を両立する Ubieのプラットフォームエンジニアリング
Avatar for teru0x1 teru0x1
0
150
Nonaka Sensei
Avatar for Yasunobu Kawaguchi kawaguti
PRO
4
740

Featured

See All Featured
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
223
9.7k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
71
4.9k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
90
590k
Making Projects Easy
Avatar for Brett Harned brettharned
116
6.2k
Designing for Performance
Avatar for Lara Hogan lara
609
69k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
337
57k
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
657
60k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
71
11k
Speed Design
Avatar for Sergey Chernyshev sergeychernyshev
31
990
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
123
52k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
206
24k

Transcript

  1. Understanding distributed processing in Python Chie Hayashida

  2. $IJF)BZBTIJEB !DIJF) • 4PGUXBSF&OHJOFFSBU$PPLQBE GSPN+BQBO! • EFWFMPQBOEDPOUSJCVUFUPTPNF044 XSJUFBSUJDMFT BOENBLF QSFTFOUBUJPOT

    • 8POGJSTUQSJ[FBUSF*OWFOU (BNF%BZ

  3. $PPLQBE ‒ worldʼs largest recipe sharing platform Try it

  4. $PPLQBE JTHMPCBMTFSWJDF

  5. "HFOEB •CBTJDLOPXMFEHFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •QBSBMMFMQSPDFTTJOHJO1ZUIPO •EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO

  6. %JTDMBJNFS • 8IBUEP*UBML • CBTJDBSDIJUFDUVSFPGQBSBMMFMEJTUSJCVUFE QSPDFTTJOH DPNQVUJOH MJCSBSJFTJO1ZUIPO • 8IBUEP*OPU

    UBML • IPXUPJOTUBMMBOETFUVQFBDIMJCSBSJFT • VTBHFTPGEFUBJMFE"1*TPGXIBUXFJOUSPEVDF

  7. 8IFOXFVTFQBSBMMFMEJTUSJCVUFE QSPDFTTJOH FH •%PXOMPBEMBSHFEBUBJOTIPSUUJNF •6TFMBSHFEBUBGPS%BUBBOBMZTJTPS .BDIJOF-FBSOJOH •)BOEMFBMPUPGSFRVFTUTTJNVMUBOFPVTMZ BU8FC4JUF

  8. 28IBUJT UIFEJGGFSFODFPGCFMPX •$PODVSSFOUQSPDFTTJOH •1BSBMMFMQSPDFTTJOH

  9. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz $PODVSSFODZJTBCPVU EFBMJOHXJUI MPUTPG UIJOHTBUPODF 1BSBMMFMJTNJTBCPVU EPJOH MPUTPGUIJOHTBUPODF 

    3PC1JLFIUUQTUBMLTHPMBOHPSHXB[BTMJEF

  10. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz )FSFJT$PODVSSFOU )FSFJTQBSBMMFMBOEBMTPDPODVSSFOU

  11. $PODVSSFOUBOEQBSBMMFMQSPHSBNTDIFNF •$PODVSSFOUQSPHSBNTDIFNF $PODVSSFOU 1SPHSBN 1SPDFTT 1SPDFTT TDIFEVMFS $16 1SPDFTT 2VFVF

    1BSBMMFM 1SPHSBN 1SPDFTT 1SPDFTT $16 $16 •1BSBMMFMQSPHSBNTDIFNF

  12. 28IBUJT UIFEJGGFSFODFPGCFMPX •1BSBMMFMQSPDFTTJOH •%JTUSJCVUFEQSPDFTTJOH

  13. l1BSBMMFMQSPDFTTJOHzBOEl%JTUSJCVUFEQSPDFTTJOHz • 1BSBMMFMQSPDFTTJOH 1SPDFTTPS T .FNPSZ 1SPDFTTPS T .FNPSZ 1SPDFTTPS

    T .FNPSZ 1SPDFTTPS 1SPDFTTPS 1SPDFTTPS .FNPSZ • %JTUSJCVUFEQSPDFTTJOH /8 /8 /8 QSPDFTTPSTIBSFTNFNPSZ*OQBSBMMFMQSPDFTTJOH POUIFPUIFSIBOEQSPDFTTPSTIBTPXONFNPSJFTJOEJTUSJCVUFEQSPDFTTJOH

  14. 8IFOBOEIPXTIPVMEXF VTFQBSBMMFMEJTUSJCVUFE BSDIJUFDUVSFJO1ZUIPO

  15. UISFBEJOHBOENVMUJQSPDFTTJOH •1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • l#BUUFSZJODMVEFEz •8FDBOQBSBMMFMJ[FPVSQSPHSBNXJUIPVUBOZUIJSE QBSUZMJCSBSJFT

  16. UISFBEJOH • UISFBEJOHNPEVMFEJWJEFTPOFQSPDFTTJOUPNVMUJQMF UISFBET .BJO5ISFBE 5ISFBE 5ISFBE 1SPDFTT

  17. UISFBEJOH • (*- (MPCBM *OUFSQSFUFS-PDL MJNJUTPOMZPOFUISFBEDBOCF JOBTUBUFPGFYFDVUJPOBUBOZQPJOUJOUJNFJO$1ZUIPO 1SPDFTT 5ISFBE 

    5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  18. 8IZ(*- • 5ISFBETJOQSPDFTTVTFTTBNFNFNPSZTQBDF • (*-XBTDIPTFOUPQSPUFDUUIFJOUFSOBMNFNPSZVTFECZ UIFJOUFSQSFUFS QSPDFTT UISFBE UISFBE .FNPSZ4QBDF

  19. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  20. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO *0 BXBJU SVO "DRVJSF -PDL *0 *0 *0 VTF$16 EPOPUVTF$16

  21. NVMUJQSPDFTTJOH •NVMUJQSPDFTTJOHTQBXOTQSPDFTTFTGPS QBSBMMFMJ[BUJPO •QBSBMMFMJ[FJOCPUIMPDBMBOESFNPUF .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT

  22. NVMUJQSPDFTTJOH • FBDIQSPDFTTFTIBTJUTPXONFNPSZTQBDF • /PUBGGFDUFECZ(*- • -FTTMJLFMZUPDBVTFFSSPSTEVFUPEBUBDPSSVQUJPOPS EFBEMPDL .BJO1SPDFTT $IJME1SPDFTT

    $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF $PQZ .FNPSZ4QBDF $PQZ

  23. *OXIBUTJUVBUJPOTJTNVMUJUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT$16CPVOETVDIBT • /VNFSJDBMDBMDVSBUJPO • 3FHVMBSFYQSFTTJPOQSPDFTTJOH • #VUUIFQFSGPSNBODFJO1ZUIPO`TNVMUJQSPDFTTJOHEPFTO`U

    JODSFBTFMJOFBSMZ

  24. NVMUJQSPDFTTJOH • 5IFSFJTPWFSIFBEEVFUPDPNNVOJDBUJPO CFUXFFOQBSFOUQSPDFTTBOEDIJMEQSPDFTT .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF

    $PQZ .FNPSZ4QBDF $PQZ

  25. NVMUJQSPDFTTJOH •UIFNBYOVNCFSPGUBTLTXIJDIDBOCFSVO TJNVMUBOFPVTMZJTSFTUSJDUFEUP$16OVNCFS 5BTL 5BTL 5BTL 5BTL 5BTL 5BTL $16

    $16

  26. +PCMJC • UIJSEQBSUZMJCSBSZGPSQBSBMMFMQSPDFTTJOHJO1ZUIPO • +PCMJC DBOVTF UISFBEJOHPSNVMUJQSPDFTTJOHGPSCBDLFOE • MB[ZFWBMVBUJPOVTJOHUSBOTQBSFOUEJTLDBDIJOHPGUIF PVUQVU

    • FBTZBOETJNQMFDPEJOH • MPHHJOHBOEUSBDJOHPGUIFFYFDVUJPO https://joblib.readthedocs.io/

  27. 4VNNBSZVQUPIFSF • 1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • UISFBEJOHJTFGGFDUJWFUP*0CPVOEQSPDFTTJOH • NVMUJQSPDFTTJOHJTFGGFDUJWFUP$16CPVOEQSPDFTTJOH • CPUIPGUXPBSFNBJOMZGPSPOMZGPSQSPDFTTFTSVOBUPOMZ POFNBDIJOF

    • TPNFPGEBUBMJCSBSJFTUSBOTQBSFOUMZVTFUIFTFQBSBMMFM QSPDFTTJOHMJCSBSJFT

  28. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO • 8IFOXFVTFMBSHFEBUBXIJDIDPVMEO`UQSPDFTTJOPOF NBDIJOF XFIBWFUPDPOTJEFSBCPVUEJTUSJCVUFE QSPDFTTJOH • QSPQFSUJFTXIJDITIPVMECFJODMVEFEJOEJTUSJCVUFE GSBNFXPSL •

    "VUPNBUJDSFDPWFSZ GSPNFSSPS • 5BTLEJWJTJPO KPCTDIFEVMJOH • 5PPMTGPS.POJUPSJOH %FCVHHBCJMJUZ

  29. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO *JOUSPEVDFUISFFGBNPVTMJCSBSJFTGPS EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO •$FMFSZ •%BTL •1Z4QBSL

  30. $FMFSZ •EJTUSJCVUFEUBTLRVFVFCVJMUJO1ZUIPO •TJNQMFBOEGMFYJCMFBSDIJUFDUVSF • &WFSZQBSUTDBOCFFYUFOE • IFBWJMZVTFEGPSUBTLCBTFEXPSLMPBETJO 1ZUIPO • XJUI%KBOHP

    • "JSGMPXCBDLFOE

  31. $FMFSZ"SDIJUFDUVSF $MJFOU CSPLFS 3BCCJU.2 3FEJT  424 FUD 2VFVF 2VFVF

    2VFVF 8PSLFS $POTVNFS 8PSLFS $POTVNFS #BDLFOE 3FTVMUT 3FEJT FUD 8PSLFS $POTVNFS 8PSLFS $POTVNFS

  32. *OXIBUTJUVBUJPOTJTDFMFSZFGGFDUJWF  • 8IFOUIFUBTLJTTJNQMFCVUUIFBNPVOUPGEBUBJTUPP NVDIGPSPOFNBDIJOF • 8IFOZPVXBOUUPVTFJUXJUIBOPUIFSGSBNFXPSLTVDIBT %KBOHP

  33. %BTL • EJTUSJCVUFEQSPDFTTJOHGSBNFXPSLCVJMUJO1ZUIPO • CFBCMFUPVTF/VN1Z1BOEBT-JTUTMJLFQBSBMMFM PCKFDUT EBTLBSSBZ EBTLEBUBGSBNF EBTLCBH BOE"1*T

    • 4DIFEVMFSJTDVTUPNJ[BCMFUPHFUHPPEQFSGPSNBODFGPS CPUIGPSQBSBMMFMQSPDFTTJOHPOMPDBMNBDIJOFBOEGPS EJTUSJCVUFEQSPDFTTJOHPODMVTUFS • )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH

  34. %BTL TDIFEVMFS http://docs.dask.org/en/latest/scheduling.html

  35. %BTL 8FC*OUFSGBDF

  36. *OXIBUTJUVBUJPOTJT%BTL FGGFDUJWF  • 8IFOXFXBOUUPQBSBMMFMJ[F/VN1ZPSQBOEBTQSPDFTTJOH • 8IFOUIFEBUBTJ[FGJUTPOFUP NBDIJOFT

  37. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF • DBOCFNPSFTDBMFPVUUIBODFMFSZBOEEBTL • 0SJHJOBMMZCVJMUUPSVOPOB)BEPPQDMVTUFS • GBTUBOEDPTUFGGJDJFOUQSPDFTTJOH •

    )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH • NVMUJGVODUJPOBM • TUSFBN "1* • NBDIJOFMFBSOJOH"1* • DBOCFVTFJONBOBHFNFOUTFSWJDFJODMPVETFSWJDF

  38. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF only with partial API

  39. 1Z4QBSL • DBOCFNPSFTDBMFPVUUIBODFMFSZBOE%BTL node number 1 4-5 thousands

  40. *OXIBUTJUVBUJPOTJT1Z4QBSL FGGFDUJWF  • 8IFOXFXBOUUPSVODPNQMFYUBTLTXJUIBMBSHFBNPVOU PGEBUB • 8IFOXFIBWF)BEPPQDMVTUFS

  41. 4VNNBSZ • &YQMBJOFEBCPVUUIFEJGGFSFODFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •*OUSPEVDFEUIFBSDIJUFDUVSFPG QBSBMMFMEJTUSJCVUFE QSPDFTTJOHMJCSBSJFT

  42. DPVMEO`UJOUSPEVDF • BTZODJP • DPODVSSFOUGVUVSF • "OPUIFSUIJSEQBSUZMJCSBSJFT

  43. %PObUCFBGSBJEUPVTFQBSBMMFM QSPDFTTJOHNPSF

  44. 2VFTUJPO