Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding distributed processing in Python

Avatar for chie8842 chie8842
June 16, 2019
Technology
2
2.1k

Understanding distributed processing in Python

#pyconth2019
https://th.pycon.org/talks/#row-47
Slides of my talk at PyConTh2019

Avatar for chie8842

chie8842

June 16, 2019
Tweet

More Decks by chie8842

See All by chie8842
MongoDB Atlas：モダンなアプリ開発を支えるデータプラットフォームのご紹介
Avatar for chie8842 chie8842
0
24
MongoDB Vectorsearchではじめるカスタマイズ可能な生成AIアプリ開発
Avatar for chie8842 chie8842
0
23
MongoDB Atlas Search のご紹介
Avatar for chie8842 chie8842
2
1.8k
MongoDB Atlas Vectorsearchではじめる生成AIアプリ開発
Avatar for chie8842 chie8842
3
1.8k
AWS GlueとAWS Lake Formationではじめるデータマネジメント
Avatar for chie8842 chie8842
0
1.1k
Distributed Processing in Python
Avatar for chie8842 chie8842
2
780
クックパッドにおける推薦（と検索）の取り組み
Avatar for chie8842 chie8842
20
8.1k
Performance Tuning Tips of TensorFlow Inference
Avatar for chie8842 chie8842
1
760
クックパッドにおけるCloud AutoML事例
Avatar for chie8842 chie8842
9
8k

Other Decks in Technology

See All in Technology
データとAIで明らかになる、私たちの課題 ~Snowflake MCP,Salesforce MCPに触れて~ / Data and AI Insights
Avatar for 株式会社カオナビ kaonavi
0
280
ざっくり学ぶ 『エンジニアリングリーダー 技術組織を育てるリーダーシップと セルフマネジメント』 / 50 minute Engineering Leader
Avatar for iwashi iwashi86
9
4.3k
OTEPsで知るOpenTelemetryの未来 / Observability Conference Tokyo 2025
Avatar for Arthur arthur1
0
430
kotlin-lsp の開発開始に触発されて、Emacs で Kotlin 開発に挑戦した記録 / kotlin‑lsp as a Catalyst: My Journey to Kotlin Development in Emacs
Avatar for nabeo nabeo
2
300
30分でわかる！！ 『OCI で学ぶクラウドネイティブ 実践 X 理論ガイド』
Avatar for oracle4engineer oracle4engineer
PRO
1
110
Databricks Free Editionで始めるMLflow
Avatar for Takaaki Yayoi taka_aki
0
770
ラスベガスの歩き方 2025年版（re:Invent 事前勉強会）
Avatar for JunjiKoide junjikoide
0
890
GCASアップデート（202508-202510）
Avatar for 高橋広和 techniczna
0
260
今から間に合う re:Invent 準備グッズと現地の地図、その他ラスベガスを周る際の Tips/reinvent-preparation-guide
Avatar for emi emiki
1
260
組織全員で向き合うAI Readyなデータ利活用
Avatar for harry gappy50
5
2.1k
Playwrightで始めるUI自動テスト入門
Avatar for とことんDevOps devops_vtj
0
110
GTC 2025 : 가속되고 있는 미래
Avatar for Jeongkyu Shin inureyes
PRO
0
150

Featured

See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
34
2.5k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
33k
Navigating Team Friction
Avatar for Lara Hogan lara
190
15k
Designing for Performance
Avatar for Lara Hogan lara
610
69k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.6k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
37
2.6k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
7
700
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M
How GitHub (no longer) Works
Avatar for Zach Holman holman
315
140k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k

Transcript

  1. Understanding distributed processing in Python Chie Hayashida

  2. $IJF)BZBTIJEB !DIJF) • 4PGUXBSF&OHJOFFSBU$PPLQBE GSPN+BQBO! • EFWFMPQBOEDPOUSJCVUFUPTPNF044 XSJUFBSUJDMFT BOENBLF QSFTFOUBUJPOT

    • 8POGJSTUQSJ[FBUSF*OWFOU (BNF%BZ

  3. $PPLQBE ‒ worldʼs largest recipe sharing platform Try it

  4. $PPLQBE JTHMPCBMTFSWJDF

  5. "HFOEB •CBTJDLOPXMFEHFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •QBSBMMFMQSPDFTTJOHJO1ZUIPO •EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO

  6. %JTDMBJNFS • 8IBUEP*UBML • CBTJDBSDIJUFDUVSFPGQBSBMMFMEJTUSJCVUFE QSPDFTTJOH DPNQVUJOH MJCSBSJFTJO1ZUIPO • 8IBUEP*OPU

    UBML • IPXUPJOTUBMMBOETFUVQFBDIMJCSBSJFT • VTBHFTPGEFUBJMFE"1*TPGXIBUXFJOUSPEVDF

  7. 8IFOXFVTFQBSBMMFMEJTUSJCVUFE QSPDFTTJOH FH •%PXOMPBEMBSHFEBUBJOTIPSUUJNF •6TFMBSHFEBUBGPS%BUBBOBMZTJTPS .BDIJOF-FBSOJOH •)BOEMFBMPUPGSFRVFTUTTJNVMUBOFPVTMZ BU8FC4JUF

  8. 28IBUJT UIFEJGGFSFODFPGCFMPX •$PODVSSFOUQSPDFTTJOH •1BSBMMFMQSPDFTTJOH

  9. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz $PODVSSFODZJTBCPVU EFBMJOHXJUI MPUTPG UIJOHTBUPODF 1BSBMMFMJTNJTBCPVU EPJOH MPUTPGUIJOHTBUPODF 

    3PC1JLFIUUQTUBMLTHPMBOHPSHXB[BTMJEF

  10. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz )FSFJT$PODVSSFOU )FSFJTQBSBMMFMBOEBMTPDPODVSSFOU

  11. $PODVSSFOUBOEQBSBMMFMQSPHSBNTDIFNF •$PODVSSFOUQSPHSBNTDIFNF $PODVSSFOU 1SPHSBN 1SPDFTT 1SPDFTT TDIFEVMFS $16 1SPDFTT 2VFVF

    1BSBMMFM 1SPHSBN 1SPDFTT 1SPDFTT $16 $16 •1BSBMMFMQSPHSBNTDIFNF

  12. 28IBUJT UIFEJGGFSFODFPGCFMPX •1BSBMMFMQSPDFTTJOH •%JTUSJCVUFEQSPDFTTJOH

  13. l1BSBMMFMQSPDFTTJOHzBOEl%JTUSJCVUFEQSPDFTTJOHz • 1BSBMMFMQSPDFTTJOH 1SPDFTTPS T .FNPSZ 1SPDFTTPS T .FNPSZ 1SPDFTTPS

    T .FNPSZ 1SPDFTTPS 1SPDFTTPS 1SPDFTTPS .FNPSZ • %JTUSJCVUFEQSPDFTTJOH /8 /8 /8 QSPDFTTPSTIBSFTNFNPSZ*OQBSBMMFMQSPDFTTJOH POUIFPUIFSIBOEQSPDFTTPSTIBTPXONFNPSJFTJOEJTUSJCVUFEQSPDFTTJOH

  14. 8IFOBOEIPXTIPVMEXF VTFQBSBMMFMEJTUSJCVUFE BSDIJUFDUVSFJO1ZUIPO

  15. UISFBEJOHBOENVMUJQSPDFTTJOH •1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • l#BUUFSZJODMVEFEz •8FDBOQBSBMMFMJ[FPVSQSPHSBNXJUIPVUBOZUIJSE QBSUZMJCSBSJFT

  16. UISFBEJOH • UISFBEJOHNPEVMFEJWJEFTPOFQSPDFTTJOUPNVMUJQMF UISFBET .BJO5ISFBE 5ISFBE 5ISFBE 1SPDFTT

  17. UISFBEJOH • (*- (MPCBM *OUFSQSFUFS-PDL MJNJUTPOMZPOFUISFBEDBOCF JOBTUBUFPGFYFDVUJPOBUBOZQPJOUJOUJNFJO$1ZUIPO 1SPDFTT 5ISFBE 

    5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  18. 8IZ(*- • 5ISFBETJOQSPDFTTVTFTTBNFNFNPSZTQBDF • (*-XBTDIPTFOUPQSPUFDUUIFJOUFSOBMNFNPSZVTFECZ UIFJOUFSQSFUFS QSPDFTT UISFBE UISFBE .FNPSZ4QBDF

  19. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  20. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO *0 BXBJU SVO "DRVJSF -PDL *0 *0 *0 VTF$16 EPOPUVTF$16

  21. NVMUJQSPDFTTJOH •NVMUJQSPDFTTJOHTQBXOTQSPDFTTFTGPS QBSBMMFMJ[BUJPO •QBSBMMFMJ[FJOCPUIMPDBMBOESFNPUF .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT

  22. NVMUJQSPDFTTJOH • FBDIQSPDFTTFTIBTJUTPXONFNPSZTQBDF • /PUBGGFDUFECZ(*- • -FTTMJLFMZUPDBVTFFSSPSTEVFUPEBUBDPSSVQUJPOPS EFBEMPDL .BJO1SPDFTT $IJME1SPDFTT

    $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF $PQZ .FNPSZ4QBDF $PQZ

  23. *OXIBUTJUVBUJPOTJTNVMUJUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT$16CPVOETVDIBT • /VNFSJDBMDBMDVSBUJPO • 3FHVMBSFYQSFTTJPOQSPDFTTJOH • #VUUIFQFSGPSNBODFJO1ZUIPO`TNVMUJQSPDFTTJOHEPFTO`U

    JODSFBTFMJOFBSMZ

  24. NVMUJQSPDFTTJOH • 5IFSFJTPWFSIFBEEVFUPDPNNVOJDBUJPO CFUXFFOQBSFOUQSPDFTTBOEDIJMEQSPDFTT .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF

    $PQZ .FNPSZ4QBDF $PQZ

  25. NVMUJQSPDFTTJOH •UIFNBYOVNCFSPGUBTLTXIJDIDBOCFSVO TJNVMUBOFPVTMZJTSFTUSJDUFEUP$16OVNCFS 5BTL 5BTL 5BTL 5BTL 5BTL 5BTL $16

    $16

  26. +PCMJC • UIJSEQBSUZMJCSBSZGPSQBSBMMFMQSPDFTTJOHJO1ZUIPO • +PCMJC DBOVTF UISFBEJOHPSNVMUJQSPDFTTJOHGPSCBDLFOE • MB[ZFWBMVBUJPOVTJOHUSBOTQBSFOUEJTLDBDIJOHPGUIF PVUQVU

    • FBTZBOETJNQMFDPEJOH • MPHHJOHBOEUSBDJOHPGUIFFYFDVUJPO https://joblib.readthedocs.io/

  27. 4VNNBSZVQUPIFSF • 1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • UISFBEJOHJTFGGFDUJWFUP*0CPVOEQSPDFTTJOH • NVMUJQSPDFTTJOHJTFGGFDUJWFUP$16CPVOEQSPDFTTJOH • CPUIPGUXPBSFNBJOMZGPSPOMZGPSQSPDFTTFTSVOBUPOMZ POFNBDIJOF

    • TPNFPGEBUBMJCSBSJFTUSBOTQBSFOUMZVTFUIFTFQBSBMMFM QSPDFTTJOHMJCSBSJFT

  28. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO • 8IFOXFVTFMBSHFEBUBXIJDIDPVMEO`UQSPDFTTJOPOF NBDIJOF XFIBWFUPDPOTJEFSBCPVUEJTUSJCVUFE QSPDFTTJOH • QSPQFSUJFTXIJDITIPVMECFJODMVEFEJOEJTUSJCVUFE GSBNFXPSL •

    "VUPNBUJDSFDPWFSZ GSPNFSSPS • 5BTLEJWJTJPO KPCTDIFEVMJOH • 5PPMTGPS.POJUPSJOH %FCVHHBCJMJUZ

  29. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO *JOUSPEVDFUISFFGBNPVTMJCSBSJFTGPS EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO •$FMFSZ •%BTL •1Z4QBSL

  30. $FMFSZ •EJTUSJCVUFEUBTLRVFVFCVJMUJO1ZUIPO •TJNQMFBOEGMFYJCMFBSDIJUFDUVSF • &WFSZQBSUTDBOCFFYUFOE • IFBWJMZVTFEGPSUBTLCBTFEXPSLMPBETJO 1ZUIPO • XJUI%KBOHP

    • "JSGMPXCBDLFOE

  31. $FMFSZ"SDIJUFDUVSF $MJFOU CSPLFS 3BCCJU.2 3FEJT  424 FUD 2VFVF 2VFVF

    2VFVF 8PSLFS $POTVNFS 8PSLFS $POTVNFS #BDLFOE 3FTVMUT 3FEJT FUD 8PSLFS $POTVNFS 8PSLFS $POTVNFS

  32. *OXIBUTJUVBUJPOTJTDFMFSZFGGFDUJWF  • 8IFOUIFUBTLJTTJNQMFCVUUIFBNPVOUPGEBUBJTUPP NVDIGPSPOFNBDIJOF • 8IFOZPVXBOUUPVTFJUXJUIBOPUIFSGSBNFXPSLTVDIBT %KBOHP

  33. %BTL • EJTUSJCVUFEQSPDFTTJOHGSBNFXPSLCVJMUJO1ZUIPO • CFBCMFUPVTF/VN1Z1BOEBT-JTUTMJLFQBSBMMFM PCKFDUT EBTLBSSBZ EBTLEBUBGSBNF EBTLCBH BOE"1*T

    • 4DIFEVMFSJTDVTUPNJ[BCMFUPHFUHPPEQFSGPSNBODFGPS CPUIGPSQBSBMMFMQSPDFTTJOHPOMPDBMNBDIJOFBOEGPS EJTUSJCVUFEQSPDFTTJOHPODMVTUFS • )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH

  34. %BTL TDIFEVMFS http://docs.dask.org/en/latest/scheduling.html

  35. %BTL 8FC*OUFSGBDF

  36. *OXIBUTJUVBUJPOTJT%BTL FGGFDUJWF  • 8IFOXFXBOUUPQBSBMMFMJ[F/VN1ZPSQBOEBTQSPDFTTJOH • 8IFOUIFEBUBTJ[FGJUTPOFUP NBDIJOFT

  37. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF • DBOCFNPSFTDBMFPVUUIBODFMFSZBOEEBTL • 0SJHJOBMMZCVJMUUPSVOPOB)BEPPQDMVTUFS • GBTUBOEDPTUFGGJDJFOUQSPDFTTJOH •

    )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH • NVMUJGVODUJPOBM • TUSFBN "1* • NBDIJOFMFBSOJOH"1* • DBOCFVTFJONBOBHFNFOUTFSWJDFJODMPVETFSWJDF

  38. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF only with partial API

  39. 1Z4QBSL • DBOCFNPSFTDBMFPVUUIBODFMFSZBOE%BTL node number 1 4-5 thousands

  40. *OXIBUTJUVBUJPOTJT1Z4QBSL FGGFDUJWF  • 8IFOXFXBOUUPSVODPNQMFYUBTLTXJUIBMBSHFBNPVOU PGEBUB • 8IFOXFIBWF)BEPPQDMVTUFS

  41. 4VNNBSZ • &YQMBJOFEBCPVUUIFEJGGFSFODFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •*OUSPEVDFEUIFBSDIJUFDUVSFPG QBSBMMFMEJTUSJCVUFE QSPDFTTJOHMJCSBSJFT

  42. DPVMEO`UJOUSPEVDF • BTZODJP • DPODVSSFOUGVUVSF • "OPUIFSUIJSEQBSUZMJCSBSJFT

  43. %PObUCFBGSBJEUPVTFQBSBMMFM QSPDFTTJOHNPSF

  44. 2VFTUJPO