Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Deploying Dask Distributed
Search
Jacob Tomlinson
May 19, 2021
Technology
0
230
Deploying Dask Distributed
Jacob Tomlinson
May 19, 2021
Tweet
Share
More Decks by Jacob Tomlinson
See All by Jacob Tomlinson
Dask on HPC in 2024 - Lightning Talk
jacobtomlinson
0
15
GPU Acceleration in the PyData community
jacobtomlinson
0
9
Dask Demo Day - How RAPIDS users use Dask
jacobtomlinson
0
21
Deploying multi-GPU workloads on Kubernetes in Python
jacobtomlinson
0
190
Overview of Dask on Kubernetes
jacobtomlinson
0
140
k8s-batch-sig - Dask on Kubernetes
jacobtomlinson
0
110
Deploying multi-GPU workloads on Kubernetes in Python
jacobtomlinson
0
140
Dask Overview - Stanford Legion all-hands
jacobtomlinson
0
190
Container Canary - Lightning Talk
jacobtomlinson
0
62
Other Decks in Technology
See All in Technology
飲食店データの分析事例とそれを支えるデータ基盤
kimujun
0
190
組織成長を加速させるオンボーディングの取り組み
sudoakiy
2
220
Flutterによる 効率的なAndroid・iOS・Webアプリケーション開発の事例
recruitengineers
PRO
0
120
テストコード品質を高めるためにMutation Testingライブラリ・Strykerを実戦導入してみた話
ysknsid25
7
2.7k
Lambdaと地方とコミュニティ
miu_crescent
2
370
Security-JAWS【第35回】勉強会クラウドにおけるマルウェアやコンテンツ改ざんへの対策
4su_para
0
180
プロダクト活用度で見えた真実 ホリゾンタルSaaSでの顧客解像度の高め方
tadaken3
0
200
マルチプロダクトな開発組織で 「開発生産性」に向き合うために試みたこと / Improving Multi-Product Dev Productivity
sugamasao
1
310
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
8
880
OCI Network Firewall 概要
oracle4engineer
PRO
0
4.2k
AGIについてChatGPTに聞いてみた
blueb
0
130
生成AIが変えるデータ分析の全体像
ishikawa_satoru
0
170
Featured
See All Featured
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
900
Rails Girls Zürich Keynote
gr2m
94
13k
Mobile First: as difficult as doing things right
swwweet
222
8.9k
Scaling GitHub
holman
458
140k
Music & Morning Musume
bryan
46
6.2k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
109
49k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
BBQ
matthewcrist
85
9.3k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Automating Front-end Workflow
addyosmani
1366
200k
Transcript
Deployment Workshop Deploying Dask Distributed Jacob Tomlinson
Dask Distributed A centrally managed, distributed, dynamic task scheduler
Dask Overview
None
Worker Worker Worker Scheduler Client Protocols TCP UCX Websocket Dask
components can communicate via a variety of different protocols.
Scheduler Starting a scheduler
Connecting a worker Worker Scheduler
Client Scheduler Worker Connecting a client
Client Scheduler Worker Submitting work
Dask Dashboard
JupyterLab Extension
Cluster Managers Utility classes to simplify cluster creation
Local Cluster Scheduler Worker Worker Worker Worker LocalCluster creates everything
for you. It will break down a large CPU into multiple workers withy multiple threads as this can be more performant.
Client Local Cluster Scheduler Worker Worker Worker Worker
Get logs
Scaling
How do I get more resource? Moving beyond a single
machine
SSH ... You could SSH to a bunch of machines
and start the Dask components manually.
SSHCluster Or you could use SSHCluster which will bootstrap a
cluster for you on a list of machines. All you need is passwordless SSH configured for each machine.
None
Deployment Workshop Thank you! @_jacobtomlinson