Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Deploying Dask Distributed
Search
Jacob Tomlinson
May 19, 2021
Technology
0
220
Deploying Dask Distributed
Jacob Tomlinson
May 19, 2021
Tweet
Share
More Decks by Jacob Tomlinson
See All by Jacob Tomlinson
Dask Demo Day - How RAPIDS users use Dask
jacobtomlinson
0
20
Deploying multi-GPU workloads on Kubernetes in Python
jacobtomlinson
0
180
Overview of Dask on Kubernetes
jacobtomlinson
0
140
k8s-batch-sig - Dask on Kubernetes
jacobtomlinson
0
100
Deploying multi-GPU workloads on Kubernetes in Python
jacobtomlinson
0
130
Dask Overview - Stanford Legion all-hands
jacobtomlinson
0
180
Container Canary - Lightning Talk
jacobtomlinson
0
58
Parallelizing Your ETL with Dask on Kubeflow
jacobtomlinson
0
800
GPU accelerating your computation in Python
jacobtomlinson
0
240
Other Decks in Technology
See All in Technology
Amazon FSx for NetApp ONTAPを利用するにあたっての要件整理と設計のポイント
non97
1
160
日経電子版におけるリアルタイムレコメンドシステム開発の事例紹介/nikkei-realtime-recommender-system
yng87
1
500
使えそうで使われないCloudHSM
maikamibayashi
0
170
生成AIとAWS CDKで実現! 自社ブログレビューの効率化
ymae
2
330
ユーザーの購買行動モデリングとその分析 / dsc-purchase-analysis
cyberagentdevelopers
PRO
2
100
「視座」の上げ方が成人発達理論にわかりやすくまとまってた / think_ perspective_hidden_dimensions
shuzon
2
4.1k
GitHub Universe: Evaluating RAG apps in GitHub Actions
pamelafox
0
180
omakaseしないための.rubocop.yml のつくりかた / How to Build Your .rubocop.yml to Avoid Omakase #kaigionrails
linkers_tech
3
740
AWS CDKでデータリストアの運用、どのように設計する?~Aurora・EFSの実践事例を紹介~/aws-cdk-data-restore-aurora-efs
mhrtech
4
650
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
49k
話題のGraphRAG、その可能性と課題を理解する
hide212131
4
1.5k
急成長中のWINTICKETにおける品質と開発スピードと向き合ったQA戦略と今後の展望 / winticket-autify
cyberagentdevelopers
PRO
1
160
Featured
See All Featured
Adopting Sorbet at Scale
ufuk
73
9k
For a Future-Friendly Web
brad_frost
175
9.4k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.8k
Raft: Consensus for Rubyists
vanstee
136
6.6k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Build The Right Thing And Hit Your Dates
maggiecrowley
32
2.4k
RailsConf 2023
tenderlove
29
880
We Have a Design System, Now What?
morganepeng
50
7.2k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Scaling GitHub
holman
458
140k
Transcript
Deployment Workshop Deploying Dask Distributed Jacob Tomlinson
Dask Distributed A centrally managed, distributed, dynamic task scheduler
Dask Overview
None
Worker Worker Worker Scheduler Client Protocols TCP UCX Websocket Dask
components can communicate via a variety of different protocols.
Scheduler Starting a scheduler
Connecting a worker Worker Scheduler
Client Scheduler Worker Connecting a client
Client Scheduler Worker Submitting work
Dask Dashboard
JupyterLab Extension
Cluster Managers Utility classes to simplify cluster creation
Local Cluster Scheduler Worker Worker Worker Worker LocalCluster creates everything
for you. It will break down a large CPU into multiple workers withy multiple threads as this can be more performant.
Client Local Cluster Scheduler Worker Worker Worker Worker
Get logs
Scaling
How do I get more resource? Moving beyond a single
machine
SSH ... You could SSH to a bunch of machines
and start the Dask components manually.
SSHCluster Or you could use SSHCluster which will bootstrap a
cluster for you on a list of machines. All you need is passwordless SSH configured for each machine.
None
Deployment Workshop Thank you! @_jacobtomlinson