Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deploying Dask Distributed

Deploying Dask Distributed

Jacob Tomlinson

May 19, 2021
Tweet

More Decks by Jacob Tomlinson

Other Decks in Technology

Transcript

  1. Worker Worker Worker Scheduler Client Protocols TCP UCX Websocket Dask

    components can communicate via a variety of different protocols.
  2. Local Cluster Scheduler Worker Worker Worker Worker LocalCluster creates everything

    for you. It will break down a large CPU into multiple workers withy multiple threads as this can be more performant.
  3. SSH ... You could SSH to a bunch of machines

    and start the Dask components manually.
  4. SSHCluster Or you could use SSHCluster which will bootstrap a

    cluster for you on a list of machines. All you need is passwordless SSH configured for each machine.