Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
160
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
60
What Python Can Learn From Other Languages (with notes)
coderanger
0
160
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
220
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
58
How to look at space: PyCon AU
coderanger
0
100
Swiss Army Django: Small Footprint ETL
coderanger
0
88
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
81
Minimum Viable Kubernetes
coderanger
0
37
Minimum Viable Kubernetes (with notes)
coderanger
0
440
Other Decks in Programming
See All in Programming
「Cursor/Devin全社導入の理想と現実」のその後
saitoryc
0
140
Select API from Kotlin Coroutine
jmatsu
1
190
Java on Azure で LangGraph!
kohei3110
0
170
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
1
680
生成AIで日々のエラー調査を進めたい
yuyaabo
0
640
LINEヤフー データグループ紹介
lycorp_recruit_jp
0
870
Go1.25からのGOMAXPROCS
kuro_kurorrr
1
800
技術同人誌をMCP Serverにしてみた
74th
0
290
Cline指示通りに動かない? AI小説エージェントで学ぶ指示書の書き方と自動アップデートの仕組み
kamomeashizawa
1
570
ruby.wasmで多人数リアルタイム通信ゲームを作ろう
lnit
2
260
XSLTで作るBrainfuck処理系
makki_d
0
210
git worktree × Claude Code × MCP ~生成AI時代の並列開発フロー~
hisuzuya
1
450
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
The Straight Up "How To Draw Better" Workshop
denniskardys
233
140k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
We Have a Design System, Now What?
morganepeng
53
7.7k
Rebuilding a faster, lazier Slack
samanthasiow
81
9.1k
Being A Developer After 40
akosma
90
590k
VelocityConf: Rendering Performance Case Studies
addyosmani
330
24k
Scaling GitHub
holman
459
140k
Six Lessons from altMBA
skipperchong
28
3.8k
Agile that works and the tools we love
rasmusluckow
329
21k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?