Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
160
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
61
What Python Can Learn From Other Languages (with notes)
coderanger
0
160
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
220
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
59
How to look at space: PyCon AU
coderanger
0
100
Swiss Army Django: Small Footprint ETL
coderanger
0
90
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
82
Minimum Viable Kubernetes
coderanger
0
38
Minimum Viable Kubernetes (with notes)
coderanger
0
450
Other Decks in Programming
See All in Programming
GPUを計算資源として使おう!
primenumber
1
200
AI時代の『改訂新版 良いコード/悪いコードで学ぶ設計入門』 / ai-good-code-bad-code
minodriven
22
9k
LT 2025-06-30: プロダクトエンジニアの役割
yamamotok
0
810
NPOでのDevinの活用
codeforeveryone
0
870
20250704_教育事業におけるアジャイルなデータ基盤構築
hanon52_
5
910
ペアプロ × 生成AI 現場での実践と課題について / generative-ai-in-pair-programming
codmoninc
2
20k
MCPを使ってイベントソーシングのAIコーディングを効率化する / Streamlining Event Sourcing AI Coding with MCP
tomohisa
0
160
20250708_JAWS_opscdk
takuyay0ne
2
120
型で語るカタ
irof
0
520
イベントストーミング図からコードへの変換手順 / Procedure for Converting Event Storming Diagrams to Code
nrslib
2
970
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
330
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
580
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Writing Fast Ruby
sferik
628
62k
The Cult of Friendly URLs
andyhume
79
6.5k
Why Our Code Smells
bkeepers
PRO
336
57k
Six Lessons from altMBA
skipperchong
28
3.9k
Optimizing for Happiness
mojombo
379
70k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Designing for humans not robots
tammielis
253
25k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
510
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?