Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
170
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
65
What Python Can Learn From Other Languages (with notes)
coderanger
0
170
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
240
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
64
How to look at space: PyCon AU
coderanger
0
110
Swiss Army Django: Small Footprint ETL
coderanger
0
94
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
86
Minimum Viable Kubernetes
coderanger
0
41
Minimum Viable Kubernetes (with notes)
coderanger
0
460
Other Decks in Programming
See All in Programming
Vue・React マルチプロダクト開発を支える Vite
andpad
0
110
AIを活用し、今後に備えるための技術知識 / Basic Knowledge to Utilize AI
kishida
22
5.7k
Amazon RDS 向けに提供されている MCP Server と仕組みを調べてみた/jawsug-okayama-2025-aurora-mcp
takahashiikki
1
110
「手軽で便利」に潜む罠。 Popover API を WCAG 2.2の視点で安全に使うには
taitotnk
0
850
How Android Uses Data Structures Behind The Scenes
l2hyunwoo
0
430
JSONataを使ってみよう Step Functionsが楽しくなる実践テクニック #devio2025
dafujii
1
530
Processing Gem ベースの、2D レトロゲームエンジンの開発
tokujiros
2
130
個人軟體時代
ethanhuang13
0
320
さようなら Date。 ようこそTemporal! 3年間先行利用して得られた知見の共有
8beeeaaat
3
1.4k
AI時代のUIはどこへ行く?
yusukebe
18
8.8k
テストカバレッジ100%を10年続けて得られた学びと品質
mottyzzz
2
590
アセットのコンパイルについて
ojun9
0
120
Featured
See All Featured
Site-Speed That Sticks
csswizardry
10
810
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
How STYLIGHT went responsive
nonsquared
100
5.8k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
Optimizing for Happiness
mojombo
379
70k
Typedesign – Prime Four
hannesfritz
42
2.8k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.9k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?