Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
170
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
67
What Python Can Learn From Other Languages (with notes)
coderanger
0
170
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
240
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
66
How to look at space: PyCon AU
coderanger
0
110
Swiss Army Django: Small Footprint ETL
coderanger
0
95
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
86
Minimum Viable Kubernetes
coderanger
0
43
Minimum Viable Kubernetes (with notes)
coderanger
0
470
Other Decks in Programming
See All in Programming
意外と簡単!?フロントエンドでパスキー認証を実現する WebAuthn
teamlab
PRO
2
780
MCPでVibe Working。そして、結局はContext Eng(略)/ Working with Vibe on MCP And Context Eng
rkaga
5
2.3k
Platformに“ちょうどいい”責務ってどこ? 関心の熱さにあわせて考える、責務分担のプラクティス
estie
1
240
How Android Uses Data Structures Behind The Scenes
l2hyunwoo
0
490
🔨 小さなビルドシステムを作る
momeemt
4
690
@Environment(\.keyPath)那么好我不允许你们不知道! / atEnvironment keyPath is so good and you should know it!
lovee
0
130
1から理解するWeb Push
dora1998
7
2k
今だからこそ入門する Server-Sent Events (SSE)
nearme_tech
PRO
3
260
Updates on MLS on Ruby (and maybe more)
sylph01
1
190
スケールする組織の実現に向けた インナーソース育成術 - ISGT2025
teamlab
PRO
2
180
Azure SRE Agentで運用は楽になるのか?
kkamegawa
0
2.6k
プロポーザル駆動学習 / Proposal-Driven Learning
mackey0225
2
1.3k
Featured
See All Featured
Optimising Largest Contentful Paint
csswizardry
37
3.4k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
Intergalactic Javascript Robots from Outer Space
tanoku
272
27k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Embracing the Ebb and Flow
colly
87
4.8k
Optimizing for Happiness
mojombo
379
70k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
How to Ace a Technical Interview
jacobian
279
23k
Producing Creativity
orderedlist
PRO
347
40k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Thoughts on Productivity
jonyablonski
70
4.8k
Statistics for Hackers
jakevdp
799
220k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?