Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Starts Airflow task execution directly from the...
Search
Lee Wei
May 08, 2024
Technology
0
100
Starts Airflow task execution directly from the triggerer
Lee Wei
May 08, 2024
Tweet
Share
More Decks by Lee Wei
See All by Lee Wei
20250328 Airflow 3.0 The First Glance
leew
0
61
踏入開源的第一步
leew
0
140
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
0
55
Unlocking Python's Core Magic
leew
1
240
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
1
150
Intro to Airflow - From Zero to Hero
leew
0
920
Does Kobayashi-San write code vulnerable to SQL injection?
leew
0
110
Back to the last time we were here
leew
0
30
Atomic Commits - An Easy & Proven Way to Manage & Automate Release Process
leew
0
450
Other Decks in Technology
See All in Technology
Computer Use〜OpenAIとAnthropicの比較と将来の展望〜
pharma_x_tech
6
950
製造業向けIoTソリューション提案資料.pdf
haruki_uiru
0
130
LINE 購物幕後推手
line_developers_tw
PRO
0
310
LLM アプリケーションのためのクラウドセキュリティ - CSPM の実装ポイント-
osakatechlab
0
130
OpenLane-V2ベンチマークと代表的な手法
kzykmyzw
0
150
QA/SDETの現在と、これからの挑戦
imtnd
0
210
SREからゼロイチプロダクト開発へ ー越境する打席の立ち方と期待への応え方ー / Product Engineering Night #8
itkq
2
1.1k
Perl歴約10年のエンジニアがフルスタックTypeScriptに出会ってみた
papix
1
260
Linuxのパッケージ管理とアップデート基礎知識
go_nishimoto
1
700
ビジネスとデザインとエンジニアリングを繋ぐために 一人のエンジニアは何ができるか / What can a single engineer do to connect business, design, and engineering?
kaminashi
2
860
Новые мапы в Go. Вова Марунин, Clatch, МТС
lamodatech
0
1.5k
クラウド開発環境Cloud Workstationsの紹介
yunosukey
0
220
Featured
See All Featured
The Invisible Side of Design
smashingmag
299
50k
What's in a price? How to price your products and services
michaelherold
245
12k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
23
2.7k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
41
2.3k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
780
Java REST API Framework Comparison - PWX 2021
mraible
31
8.5k
A designer walks into a library…
pauljervisheath
205
24k
A Modern Web Designer's Workflow
chriscoyier
693
190k
GraphQLの誤解/rethinking-graphql
sonatard
71
10k
A better future with KSS
kneath
239
17k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
How to Ace a Technical Interview
jacobian
276
23k
Transcript
Starts Air fl ow task execution directly from the triggerer
$ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__
= [ Software Engineer @ Astronomer, Committer @ Apache Airflow, Volunteers @ PyCon Taiwan, Maintainer of commitizen-tools, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
File "weilee.py", line 1 __name__ = 李唯 / Wei Lee
^^^ SyntaxError: invalid syntax $ python weilee.py
None
How did deferrable operators work before this change?
What has been changed?
What has been changed?
What it could be in the future
But why?
"execute" methods are not always needed This is the only
logic when deferrable is set to true and does not need to be executed in the worker.
How does it affect DAG authors? More ef fi cient
operators to come
How does it affect operator authors? A new way to
implement operators in an async manner
The Original Way
The New Way
How does it work under the hook?
How does it work under the hook?
How does it work under the hook?
Limitations • It is necessary to go back to workers
to run the next_method (e.g., "execute_complete") • This feature does not yet fully support MappedOperator.