Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Starts Airflow task execution directly from the...
Search
Lee Wei
May 08, 2024
Technology
0
38
Starts Airflow task execution directly from the triggerer
Lee Wei
May 08, 2024
Tweet
Share
More Decks by Lee Wei
See All by Lee Wei
Unlocking Python's Core Magic
leew
0
100
Unleash the Chaos - Developing a Linter for Un-Pythonic Code!
leew
0
70
Intro to Airflow - From Zero to Hero
leew
0
760
Does Kobayashi-San write code vulnerable to SQL injection?
leew
0
69
Back to the last time we were here
leew
0
10
Atomic Commits - An Easy & Proven Way to Manage & Automate Release Process
leew
0
380
Python Table Manners @ Taichung.py
leew
0
310
Python Table Manners- Cut the Cookie Gracefully @ Hualien py
leew
0
170
小林的程式碼會不會遇到 SQL Injection
leew
1
96
Other Decks in Technology
See All in Technology
Sansanにおける全社横断データ分析基盤の挑戦と未来 / Challenges and Future of Cross-Organizational Data Analytics Platform at Sansan
sansan_randd
2
300
Report of JAWS PANKRATION 2024, a global online event using Amazon IVS
yoshimi0227
1
130
たった一人で始めた音楽制作が気がついたら会社公認の部活動になっていた話〜組織の垣根を超えるコラボレーションを実現するには〜 / On-KAG-bu
piyonakajima
0
130
【shownet.conf_】ネットワークテストの最適化と利便性の追求
shownet
PRO
0
230
How CERN serves 1EB of data via FUSE
ennael
PRO
0
15k
OPENLOGI Company Profile for engineer
hr01
1
12k
SQLによるオブザーバビリティの進化とClickHouseの実力
mikimatsumoto
0
150
Interfacing Kernel C APIs from Rust
ennael
PRO
0
210
BLEA v3.0.0の新しいベストプラクティスを取り入れた効率的なAWS CDK開発/jawsug_cdk16
mhrtech
3
350
AWS Lambdaで実現するスケーラブルで低コストなWebサービス構築/YAPC::Hakodate2024
fujiwara3
3
660
【ログ分析勉強会】EDR ログで内部不正を検出できるのか、Copilot に聞いてみた
hssh2_bin
2
260
k6を活用した再現性・拡張性の高い負荷試験基盤の構築
biwashi
11
2.8k
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
7.5k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
5
210
The Straight Up "How To Draw Better" Workshop
denniskardys
231
130k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
30
2.6k
Ruby is Unlike a Banana
tanoku
96
11k
Build The Right Thing And Hit Your Dates
maggiecrowley
31
2.3k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
22k
YesSQL, Process and Tooling at Scale
rocio
167
14k
Scaling GitHub
holman
458
140k
Into the Great Unknown - MozCon
thekraken
30
1.4k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.3k
Transcript
Starts Air fl ow task execution directly from the triggerer
$ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__
= [ Software Engineer @ Astronomer, Committer @ Apache Airflow, Volunteers @ PyCon Taiwan, Maintainer of commitizen-tools, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
File "weilee.py", line 1 __name__ = 李唯 / Wei Lee
^^^ SyntaxError: invalid syntax $ python weilee.py
None
How did deferrable operators work before this change?
What has been changed?
What has been changed?
What it could be in the future
But why?
"execute" methods are not always needed This is the only
logic when deferrable is set to true and does not need to be executed in the worker.
How does it affect DAG authors? More ef fi cient
operators to come
How does it affect operator authors? A new way to
implement operators in an async manner
The Original Way
The New Way
How does it work under the hook?
How does it work under the hook?
How does it work under the hook?
Limitations • It is necessary to go back to workers
to run the next_method (e.g., "execute_complete") • This feature does not yet fully support MappedOperator.