Orchestration Still Matters: Apache Airflow in the Age of AI

Software Freedom Day · Bukidnon 2026

Transcript

1. Orchestration Still Matters: Apache Airflow in the Age of AI Software Freedom Day · Bukidnon 2026 Wei Lee wei-lee.me
2. 02 / 114
3. This is Airflow in the Age of AI. 03 / 114
4. Q.E.D. 04 / 114
5. But we still have 29 minutes. 🤔 05 / 114
6. Or maybe we start with what Airflow is. 06 / 114
7. 01 / 06 Rewind Where this thing came from. REWIND MADE IN HEAVEN WHY IT MATTERS AI FUTURE OPEN SOURCE 07 / 114
8. Let's go back a bit. ／ ・ ・ Homura Akemi · ©Magica Quartet Aniplex Madoka Partners MBS 08 / 114
9. 09 / 114
10. 10 / 114
11. 11 / 114
12. 12 / 114
13. Apache Airflow Taiwan User Group @ Taipei.py 13 / 114
14. 2014 Airflow is born. At Airbnb. 14 / 114
15. This is what it looked like. 15 / 114
16. So what is it? 16 / 114
17. In the author's own words. medium.com/airbnb-engineering/airflow-a-workflow-management-platform-46318b977fd8 17 / 114
18. Problem to solve. Scripts calling scripts, dependencies everywhere. 18 / 114
19. A task is one step. extract 19 / 114
20. Tasks have an order. extract transform load 20 / 114
21. That graph is a DAG. report extract transform alert 21 / 114
22. Directed Acyclic Graph Arrows point one way. Nothing loops back. 22 / 114
23. cron says when. Airflow says when, and in what order, and what to do when it breaks. 23 / 114
24. And this is a real one. 24 / 114
25. That's orchestration. 25 / 114
26. Not "run this at 3am". Run these, in this order, and cope when one fails. 26 / 114
27. Then it grew up. 27 / 114
28. 2016 Airflow enters the Apache Incubator. 31 March 2016 28 / 114
29. 2019 A Top-Level Project of the Apache Software Foundation. announced 8 January 2019 29 / 114
30. I guess we have enough history lesson. 30 / 114
31. 02 / 06 Made in Heaven Eleven years, at speed. REWIND MADE IN HEAVEN WHY IT MATTERS AI FUTURE OPEN SOURCE 31 / 114
32. Time, accelerated. Made in Heaven · © Hirohiko Araki/LUCKY LAND COMMUNICATIONS 32 / 114
33. 2 days ago. 33 / 114
34. One thing did change. 34 / 114
35. DAG → Dag 35 / 114
36. Not a graph theory lecture. A Dag is just a workflow in Airflow. 36 / 114
37. 03 / 06 Why It Still Matters Three words. They are the whole argument. REWIND MADE IN HEAVEN WHY IT MATTERS AI FUTURE OPEN SOURCE 37 / 114
38. Your first AI feature takes an afternoon. 38 / 114
39. The second week is the problem. 39 / 114
40. In 2024, a chatbot got it wrong. It promised a refund the airline's own policy did not offer. 40 / 114
41. And the airline paid. A tribunal held the airline responsible for its chatbot. Moffatt v. Air Canada · 2024 BCCRT 149 · BC Civil Resolution Tribunal, 14 Feb 2024 · damages, interest and 41 / 114 fees awarded
42. Every pipeline must be... 01 02 03 Reliable Observable Retryable It runs. Every time. Even You can see what is Failures are normal. You when things break happening. What failed. can pick up where it upstream. And why. broke. 42 / 114
43. Reliable 43 / 114
44. It runs. Every time. Even when things break upstream. 44 / 114
45. Observable 45 / 114
46. "It failed" is not an answer. Which step? How long? What did it get back? 46 / 114
47. This is what that looks like. 47 / 114
48. Retryable 48 / 114
49. He who has never seen a network timeout, cast the first stone. 49 / 114
50. Got paged at 3 a.m. for a network error. "This Is Fine" · © KC Green, Gunshow #648 (2013) 50 / 114
51. Airflow's got you covered. 51 / 114
52. Retry the step. Not the pipeline. Step 7 of 9 failed. Run step 7. 52 / 114
53. 04 / 06 Let's talk about AI Where the model actually goes. REWIND MADE IN HEAVEN WHY IT MATTERS AI FUTURE OPEN SOURCE 53 / 114
54. Everyone says context now. 54 / 114
55. It's a fancy word for data. System prompt ──▶ a string you wrote Conversation ──▶ rows you stored Tool output ──▶ an API response Retrieved chunks ──▶ a database query All of it is data. All of it has to arrive. 55 / 114
56. Say you build a chatbot for your docs. fetch docs chunk embed index 56 / 114
57. Then someone asks a question. question retrieve ask model answer 57 / 114
58. Look at what those steps are. fetch docs ──▶ an external service chunk ──▶ data processing embed ──▶ a model index ──▶ an external service retrieve ──▶ an external service ask model ──▶ a model 58 / 114
59. The abstraction held. 59 / 114
60. Volker Janz orchestrated Airflow through Minecraft. Volker Janz · "The Rise of Abstraction in Dag Authoring: From YAML to Minecraft" 60 / 114
61. And that is why it still fits. Powerful 10 years ago. Still powerful now. 61 / 114
62. Airflow orchestrates. 62 / 114
63. The core learned scheduling. Everything else plugs in. 63 / 114
64. Airflow Registry 64 / 114
65. Yes, there is one for that. ／ ・ ・ Fate/Zero · ©Nitroplus TYPE-MOON ufotable FZPC 65 / 114
66. So when AI showed up... 66 / 114
67. We added AI providers. openai · anthropic · cohere pgvector · qdrant · weaviate 67 / 114
68. And also... 68 / 114
69. common.ai 69 / 114
70. No need to pick a vendor. One task. Swap the model underneath. 70 / 114
71. @task.llm — one call @task.agent — a whole agent 71 / 114
72. This is the whole thing. @task.llm( llm_conn_id="ollama_local", system_prompt="Reply in one sentence.", ) def draft_reply(ticket: str): return f"Draft a reply to: {ticket}" 72 / 114
73. So a model call gets retries and logs, like anything else. 73 / 114
74. LLMRetryPolicy The retry decision, made by a model. With a deterministic fallback. 74 / 114
75. A Spark task ran out of memory. A number cannot read a stack trace. It re-ran with 16g instead of 11g. 75 / 114
76. Or ask a human. ApprovalOperator the Dag waits for a person. 76 / 114
77. The Dag waits. 77 / 114
78. 05 / 06 What's coming Two things, honestly labelled. REWIND MADE IN HEAVEN WHY IT MATTERS AI FUTURE OPEN SOURCE 78 / 114
79. An agent doesn't know how many steps it needs. 79 / 114
80. It tries, looks, and decides what's next. 80 / 114
81. AIP-111 · Task Loops @task def refine(draft): ... refine.loop(max_iterations=5, until=good_enough) Accepted · targeting 3.4.0 81 / 114
82. And it doesn't know its own shape. 82 / 114
83. AIP-113 · Dynamic Task Groups @task_group( catalog=[refund, escalate, enrich], planner=decide_shape, max_selected=3, ) def respond(): ... Draft · targeting 3.5.0 83 / 114
84. The planner can be a model. 84 / 114
85. AIP-112 Task Steps AIP-115 On-Demand Task Sections …and more to come. 85 / 114
86. 06 / 06 Open Source, Open Governance Who decides what Airflow becomes. REWIND MADE IN HEAVEN WHY IT MATTERS AI FUTURE OPEN SOURCE 86 / 114
87. Who owns Apache Airflow? 87 / 114
88. The Apache Software Foundation. Not a company. Not a vendor. 88 / 114
89. Open source is the licence. Open governance is who gets to decide. 89 / 114
90. You contribute. You gain trust. Then your voice carries. 90 / 114
91. Writing code got easier. 91 / 114
92. Earning trust got harder. 92 / 114
93. And that is why Airflow still stands strong. 93 / 114
94. We are open governance. We stay open source. 94 / 114
95. The community trusts us. And we keep strengthening that trust. 95 / 114
96. Which is why open source matters more than ever. 96 / 114
97. When anyone can generate code, being able to read it is the only guarantee left. 97 / 114
98. Join us. 98 / 114
99. 99 / 114
100. You don't need to be good at Python. 100 / 114
101. Report what broke. Fix a doc. Translate the interface. 101 / 114
102. And you know what? 102 / 114
103. A Dag doesn't need to be all Python. Since Airflow 3.3, a worker can run task code written in another language. 103 / 114
104. Go · Java · TypeScript …and more to come. © Lucasfilm Ltd. / The Walt Disney Company 104 / 114
105. Where we went. 2014 — steps, in order. 105 / 114
106. 2026 — steps, in order, and the shape can change as it runs. 106 / 114
107. Orchestration still matters. 107 / 114
108. Reliable. Observable. Retryable. 108 / 114
109. And we keep making that easier to see, and easier to use. 109 / 114
110. Wei Lee / 李唯 Apache Airflow PMC Member PyCon TW Volunteer wei-lee.me 110 / 114
111. PyCon TW Oct 17–18, 2026 Taipei Medical University tw.pycon.org/2026 111 / 114
112. And a small Airflow meetup inside it. With friends from the Apache Airflow Korea User Group. 112 / 114
113. Salamat! 🇵🇭 113 / 114
114. Questions? Wei Lee wei-lee.me