everything • Uses existing database - no new components needed, no extra operational burden • Plan to use row-level-locks in the DB • Will re-evaluate if performance/stress testing show the need
scheduling This removes the tie between parsing and scheduling that is still present • Run a mini scheduler in the worker after each task is completed A.K.A. "fast follow". Look at immediate down stream tasks of what just finished and see what we can schedule • Test it to destruction This is a big architectural change, we need to be sure it works well.
DAG files, serializes them in JSON format & saves them in the Metadata DB. • Lazy Loading of DAGs: Instead of loading an entire DagBag when the Webserver starts we only load each DAG on demand. This helps reduce Webserver startup time and memory. This reduction is notable with large number of DAGs. • Deploying new DAGs to Airflow - no longer requires long restarts of webserver (if DAGs are baked in Docker image) • Feature to use the “JSON” library of choice for Serialization (default is inbuilt ‘json’ library) • Paves way for DAG Versioning & Scheduler HA
Parsing and Serializing from the scheduling loop. • Scheduler will fetch DAGs from DB • DAG will be parsed, serialized and saved to DB by a separate component “Serializer”/ “Dag Parser” • This should reduce the delay in Scheduling tasks when the number of DAGs are large
viewing previous DagRuns too • Not possible to view the code associated with a specific DagRun Goal: • Support for storing multiple versions of Serialized DAGs • Baked-In Maintenance DAGs to cleanup old DagRuns & associated Serialized DAGs • Graph View shows the DAG associated with that DagRun
◦ Easier way to test Kubernetes Tests locally • Quarantined tests ◦ Process of fixing the Quarantined tests • Thinning CI image ◦ Move integrations out of the image (hadoop etc) • Automated System Tests (AIP-21)
allocate • With KEDA, queues are free! (can have 100 queues) • KEDA works with k8s deployments so any customization you can make in a k8s pod, you can make in a k8s queue (worker size, GPU, secrets, etc.)
modify certain parts of the pod, but many features of the k8s API are abstracted away • We did this because at the time the airflow community was not well acquainted with the k8s API • We want to enable users to modify their worker pods to better match their use-cases
config in their airflow.cfg • Given a path, the KubernetesExecutor will now parse the yaml file when launching a worker pod • Huge thank you to @davlum for this feature
official helm chart that we have used both in our enterprise and in our cloud offerings (thousands of deployments of varying sizes) • Users can turn on KEDA autoscaling through helm variables
to convert a function to an operator ➔ Simplified way of writing DAGs ➔ Pluggable XCom Storage engine Example: store and retrieve DataFrames on GCS or S3 buckets without boilerplate code
be unique It was often confusing, and there are better ways to do load balancing • Python 3 only Python 2.7 unsupported upstream since Jan 1, 2020 • "RBAC" UI is now the only UI. Was a config option before, now only option. Charts/data profiling removed due to security risks
Breaking changes should be avoided where we can – if upgrade is to difficult users will be left behind • Release "backport providers" to make new code layout available "now": • Before 2.0 we want to make sure we've fixed everything we want to remove or break. pip install apache-airflow-backport-providers-aws \ apache-airflow-backport-providers-google
kaxil
risatube
oomatomo
handy
bitkey
ysakotch
syossan27
murashotaro
cybozuinsideout
spatial_ai_network
yositosi
oniak3ibm
yuj1osm
hannesfritz
quramy
lara
scottboms
paigeccino
jcasabona
bkeepers
lemiorhan
maltzj
destraynor
mongodb
mraible
