Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intro to Airflow - From Zero to Hero

Lee Wei
February 17, 2024
760

Intro to Airflow - From Zero to Hero

Lee Wei

February 17, 2024
Tweet

Transcript

  1. $ cat weilee.py __name__ = 李唯 / Wei Lee __what_i_am_doing__

    = [ Software Engineer @ Astronomer, Volunteers @ PyCon Taiwan, Maintainer of commitizen-tools, ] __github__ = G Lee-W __linkedin__ = l clleew __site__ = p http://wei-lee.me
  2. File "weilee.py", line 1 __name__ = 李唯 / Wei Lee

    ^^^ SyntaxError: invalid syntax $ python weilee.py
  3. No

  4. The truth is I don't know. From what I heard,

    we could, but we don't do it this way
  5. What is Airflow Apache Airflow™ is an open-source platform for

    developing, scheduling, and monitoring batch-oriented workflows. - from the Official Site
  6. What Airflow is not? ❌ • The best framework to

    run event-based workflows infinitely. • The best streaming solution.
  7. Why Airflow Integrations with 3-party projects "JSCZUF "MJCBCB "NB[PO "QQSJTF

    "TBOB "SBOHP%# "QBDIF4QBSL "QBDIF1JOPU "QBDIF1JH "QBDIF-JWZ "QBDIF,ZMJO "QBDIF,BGLB "QBDIF)JWF "QBDIF)%'4 "QBDIF'MJOL "QBDIF%SVJE "QBDIF%SJMM "QBDIF$BTTBOESB "QBDIF#FBN %PDLFS %JTDPSE %JOHEJOH ECU %BUBEPH %BUBCSJDLT $PNNPO42- $PIFSF ,VCFSOFUFT $FMFSZ +JSB *#.$MPVEBOU )551 )BTIJDPSQ H31$ (PPHMF 'BDFCPPL '51 'BDFCPPL &YBTPM &MBTUJDTFBSDI 0QFO-JOFBHF 0QFO"* 0QFO'BB4 0%#$ /FPK .Z42- .POHP%# 8JO3. .442- 1431 .JDSPTPGU1PXFS4IFMM .JDSPTPGU"[VSF +FOLJOT +%#$ *."1 *OGMVY%# 1BQFSNJMM 1BHFSEVUZ 0SBDMF 0QTHFOJF 0QFO4FBSDI 4FHNFOU 4BNCB 4BMFTGPSDF 3FEJT 1SFTUP 1PTUHSF42- 1JOFDPOF 1H7FDUPS 5BCMFBV 5BCVMBS 44) 42-JUF 4OPXGMBLF 4.51 4MBDL 4JOHVMBSJUZ 4'51 4FOEHSJE 7FSUJDB 5SJOP 5FMFHSBN ;FOEFTL :BOEFY 8FBWJBUF
  8. How does Airflow work? DAG authors change DAGs code Web

    Server Metadata Database Scheduler Triggerer Worker DAG Processor
  9. How does Airflow work? Users trigger DAGs from WebUI Web

    Server Metadata Database Scheduler Triggerer Worker DAG Processor
  10. How does Airflow work? Users trigger DAGs from WebUI Web

    Server Metadata Database Scheduler Triggerer Worker DAG Processor
  11. How does Airflow work? Users trigger DAGs from WebUI Web

    Server Metadata Database Scheduler Triggerer Worker DAG Processor
  12. How does Airflow work? Users trigger DAGs from WebUI Web

    Server Metadata Database Scheduler Triggerer Worker DAG Processor