digdag-Introduction

Digdagを本番導入したので社内勉強会で発表した資料です。

Transcript

1. Introduction of Digdag.
2. Who am I. • Twitter/GitHub Account • @smdmts • Main Fields • Scala & Java8 & React.js & Python • DDD CleanArchitecture @ akka-http • Workflow • Hive/Presto
3. Agenda • Digdag = Workflow automation system.
4. ϫʔΫϑϩʔΤϯδϯͷओͳཁ݅ • ఆظతͳλεΫͷ࣮ߦ • λεΫͷॱ࣮࣍ߦ • γεςϜؒͷσʔλࣗಈ࿈ܞ • όονʹΑΔσʔλूܭͷࣗಈԽ • όονδϣϒ׬ྃޙͷϝʔϧ/SlackͳͲ΁௨஌ • ϦτϥΠ࣌ʹ͓͚Δႈ౳ੑ
5. Digdagͱ͸ • DAGʢDirected acyclic graph)Λ࣮ݱ͢ΔϫʔΫϑ ϩʔΤϯδϯ • YAMLͰDAGΛදݱ͢ΔͨΊఆٛମ͸Git؅ཧՄೳ ʢWorkflow as Codeʣ • LocalϞʔυͰ։ൃ͠ɺClient/ServerϞʔυͰຊ൪ ͰՔಇͤ͞Δ • Python/Ruby/Bash/DockerͳͲͰαϒλεΫ͕࣮ ߦՄೳ
6. Digdagͱ͸ʢClient/ServerϞʔυʣ • PostgreSQLͰQueueΛ࣮ݱ͍ͯ͠Δ • αϒλεΫຖͰQueueԽ͞Ε͓ͯΓαʔόෳ਺ ୆Ͱ࣮ߦ؀ڥ͕εέʔϧՄೳ • Workflowͷ࣮ମ͸PostgreSQLʹӬଓԽ͞ΕΔ • Client͕ίϚϯυͰWorkflowΛpush͢Δ • Workflow͸ੈ୅؅ཧ͞ΕΔ • ࠶ىಈෆཁͰδϣϒొ࿥ʗ࠶࣮ߦՄೳ
7. DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ ʢwikipedia) άϥϑཧ࿦ʹ͓͚Δด࿏ͷͳ͍༗޲άϥϑͷࣄ ༗޲άϥϑ͸௖఺ͱ༗޲ลʢํ޲Λࣔ͢໼ҹ෇͖ ͷลʣ͔ΒͳΓɺล͸௖఺ಉ࢜Λͭͳ͙͕ɺ͋Δ ௖఺ v ͔Βग़ൃ͠ɺลΛͨͲΓɺ௖఺ v ʹ໭ͬͯ ͜ͳ͍ͷ͕༗޲ඇ८ճάϥϑͰ͋Δɻ
8. DAG (Directed acyclic graph)ͱ͸ • DAGʢ༗޲ඇ८ճάϥϑʣͱ͸ • ୺తʹݴ͏ͱऴ఺͕ଘࡏ͠։࢝఺ʹ໭ͬͯ͜ͳ ͍άϥϑ
9. DigdagͰͷදݱํ๏ • YAMLͰΦϖϨʔλΛఆٛ timezone: UTC _export: mail: ..... # Definition of mail +step1_input: py>: tasks.load _error: mail>: body.txt subject: input error! to: [me@example.com] +step2_process: sh>: echo process. +step2_report: sh>: echo report.
10. δϣϒϑϩʔߏ੒ུ֓ਤ
11. δϣϒϑϩʔߏ੒ུ֓ਤʹ͓͚Δఆٛ timezone: UTC +prepare_load_aws_env: py>: tasks.load_aws_env +step1_produce_tasks: # Generate SQL Queries for Redshift. !include : 'child_tasks/produce_tasks/bootstrap.dig' +step2_create_redshift_buffer: # Internal S3 or TreasureData to Redshift temporary buffer. !include : 'child_tasks/create_redshift_buffer/bootstrap.dig' +step3_create_publisher_s3: # Create Redshift buffer to publisher s3 bucket. !include : 'child_tasks/create_publisher_s3/bootstrap.dig'
12. ։ൃ/ӡ༻ͯ͠Έͨײ૝ • Workflow͕ίʔυͰදݱ͞Ε σόοά΋༰қ ͳ ͷͰ ී௨ͷ։ൃͷϊϦ Ͱॱ൪ͱΤϥʔϋϯυϦ ϯάΛҙࣝͨ͠δϣϒΛΧδϡΞϧʹ࡞Εͨ • Πϯετʔϧͷ؆қੑ΍ɺ࠶ىಈෆཁͷδϣϒ࠶ ొ࿥ʗ࣮ߦՄೳͳͲɺಋೖ/։ൃ/ӡ༻ָ͕ʹͳΔ ͜ͱ͕ҙࣝͯ͠ઃܭ͞Ε͍ͯΔҹ৅ • ࣮ߦॱংɺฒྻԽɺΤϥʔϋϯυϦϯάͷ੍ޚ͕ ඇৗʹ༰қͳҝɺશόονܥΛDigdagʹҠ؅͢Δ ࣄΛܾఆ
13. ։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ • py operatorར༻࣌ʹগ͠ϋϚͬͨ • ςετίʔυΛॻͨ͘Ίʹ͸import digdagͷ ϞοΫίʔυ͕ඞཁ • !includeͰผσΟϨΫτϦʹdigΛஔ͘ͱ PythonεΫϦϓτؒͷґଘղܾͷ࣮૷͕ඞཁ • λεΫؒͷม਺࿈ܞdigdag.env.storeͰ͸ɺ શλεΫԣஅͰΩʔͷ໊લΛҰҙʹ͢Δඞཁ͋ ΓʢಉҰ໊শͰ͸্ॻ͖͕ൃੜ͢Δ৔߹༗Γʣ
14. ։ൃ࣌ʹൃੜͨ͠໰୊/՝୊ʢิ଍ʣ • rb operator͸ར༻ͯ͠·ͤΜ
15. ӡ༻࣌ʹൃੜͨ͠໰୊/՝୊ • ӡ༻Ͱཉ͍͠ػೳ͕͋Δঢ়گʢ։ൃதʁʣ • ֬ೝը໘ (ίϚϯυͰճආத) • ਐߦঢ়گɾ࣮ߦ݁ՌɾΤϥʔͳͲ • ϩάͷS3ӬଓԽ (S3FSͰճආத)
16. ·ͱΊ • ࣗಈԽ͸ਖ਼ٛʂ