Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WorkFlowEngine Digdagの導入

amesho
March 07, 2017

WorkFlowEngine Digdagの導入

amesho

March 07, 2017
Tweet

More Decks by amesho

Other Decks in Programming

Transcript

  1. ϫʔΫϑϩʔͷఆٛ ม਺ͷຒΊࠐΈ timezone: UTC + task_name: sh> /bin/touch /tmp/hello_digdag +

    task_name2: echo> ${moment(session_time).utc().format("YYYY-MM-DD HH:mm:ss")}
  2. ϫʔΫϑϩʔͷఆٛ ฒྻ࣮ߦ timezone: UTC +run: _parallel: true +task_name: sh>: /usr/bin/touch

    /tmp/hello_digdag +task_name2: sh>: /usr/bin/touch /tmp/hello_digdag2 +task_name3: sh>: /usr/bin/touch /tmp/hello_digdag3
  3. ϫʔΫϑϩʔͷఆٛ ฒྻ࣮ߦ ฒྻʹ࣮ߦ͞Ε·͢ʢॱ൪͸อূ͞Εͳ͍ 2017-03-07 09:23:34 +0900 [INFO] (0018@+hoge+run+task_name2): sh>: /usr/bin/touch

    /tmp/hello_digdag2 2017-03-07 09:23:34 +0900 [INFO] (0017@+hoge+run+task_name): sh>: /usr/bin/touch /tmp/hello_digdag 2017-03-07 09:23:34 +0900 [INFO] (0019@+hoge+run+task_name3): sh>: /usr/bin/touch /tmp/hello_digdag3
  4. ϫʔΫϑϩʔͷఆٛ Τϥʔ௨஌ 2017-03-07 09:30:34 +0900 [INFO] (0017@+error+run+task_name): sh>: /usr/touch /tmp/hello_digdag

    /bin/sh: line 1: /usr/touch: No such file or directory 2017-03-07 09:30:34 +0900 [ERROR] (0017@+error+run+task_name): Task failed with unexpected error: Command failed with code 127 java.lang.RuntimeException: Command failed with code 127 at io.digdag.standards.operator.ShOperatorFactory$ShOperator.runTask(ShOperatorFactory.java:143) at io.digdag.util.BaseOperator.run(BaseOperator.java:35) at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:314) at io.digdag.cli.Run$OperatorManagerWithSkip.callExecutor(Run.java:674) at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:255) at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:138) at io.digdag.core.agent.LocalWorkspaceManager.withExtractedArchive(LocalWorkspaceManager.java:25) at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:136) at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:120) at io.digdag.cli.Run$OperatorManagerWithSkip.run(Run.java:656) at io.digdag.core.agent.MultiThreadAgent.lambda$run$0(MultiThreadAgent.java:95) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-03-07 09:30:35 +0900 [INFO] (0017@+error+run^error): sh>: /usr/bin/touch /tmp/error 2017-03-07 09:30:35 +0900 [INFO] (0017@+error^failure-alert): type: notify error: * +error+run+task_name: Command failed with code 127 (runtime)
  5. εέδϡʔϦϯάϫʔΫϑϩʔ εέδϡʔϥͷεςʔλε֬ೝ Schedules (1 entries): mydag: daily>: "19:40:00" first session

    time: 2017-03-07 00:00:00 +0900 first scheduled to run at: 2017-03-07 19:40:00 +0900 (in 9h 4m 52s)
  6. εέδϡʔϦϯάϫʔΫϑϩʔ ࣮ߦ࣌ؒʹؔ͢ΔΞϥʔτ timezone: UTC schedule: daily>: 07:00:00 sla: # triggers

    this task at 02:00 time: 02:00 +notice: sh>: notice.sh +long_running_job: sh>: long_running_job.sh
  7. ղܾ͞ΕΔ໰୊ ૉఢͳґଘؔ܎ʹ͋Δcron server A # 4࣌ʹىಈͯ͠ڪΒ͘2࣌ؒͰऴΘΔॲཧ 0 4 * *

    * /path/to/your_script.rb server B # server Aͷॲཧ͕6࣌ʹऴ͍ྃͯ͠Δ͸ͣͳͷͰͦΕΛݩʹॲཧ # ࠶࣮ߦ͢Δͱࠅ͍͜ͱʹͳΔͷͰ஫ҙ 0 8 * * * /path/to/your_script.rb
  8. ԋࢉࢠ εΫϦϓτԋࢉࢠ • sh>: γΣϧ • py>: PythonεΫϦϓτ • rb>:

    RubyεΫϦϓτ • embulk>: embulkͷσʔλసૹ • ݱࡏഇࢭ͞Ε͍ͯ·͢