Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Argo Workflow による機械学習ワークフロー管理
Search
Livesense Inc.
PRO
June 27, 2019
Technology
2
3.3k
Argo Workflow による機械学習ワークフロー管理
2019/06/27
Data Pipeline Casual Talk #3
Livesense Inc.
PRO
June 27, 2019
Tweet
Share
More Decks by Livesense Inc.
See All by Livesense Inc.
株式会社リブセンス会社紹介資料 / Invent the next common.
livesense
PRO
1
4.2k
26新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
1
1.6k
中途セールス職_会社説明資料
livesense
PRO
0
88
EM候補者向け転職会議説明資料
livesense
PRO
0
41
コロナで失われたノベルティ作成ノウハウを復活させた話
livesense
PRO
0
140
株式会社リブセンス 会社説明資料(報道関係者様向け)
livesense
PRO
0
580
転職会議でGPT-3を活用した企業口コミ要約機能をリリースした話
livesense
PRO
0
1.2k
株式会社リブセンス マッハバイト_プレイブック
livesense
PRO
0
580
Tech Award 2021 選出方法
livesense
PRO
0
1.1k
Other Decks in Technology
See All in Technology
waitany と waitall を作った話
mrkn
0
110
いまからでも遅くない! コンテナでWebアプリケーションを 動かしてみよう(1)Gitハンズオン
nomu
0
280
エンジニア採用ノウハウ0から継続的な採用に至るまで / Try recruiting engineers
ohageeq
0
130
AWS版GitHub?Amazon CodeCatalystの全体像をまとめてみた
oshanqq
1
3.2k
FastlyとfalcoでNode.jsレスな Webサーバー構築 : IPTV版 ABEMAアプリインフラ刷新 / Building web server using Fastly and falco without Node.js for IPTV ABEMA app
nodaguti
0
240
#Zenoh 完全に理解した 〜組込み純情篇〜
takasehideki
1
450
Estrategias de escalabilidade para projetos web
jessilyneh
1
150
Bidirectional Quadratic Voting Leveraging Issue-Based Matching
kentaro
2
170
夏休みの(最後の)宿題 for JuliaTokyo #12
antimon2
0
130
Oracle Exadata Database Service(Dedicated Infrastructure):サービス概要のご紹介
oracle4engineer
PRO
0
9.3k
標準ライブラリの奥深アップデートを掘り下げよう!
logica0419
2
390
LLM を現場で評価する
asei
4
680
Featured
See All Featured
Music & Morning Musume
bryan
46
6k
Building Flexible Design Systems
yeseniaperezcruz
324
37k
Designing for Performance
lara
604
68k
Principles of Awesome APIs and How to Build Them.
keavy
125
16k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
22
3.9k
Six Lessons from altMBA
skipperchong
26
3.3k
How GitHub (no longer) Works
holman
309
140k
Being A Developer After 40
akosma
82
580k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
165
48k
Faster Mobile Websites
deanohume
304
30k
WebSockets: Embracing the real-time Web
robhawkes
59
7.3k
Fashionably flexible responsive web design (full day workshop)
malarkey
400
65k
Transcript
Argo Workflow ʹΑΔ ػցֶशϫʔΫϑϩʔཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε
Data Pipeline Casual Talk #3
͢͜ͱ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ϦϒηϯεͷαʔϏεͱMLγεςϜ • MLγεςϜͷ։ൃɾӡ༻ࣄ •
MLγεςϜͷίϯϙʔωϯτׂͱίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • Argo Workflow ͷجຊػೳ • ϦϒηϯεͰͷ Argo Workflow ӡ༻ ※ Kubernetes ͷجૅࣝΛલఏͱ͍ͯ͠·͢
αʔϏεͱMLγεςϜ
ϦϒηϯεͷαʔϏε
ϦϒηϯεͰͷMLར༻ αʔϏεͱML • ٻਓɾෆಈ࢈ྖҬͰෳͷWebαʔϏεΛӡӦ • MLγεςϜͷ։ൃɾӡ༻νʔϜԣஅ৫ͱͯ͠αʔϏε͔Βಠཱ • ֤αʔϏεʹϨίϝϯυޮՌ༧ଌϞσϧͳͲෳͷMLγεςϜΛఏڙ ओͳMLγεςϜ •
ٻਓϨίϝϯυΤϯδϯ • Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ • A/BςετɾόϯσΟοτπʔϧ
ٻਓϨίϝϯυΤϯδϯ • ϚοϋόΠτɾస৬φϏͳͲͷϢʔβʹٻਓΛਪન • ϝʔϧɾWebαΠτɾωΠςΟϒΞϓϦͳͲ༷ʑͳॴͰಋೖ
ٻਓϨίϝϯυΤϯδϯͷ෦ • ධՁɾίϯςϯπΛͱʹ MF, FM ͳͲͷΞϧΰϦζϜΛద༻͠είΞΛࢉग़ • user-item item-item
ͷϦετΛόονॲཧͰੜ֤͠αʔϏεʹఏڙ
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ • ϚοϋόΠτɾస৬φϏͳͲͷٻਓͷCVRԠื୯ՁΛࢉग़ • ݕࡧ݁ՌͷॱҐ੍ޚࠂग़ߘͷ࠷దԽʹ׆༻
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧͷ෦ • ϩάΛ༻͍ͯϕΠζਪఆɾϩδεςΟοΫճؼͰ༧ଌɾਪఆ • σΟϨΫλʔ͚ʹ؆қతͳϏϡʔϫΛWebΞϓϦͱͯ͠։ൃ
A/BςετɾόϯσΟοτπʔϧ • A/Bςετͷύλʔϯ৴ൺΛόϯσΟοτΞϧΰϦζϜͰࣗಈௐ • WebαΠτɾωΠςΟϒΞϓϦͷ࠷దԽΛޮԽ
A/BςετɾόϯσΟοτπʔϧͷ෦ • ཧը໘͔ΒύλʔϯΛొ͠ɺWeb APIͰϥϯμϜʹ৴ • CVϩάΛੳج൫Ͱूܭ͠ɺύλʔϯ৴ൺΛࣗಈߋ৽
MLγεςϜͷ։ൃɾӡ༻ࣄ
ϦϒηϯεͷMLγεςϜͷಛ ֶशɾ༧ଌͱେ෦͕όονॲཧ • ֶश: CVR༧ଌϞσϧͷֶशϨίϝϯυͷҼࢠղ • ༧ଌ: ݕࡧɾϨίϝϯυ༻ͷείΞΛࣄલʹҰׅܭࢉ όονॲཧͷߏ͕ෳࡶ •
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ • ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍ • తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ • ϨίϝϯυΤϯδϯͰෳͷΞϧΰϦζϜΛΈ߹Θͤͯ͏ • ϑΟϧλϦϯάϦετͷϚʔδΛߦͬͯϨίϝϯυϦετΛੜ
ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍ • ಉαʔϏεͰA/BςετͷͨΊΞϧΰϦζϜ͚ͩมߋ • ผαʔϏεͷԣల։ͷࡍʹΞϧΰϦζϜΛ࠶ར༻
తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ ٻਓαʔϏεࠂαʔϏεͳͲͱൺ୯Ձ͕େ͖͘CVR͕খ͍͞ → ࠷ਪఆϕʔεͷҰൠతͳMLϥΠϒϥϦ͕ద͠ͳ͍͜ͱ → ϞσϧɾΞϧΰϦζϜͷࣗલ࣮ͷͨΊݴޠɾϥΠϒϥϦΛ͍͚Δ • ϨίϝϯυΞϧΰϦζϜΛ Julia Ͱ࣮
• Alternating Least SquaresʹΑΔFactorization Machinesͷύϥϝʔλਪఆ • Factorization MachinesΛϨίϝϯσʔγϣϯͰ͏ͱ͖ͷධՁਪఆܭࢉ • ਪఆɾ༧ଌϞσϧͰ Stan Λར༻ • ֊ϕΠζʹΑΔখඪຊσʔλͷൺͷਪఆ
ෳࡶԽ͢Δߏͷରॲ Ҏલ֤γεςϜ͕୯ҰϨϙδτϦͰཧ͞ΕΔϞϊϦγοΫͳߏ → ߏͷෳࡶԽͰ։ൃɾӡ༻͕·ΘΒͳ͘ͳ͖ͬͯͨ • MLͷίΞ෦ͱDBIO͕ີ݁߹͠ݸผ࣮ߦͰ͖ͳ͍ • γεςϜؒͰڞ௨͢ΔΞϧΰϦζϜ͕ίϐϖ͞ΕΔ • ಉҰͷόονॲཧͰεςοϓ͝ͱʹݴޠΛม͑ʹ͍͘
→ γεςϜΛػೳ͝ͱʹׂɾ࠶ߏங͢Δ͜ͱʹ
ίϯϙʔωϯτׂͱίϯςφԽ
ίϯϙʔωϯτͷׂ ·ͣγεςϜΛ࣍ͷΑ͏ͳ୯ػೳίϯϙʔωϯτʹׂͨ͠ • ֤ίϯϙʔωϯτ CLI Ͱ୯ಠ࣮ߦͰ͖Δ • ίϯϙʔωϯτؒͷೖग़ྗͯ͢ϑΝΠϧΛհ͢Δ name role
input file output file sqlkit DBIO SQL CSV nlpkit ࣗવݴޠॲཧ ςΩετ BoWϕΫτϧ recommender Ϩίϝϯυ ධՁ ਪનείΞ
ίϯϙʔωϯτͷίϯςφԽ ͞Βʹ֤ίϯϙʔωϯτΛ୯ҰͷίϯςφΠϝʔδʹͨ͠ • ֤ίϯςφίϯϙʔωϯτ docker run kubectl run Ͱ࣮ߦͰ͖Δ
• γεςϜ͝ͱͷࠩ΄΅ઃఆϑΝΠϧSQL͚ͩͰදݱ # load dataset docker run -v $(pwd):/workdir sqlkit select ratings.sql /workdir/ratings.csv docker run -v $(pwd):/workdir sqlkit select content.sql /workdir/content.csv # preprocess docker run -v $(pwd):/workdir nlpkit vectorize /workdir/content.csv /workdir/features.csv # run recommender docker run -v $(pwd):/workdir recommender predict config.yaml /workdir
ίϯϙʔωϯτͷׂͱίϯςφԽ
ϫʔΫϑϩʔΛͲ͏࣮ݱ͢Δ͔ʁ ίϯϙʔωϯτͷׂͱίϯςφԽʹΑΓෳͷ՝ΛղܾͰ͖ͨ • ີ݁߹ͷղফɾεςοϓ࣮ߦͷՄೳԽ • ڞ௨෦ͷ࠶ར༻ՄೳԽ • ݴޠɾϥΠϒϥϦͷ͍͚ͷ༰қԽ ͔͠͠ɺෳࡶͳϫʔΫϑϩʔΛͲ͏ߏஙɾཧ͢Δ͔ͷ՝Δ •
୯७ͳόονॲཧͳΒ docker run kubectl run Λஞ࣮࣍ߦ͢Δ͚ͩ • ࣮ࡍʹ͜ͷํࣜͰຊ൪Քಇ͍ͯ͠ΔγεςϜଘࡏ • ฒྻԽɾϦτϥΠͳͲͷߴͳϫʔΫϑϩʔΛ࣮ݱ͍ͨ͠߹ʁ
ͦΜͳ͋Δ (2017)
Argo Workflow Λൃݟ https://argoproj.github.io/
Argo Workflow "Container native workflow engine for Kubernetes" Kubernetes ্Ͱෳͷίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ࣮ߦͰ͖Δ
ͻͱ͜ͱͰݴ͏ͱʮߴػೳͳ k8s Jobʯ • ෳίϯςφͷྻɾฒྻɾDAG࣮ߦ • ذɾϧʔϓɾϑοΫͷ੍ޚϑϩʔ • ϦτϥΠɾλΠϜΞτɾϫʔΧʔϊʔυબ • ϞχλϦϯά༻ Web UI
Argo Workflow ͷಛ CRD controller ͱ࣮ͯ͠͞Ε͍ͯΔ • argo submit Ͱ࡞͞Εͨ
Workflow ϦιʔεΛ controller ͕࣮ߦ • ϫʔΫϑϩʔͷ֤εςοϓ Pod ͱͯ͠ಈ࡞ ϫʔΫϑϩʔͷ࣮ߦʹઐ೦͠ɺτϦΨʔఆظ࣮ߦͷػೳͨͳ͍ • ͍উख Airflow, Digdag ΑΓ Luigi ʹ͍ۙ • Argo Events ͱ͍͏ผπʔϧͰ༷ʑͳτϦΨʔΛఏڙ
Argo Workflow ͷجຊػೳ
୯ҰίϯςφΛ࣮ߦ͢Δ࠷؆୯ͳϫʔΫϑϩʔ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: hello-world- spec: entrypoint:
entrypoint # ࠷ॳʹ࣮ߦ͢ΔίϯςφςϯϓϨʔτΛࢦఆ templates: # ̍ͭҎ্ͷίϯςφςϯϓϨʔτΛఆٛ - name: entrypoint container: image: alpine:latest command: ["echo", "hello world"] ※Ҏ߱ͷྫ spec ԼͷΈهࡌ
None
ϫʔΫϑϩʔʹύϥϝʔλΛ͢ entrypoint: entrypoint arguments: # ϫʔΫϑϩʔ࣮ߦ࣌ʹ argo submit -p message=hello
ͷΑ͏ʹͤΔ parameters: - name: message templates: - name: entrypoint container: image: alpine:latest command: ["echo", "{{workflow.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
εςοϓʹύϥϝʔλΛ͢ entrypoint: entrypoint templates: - name: entrypoint inputs: # ޙड़ͷ
steps, dag ͳͲ͔Β͢ parameters: - name: message value: hello container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
steps: εςοϓͷྻɾฒྻ࣮ߦ templates: - name: entrypoint steps: - - name:
hello1 template: echo # ίϯςφςϯϓϨʔτΛࢦఆ arguments: {parameters: [{name: "message", value: "hello1"}]} - - name: hello2a # hello1 ͷ࣍ʹ hello2a, hello2b Λ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2a"}]} - name: hello2b # hello2a, hello2b ฒྻ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2b"}]} - name: echo inputs: {parameters: [{name: "message"}]} container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"]
None
dag: ͰλεΫͷDAG࣮ߦ templates: - name: entrypoint dag: tasks: - name:
A template: echo arguments: {parameters: [{name: message, value: A}]} - name: B dependencies: [A] # ґଘλεΫΛࢦఆ template: echo arguments: {parameters: [{name: message, value: B}]} - name: C dependencies: [A] template: echo arguments: {parameters: [{name: message, value: C}]} - name: D dependencies: [B, C] # ґଘλεΫΛෳࢦఆ template: echo arguments: {parameters: [{name: message, value: D}]}
None
artifact: εςοϓؒͰϑΝΠϧΛड͚͠ templates: - name: entrypoint steps: - - {name:
generate-artifact, template: generate-artifact} - - {name: consume-artifact, template: consume-artifact} - name: generate-artifact container: image: alpine:latest command: ["sh", "-c", "echo hello > /tmp/output.txt"] outputs: artifacts: - {name: "result", path: "/tmp/output.txt"} - name: consume-artifact container: image: alpine:latest command: ["sh", "-c", "cat /tmp/input.txt"] inputs: artifacts: - {name: "result", path: "/tmp/input.txt"}
when: ϫʔΫϑϩʔͷذ templates: - name: entrypoint steps: - - name:
flip-coin template: flip-coin # when Ͱશεςοϓͷ݁ՌΛͱʹذ - - when: "{{steps.flip-coin.outputs.result}} == heads" name: heads - when: "{{steps.flip-coin.outputs.result}} == tails" name: tails - name: flip-coin script: image: python:latest command: [python] source: "import random; print(random.choice(['heads', 'tails']))"
None
withItems, withParams: εςοϓͷ܁Γฦ͠ templates: - name: entrypoint steps: # withItems
Ͱͨ͠ item ͷ͚ͩεςοϓΛฒྻ࣮ߦ - - withItems: ["hello world", "goodbye world", "ok world"] name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]} # withParams ʹ ["hello world", "goodbye world"] ͷΑ͏ͳ JSON Λ͢͜ͱՄೳ - - withParams: "{{workflow.parameters.params}}" name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]}
None
exitHandler : ϫʔΫϑϩʔͷޭɾࣦഊ࣌ͷϋϯυϦϯά onExit: exit-handler templates: - name: entrypoint container:
image: alpine:latest command: ["exit", "1"] - name: exit-handler steps: # workflow.status Λͱʹذ - - when: "{{workflow.status}} == Succeeded" template: echo arguments: {parameters: [{name: "message", value: "SUCCESS"}]} - when: "{{workflow.status}} != Succeeded" template: echo arguments: {parameters: [{name: "message", value: "ERROR!"}]}
None
ϦτϥΠɾλΠϜΞτͳͲ templates: - name: entrypoint # ϦτϥΠճͳͲΛઃఆ retryStrategy: limit: 2
# λΠϜΞτΛઃఆ (Pod ͷه๏ͱಉ͡) activeDeadlineSeconds: 28800 # ϊʔυͷࢦఆ (Pod ͷه๏ͱಉ͡) nodeSelector: cloud.google.com/gke-nodepool: highmem-pool # Ϧιʔε੍ݶ (Pod ͷه๏ͱಉ͡) container: resources: limits: memory: "32Gi"
None
ͦͷଞ • ฒྻ࣮ߦ࣌ͷฒྻ্ݶΛઃఆ • ϘϦϡʔϜʹΑΔσʔλͷड͚͠ • ิॿίϯςφͷར༻ (Sidecar, Daemon, ...)
• ֎෦ετϨʔδͷར༻ • etc. ৄ͘͠ެࣜͷ example Λࢀর https://github.com/argoproj/argo/tree/master/examples
ϦϒηϯεͰͷ Argo Workflow ӡ༻
MLγεςϜͷ࣮ߦج൫ GCP্ͰGKE Λத৺ͱ͢Δػցֶशج൫Λߏங • ෳͷMLγεςϜΛ୯ҰͷGKEΫϥελʹू • όονॲཧ͚ͩͰͳ͘WebΞϓϦಉ͡ΫϥελͰӡ༻ Argo Workflow ͷར༻
• ίϯςφίϯϙʔωϯτGCBͰϏϧυ͠GCR ʹొ • ϫʔΫϑϩʔఆٛଞͷ manifest ͱಉ͡ϨϙδτϦͰཧ • ఆظ࣮ߦ͢ΔϫʔΫϑϩʔ CronJob Ͱ argo submit
GCP, GKE, Argo Workflow ͷߏਤ
ӡ༻ࢦ όονॲཧͱΓ͋͑ͣ Workflow ͱͯ͠ఆٛ • खݩͰ docker run ͚ͩͰࢼݧ࣮ߦͰ͖ΔΑ͏γεςϜΛ࣮ •
·ͣ୯Ұεςοϓͷ Workflow ͱͯ͠ӡ༻ʹࡌͤΔ ӡ༻͠ͳ͕ΒίϯϙʔωϯτԽΛਐΊͯຊମΛεϦϜԽ • DBIO௨ͳͲͷڞ௨ॲཧΛஈ֊తʹΓग़͍ͯ͘͠ • ฒྻԽɾϦτϥΠͳͲͳΔ͘ Workflow ଆͷػೳͰ࣮ݱ ҎԼɺࣄྫͱӡ༻ϊϋΛհ
CASE: ίϯϙʔωϯτͷΈ߹Θͤ • ϨίϝϯυΤϯδϯಛʹίϯϙʔωϯτԽ͕ਐΜͰ͍Δ • SQLઃఆϑΝΠϧͻͱͭͷίϯςφʹ·ͱΊͯ࠷ॳʹల։ templates: - name: entrypoint
steps: - - name: load-config - - name: sqlkit withItems: - sqlfile: /workspace/sql/ratings.sql - sqlfile: /workspace/sql/contents.sql - - name: nlpkit - - name: recommender
CASE: ϝΠϯͷόονॲཧͷεϦϜԽ • ਪఆɾ༧ଌϞσϧDBIO௨ͳͲΛΓग़ͯ͠ϝΠϯͷόονॲཧΛεϦϜԽ • MLΤϯδχΞɾMLج൫ΤϯδχΞͰͷ୲Λ͍ͯ͘͢͠͠Δ onExit: exit-handler templates: -
name: entrypoint steps: - - name: train-predict # MLΤϯδχΞ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLج൫ΤϯδχΞ͕࣮ - name: exit-handler # MLج൫ΤϯδχΞ͕࣮ steps: - - when: "{{workflow.status}} != Succeeded" name: notify-error
CASE: MLϞσϧͷ؆қతͳCD • ਪఆɾ༧ଌϞσϧͷ݁ՌϏϡʔϫ Deployment ͱͯ͠ӡ༻ • ਪఆॲཧྃ࣌ʹ kubectl set
env ͰϏϡʔϫʹ৽͍͠ϞσϧΛಡΈࠐ·ͤΔ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ templates: - name: entrypoint steps: - - name: train-predict - - name: import-to-db - - name: update-viewer - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
CASE: ॏ͍ɾෆ҆ఆͳMLॲཧΛѻ͏ • ਪఆɾ༧ଌϞσϧͳͲͰ Stan Λଟ༻ • ϝϞϦɾCPUΛେྔʹফඅ͢Δ߹ઐ༻ͷϊʔυͰ࣮ߦ • αϯϓϦϯά͕֬తʹࣦഊ͢ΔͷͰϦτϥΠɾλΠϜΞτ͕ඞཁ
- name: train-predict activeDeadlineSeconds: 28800 # 8h retryStrategy: limit: 2 nodeSelector: cloud.google.com/gke-nodepool: highmem-pool container: resources: limits: memory: "32Gi"
CASE: Ϟσϧਪఆͷಈతͳฒྻ࣮ߦ • όϯσΟοτπʔϧͰ࣮ࢪதͷςετ͝ͱʹਪఆॲཧ͕ඞཁ • ֤ςετͷਪఆॲཧΛಈతʹฒྻ࣮ߦ templates: - name: entrypoint
steps: # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ # લͷεςοϓͷग़ྗ͔ΒύϥϝʔλͷϦετΛಡΈࠐΈ - - withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷཁૉ͝ͱʹޙଓͷεςοϓΛ࣮ߦ name: calc-weights arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ӡ༻ TIPS Argo Workflow ͷ Web UI ͷΞΫηε • σϑΥϧτͰ
kubectl port-forward ͰΞΫηε͢Δඞཁ͕͋Δ • ΠϯλʔωοτΞΫηεΛՄೳʹ͢Δʹ Ingress ͰϩʔυόϥϯαΛཱͯΔ • GCP ͷ Identity-Aware Proxy Λ͏ͱϩʔυόϥϯαଆͰೝূΛ͔͚ΒΕΔ ݹ͍ϫʔΫϑϩʔͷΫϦʔϯΞοϓ • ࣮ߦࡁΈͷ Workflow ͱͦͷཧ͢Δ Pod Successful ͷ··Γଓ͚Δ • ఆظతʹݹ͍ Workflow Λআ͢Δ CronJob Λཱ͍ͯͯΔ • argo delete --older Φϓγϣϯ͕ศར
Pros, Cons, ·ͱΊ
Argo Workflow - Pros ଞͷϫʔΫϑϩʔΤϯδϯͱൺϩοΫΠϯ͞Εʹ͍͘ • ίϯςφԽ͞Ε͍ͯΕԿͰಈ͔ͤΔ • ࠓޙଞͷϫʔΫϑϩʔΤϯδϯ͕ग़͖ͯͯΓ͍͑͢ όονॲཧͱWebΞϓϦΛಉ͡ΫϥελͰཧͰ͖Δ
• σϓϩΠɾϩΪϯάɾϞχλϦϯάɾΤϥʔϨϙʔτͳͲΛҰݩԽ • ΦʔτεέʔϦϯάͳͲͱΈ߹ΘͤͯϦιʔεར༻ΛޮԽ
Argo Workflow - Cons ଞͷϫʔΫϑϩʔΤϯδϯ΄ͲϓϩάϥϚϒϧͰͳ͍ • Airflow, Luigi ͷΑ͏ʹ Python
DSL ͕ॻ͚ͨΓ͠ͳ͍ • ֤ΫϥυαʔϏεઐ༻ͷΦϖϨʔλ༻ҙ͞Ε͍ͯͳ͍ ࡞͞ΕͨϫʔΫϑϩʔΛଈ࣮࣌ߦ͢ΔҎ֎ͷػೳͨͳ͍ • ఆظ࣮ߦʹ CronJob ͳͲΛ͏ඞཁ͕͋Δ • Web UI ϞχλϦϯάͷΈͰϦτϥΠͳͲͷૢ࡞Ͱ͖ͳ͍ • ϫʔΫϑϩʔࣗମͷςϯϓϨʔτԽɾ࠶ར༻͕͠ʹ͍͘ • WorkflowTemplate ͕ఏҊ͞Ε͍ͯΔͷͰظ
·ͱΊ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ෳͷαʔϏεͰMLγεςϜΛར༻ • ଟ͘ͷεςοϓ͔ΒͳΔόονॲཧ͕ෳଘࡏ •
։ൃɾӡ༻ΛޮԽ͢ΔͨΊίϯϙʔωϯτΛׂͯ͠ίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • ίϯςφίϯϙʔωϯτΛΈ߹ΘͤͯϫʔΫϑϩʔΛߏங • MLγεςϜͷ։ൃɾӡ༻্ͷ߹ʹ߹Θ֤ͤͯछػೳΛ׆༻