Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Argo Workflow による機械学習ワークフロー管理
Search
Livesense Inc.
PRO
June 27, 2019
Technology
2
3.3k
Argo Workflow による機械学習ワークフロー管理
2019/06/27
Data Pipeline Casual Talk #3
Livesense Inc.
PRO
June 27, 2019
Tweet
Share
More Decks by Livesense Inc.
See All by Livesense Inc.
株式会社リブセンス会社紹介資料 / Invent the next common.
livesense
PRO
1
4.4k
26新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
1
1.8k
中途セールス職_会社説明資料
livesense
PRO
0
92
EM候補者向け転職会議説明資料
livesense
PRO
0
42
コロナで失われたノベルティ作成ノウハウを復活させた話
livesense
PRO
0
140
株式会社リブセンス 会社説明資料(報道関係者様向け)
livesense
PRO
0
590
転職会議でGPT-3を活用した企業口コミ要約機能をリリースした話
livesense
PRO
0
1.2k
株式会社リブセンス マッハバイト_プレイブック
livesense
PRO
0
590
Tech Award 2021 選出方法
livesense
PRO
0
1.1k
Other Decks in Technology
See All in Technology
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
2
170
エンジニア視点で見る、 組織で運用されるデザインシステムにするには
shunya078
1
280
Datadog を使ったプロダクトとクラウドの セキュリティモニタリング
mrtc0
0
600
AWSを始めた頃に陥りがちなポイントをまとめてみた
oshanqq
1
3.3k
分野に潜むツールの紹介
pojiro
1
350
サイボウズ 開発本部採用ピッチ / Cybozu Engineer Recruit
cybozuinsideout
PRO
9
41k
MySQLユーザ会なにやってるの?とおさそいと / たいへんなのw
sakaik
1
130
すぐに始めるAWSコスト削減。短期でできる改善策と長期的な運用負荷軽減への取り組み方を解説
ncdc
1
530
Oracle Autonomous Database:サービス概要のご紹介
oracle4engineer
PRO
1
6.9k
RAGHack: Building RAG apps in Python
pamelafox
0
150
Envoy External AuthZとgRPC Extensionを利用した「頑張らない」Microservices認証認可基盤
andoshin11
0
210
バックログを導入し やっぱやめた話
ota42y
0
190
Featured
See All Featured
Building Applications with DynamoDB
mza
89
5.9k
Art, The Web, and Tiny UX
lynnandtonic
294
20k
How to Think Like a Performance Engineer
csswizardry
16
930
YesSQL, Process and Tooling at Scale
rocio
167
14k
The Straight Up "How To Draw Better" Workshop
denniskardys
230
130k
The World Runs on Bad Software
bkeepers
PRO
64
11k
Designing Experiences People Love
moore
138
23k
Gamification - CAS2011
davidbonilla
79
4.9k
Pencils Down: Stop Designing & Start Developing
hursman
118
11k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
225
22k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Become a Pro
speakerdeck
PRO
22
4.9k
Transcript
Argo Workflow ʹΑΔ ػցֶशϫʔΫϑϩʔཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε
Data Pipeline Casual Talk #3
͢͜ͱ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ϦϒηϯεͷαʔϏεͱMLγεςϜ • MLγεςϜͷ։ൃɾӡ༻ࣄ •
MLγεςϜͷίϯϙʔωϯτׂͱίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • Argo Workflow ͷجຊػೳ • ϦϒηϯεͰͷ Argo Workflow ӡ༻ ※ Kubernetes ͷجૅࣝΛલఏͱ͍ͯ͠·͢
αʔϏεͱMLγεςϜ
ϦϒηϯεͷαʔϏε
ϦϒηϯεͰͷMLར༻ αʔϏεͱML • ٻਓɾෆಈ࢈ྖҬͰෳͷWebαʔϏεΛӡӦ • MLγεςϜͷ։ൃɾӡ༻νʔϜԣஅ৫ͱͯ͠αʔϏε͔Βಠཱ • ֤αʔϏεʹϨίϝϯυޮՌ༧ଌϞσϧͳͲෳͷMLγεςϜΛఏڙ ओͳMLγεςϜ •
ٻਓϨίϝϯυΤϯδϯ • Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ • A/BςετɾόϯσΟοτπʔϧ
ٻਓϨίϝϯυΤϯδϯ • ϚοϋόΠτɾస৬φϏͳͲͷϢʔβʹٻਓΛਪન • ϝʔϧɾWebαΠτɾωΠςΟϒΞϓϦͳͲ༷ʑͳॴͰಋೖ
ٻਓϨίϝϯυΤϯδϯͷ෦ • ධՁɾίϯςϯπΛͱʹ MF, FM ͳͲͷΞϧΰϦζϜΛద༻͠είΞΛࢉग़ • user-item item-item
ͷϦετΛόονॲཧͰੜ֤͠αʔϏεʹఏڙ
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ • ϚοϋόΠτɾస৬φϏͳͲͷٻਓͷCVRԠื୯ՁΛࢉग़ • ݕࡧ݁ՌͷॱҐ੍ޚࠂग़ߘͷ࠷దԽʹ׆༻
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧͷ෦ • ϩάΛ༻͍ͯϕΠζਪఆɾϩδεςΟοΫճؼͰ༧ଌɾਪఆ • σΟϨΫλʔ͚ʹ؆қతͳϏϡʔϫΛWebΞϓϦͱͯ͠։ൃ
A/BςετɾόϯσΟοτπʔϧ • A/Bςετͷύλʔϯ৴ൺΛόϯσΟοτΞϧΰϦζϜͰࣗಈௐ • WebαΠτɾωΠςΟϒΞϓϦͷ࠷దԽΛޮԽ
A/BςετɾόϯσΟοτπʔϧͷ෦ • ཧը໘͔ΒύλʔϯΛొ͠ɺWeb APIͰϥϯμϜʹ৴ • CVϩάΛੳج൫Ͱूܭ͠ɺύλʔϯ৴ൺΛࣗಈߋ৽
MLγεςϜͷ։ൃɾӡ༻ࣄ
ϦϒηϯεͷMLγεςϜͷಛ ֶशɾ༧ଌͱେ෦͕όονॲཧ • ֶश: CVR༧ଌϞσϧͷֶशϨίϝϯυͷҼࢠղ • ༧ଌ: ݕࡧɾϨίϝϯυ༻ͷείΞΛࣄલʹҰׅܭࢉ όονॲཧͷߏ͕ෳࡶ •
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ • ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍ • తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ • ϨίϝϯυΤϯδϯͰෳͷΞϧΰϦζϜΛΈ߹Θͤͯ͏ • ϑΟϧλϦϯάϦετͷϚʔδΛߦͬͯϨίϝϯυϦετΛੜ
ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍ • ಉαʔϏεͰA/BςετͷͨΊΞϧΰϦζϜ͚ͩมߋ • ผαʔϏεͷԣల։ͷࡍʹΞϧΰϦζϜΛ࠶ར༻
తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ ٻਓαʔϏεࠂαʔϏεͳͲͱൺ୯Ձ͕େ͖͘CVR͕খ͍͞ → ࠷ਪఆϕʔεͷҰൠతͳMLϥΠϒϥϦ͕ద͠ͳ͍͜ͱ → ϞσϧɾΞϧΰϦζϜͷࣗલ࣮ͷͨΊݴޠɾϥΠϒϥϦΛ͍͚Δ • ϨίϝϯυΞϧΰϦζϜΛ Julia Ͱ࣮
• Alternating Least SquaresʹΑΔFactorization Machinesͷύϥϝʔλਪఆ • Factorization MachinesΛϨίϝϯσʔγϣϯͰ͏ͱ͖ͷධՁਪఆܭࢉ • ਪఆɾ༧ଌϞσϧͰ Stan Λར༻ • ֊ϕΠζʹΑΔখඪຊσʔλͷൺͷਪఆ
ෳࡶԽ͢Δߏͷରॲ Ҏલ֤γεςϜ͕୯ҰϨϙδτϦͰཧ͞ΕΔϞϊϦγοΫͳߏ → ߏͷෳࡶԽͰ։ൃɾӡ༻͕·ΘΒͳ͘ͳ͖ͬͯͨ • MLͷίΞ෦ͱDBIO͕ີ݁߹͠ݸผ࣮ߦͰ͖ͳ͍ • γεςϜؒͰڞ௨͢ΔΞϧΰϦζϜ͕ίϐϖ͞ΕΔ • ಉҰͷόονॲཧͰεςοϓ͝ͱʹݴޠΛม͑ʹ͍͘
→ γεςϜΛػೳ͝ͱʹׂɾ࠶ߏங͢Δ͜ͱʹ
ίϯϙʔωϯτׂͱίϯςφԽ
ίϯϙʔωϯτͷׂ ·ͣγεςϜΛ࣍ͷΑ͏ͳ୯ػೳίϯϙʔωϯτʹׂͨ͠ • ֤ίϯϙʔωϯτ CLI Ͱ୯ಠ࣮ߦͰ͖Δ • ίϯϙʔωϯτؒͷೖग़ྗͯ͢ϑΝΠϧΛհ͢Δ name role
input file output file sqlkit DBIO SQL CSV nlpkit ࣗવݴޠॲཧ ςΩετ BoWϕΫτϧ recommender Ϩίϝϯυ ධՁ ਪનείΞ
ίϯϙʔωϯτͷίϯςφԽ ͞Βʹ֤ίϯϙʔωϯτΛ୯ҰͷίϯςφΠϝʔδʹͨ͠ • ֤ίϯςφίϯϙʔωϯτ docker run kubectl run Ͱ࣮ߦͰ͖Δ
• γεςϜ͝ͱͷࠩ΄΅ઃఆϑΝΠϧSQL͚ͩͰදݱ # load dataset docker run -v $(pwd):/workdir sqlkit select ratings.sql /workdir/ratings.csv docker run -v $(pwd):/workdir sqlkit select content.sql /workdir/content.csv # preprocess docker run -v $(pwd):/workdir nlpkit vectorize /workdir/content.csv /workdir/features.csv # run recommender docker run -v $(pwd):/workdir recommender predict config.yaml /workdir
ίϯϙʔωϯτͷׂͱίϯςφԽ
ϫʔΫϑϩʔΛͲ͏࣮ݱ͢Δ͔ʁ ίϯϙʔωϯτͷׂͱίϯςφԽʹΑΓෳͷ՝ΛղܾͰ͖ͨ • ີ݁߹ͷղফɾεςοϓ࣮ߦͷՄೳԽ • ڞ௨෦ͷ࠶ར༻ՄೳԽ • ݴޠɾϥΠϒϥϦͷ͍͚ͷ༰қԽ ͔͠͠ɺෳࡶͳϫʔΫϑϩʔΛͲ͏ߏஙɾཧ͢Δ͔ͷ՝Δ •
୯७ͳόονॲཧͳΒ docker run kubectl run Λஞ࣮࣍ߦ͢Δ͚ͩ • ࣮ࡍʹ͜ͷํࣜͰຊ൪Քಇ͍ͯ͠ΔγεςϜଘࡏ • ฒྻԽɾϦτϥΠͳͲͷߴͳϫʔΫϑϩʔΛ࣮ݱ͍ͨ͠߹ʁ
ͦΜͳ͋Δ (2017)
Argo Workflow Λൃݟ https://argoproj.github.io/
Argo Workflow "Container native workflow engine for Kubernetes" Kubernetes ্Ͱෳͷίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ࣮ߦͰ͖Δ
ͻͱ͜ͱͰݴ͏ͱʮߴػೳͳ k8s Jobʯ • ෳίϯςφͷྻɾฒྻɾDAG࣮ߦ • ذɾϧʔϓɾϑοΫͷ੍ޚϑϩʔ • ϦτϥΠɾλΠϜΞτɾϫʔΧʔϊʔυબ • ϞχλϦϯά༻ Web UI
Argo Workflow ͷಛ CRD controller ͱ࣮ͯ͠͞Ε͍ͯΔ • argo submit Ͱ࡞͞Εͨ
Workflow ϦιʔεΛ controller ͕࣮ߦ • ϫʔΫϑϩʔͷ֤εςοϓ Pod ͱͯ͠ಈ࡞ ϫʔΫϑϩʔͷ࣮ߦʹઐ೦͠ɺτϦΨʔఆظ࣮ߦͷػೳͨͳ͍ • ͍উख Airflow, Digdag ΑΓ Luigi ʹ͍ۙ • Argo Events ͱ͍͏ผπʔϧͰ༷ʑͳτϦΨʔΛఏڙ
Argo Workflow ͷجຊػೳ
୯ҰίϯςφΛ࣮ߦ͢Δ࠷؆୯ͳϫʔΫϑϩʔ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: hello-world- spec: entrypoint:
entrypoint # ࠷ॳʹ࣮ߦ͢ΔίϯςφςϯϓϨʔτΛࢦఆ templates: # ̍ͭҎ্ͷίϯςφςϯϓϨʔτΛఆٛ - name: entrypoint container: image: alpine:latest command: ["echo", "hello world"] ※Ҏ߱ͷྫ spec ԼͷΈهࡌ
None
ϫʔΫϑϩʔʹύϥϝʔλΛ͢ entrypoint: entrypoint arguments: # ϫʔΫϑϩʔ࣮ߦ࣌ʹ argo submit -p message=hello
ͷΑ͏ʹͤΔ parameters: - name: message templates: - name: entrypoint container: image: alpine:latest command: ["echo", "{{workflow.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
εςοϓʹύϥϝʔλΛ͢ entrypoint: entrypoint templates: - name: entrypoint inputs: # ޙड़ͷ
steps, dag ͳͲ͔Β͢ parameters: - name: message value: hello container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
steps: εςοϓͷྻɾฒྻ࣮ߦ templates: - name: entrypoint steps: - - name:
hello1 template: echo # ίϯςφςϯϓϨʔτΛࢦఆ arguments: {parameters: [{name: "message", value: "hello1"}]} - - name: hello2a # hello1 ͷ࣍ʹ hello2a, hello2b Λ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2a"}]} - name: hello2b # hello2a, hello2b ฒྻ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2b"}]} - name: echo inputs: {parameters: [{name: "message"}]} container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"]
None
dag: ͰλεΫͷDAG࣮ߦ templates: - name: entrypoint dag: tasks: - name:
A template: echo arguments: {parameters: [{name: message, value: A}]} - name: B dependencies: [A] # ґଘλεΫΛࢦఆ template: echo arguments: {parameters: [{name: message, value: B}]} - name: C dependencies: [A] template: echo arguments: {parameters: [{name: message, value: C}]} - name: D dependencies: [B, C] # ґଘλεΫΛෳࢦఆ template: echo arguments: {parameters: [{name: message, value: D}]}
None
artifact: εςοϓؒͰϑΝΠϧΛड͚͠ templates: - name: entrypoint steps: - - {name:
generate-artifact, template: generate-artifact} - - {name: consume-artifact, template: consume-artifact} - name: generate-artifact container: image: alpine:latest command: ["sh", "-c", "echo hello > /tmp/output.txt"] outputs: artifacts: - {name: "result", path: "/tmp/output.txt"} - name: consume-artifact container: image: alpine:latest command: ["sh", "-c", "cat /tmp/input.txt"] inputs: artifacts: - {name: "result", path: "/tmp/input.txt"}
when: ϫʔΫϑϩʔͷذ templates: - name: entrypoint steps: - - name:
flip-coin template: flip-coin # when Ͱશεςοϓͷ݁ՌΛͱʹذ - - when: "{{steps.flip-coin.outputs.result}} == heads" name: heads - when: "{{steps.flip-coin.outputs.result}} == tails" name: tails - name: flip-coin script: image: python:latest command: [python] source: "import random; print(random.choice(['heads', 'tails']))"
None
withItems, withParams: εςοϓͷ܁Γฦ͠ templates: - name: entrypoint steps: # withItems
Ͱͨ͠ item ͷ͚ͩεςοϓΛฒྻ࣮ߦ - - withItems: ["hello world", "goodbye world", "ok world"] name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]} # withParams ʹ ["hello world", "goodbye world"] ͷΑ͏ͳ JSON Λ͢͜ͱՄೳ - - withParams: "{{workflow.parameters.params}}" name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]}
None
exitHandler : ϫʔΫϑϩʔͷޭɾࣦഊ࣌ͷϋϯυϦϯά onExit: exit-handler templates: - name: entrypoint container:
image: alpine:latest command: ["exit", "1"] - name: exit-handler steps: # workflow.status Λͱʹذ - - when: "{{workflow.status}} == Succeeded" template: echo arguments: {parameters: [{name: "message", value: "SUCCESS"}]} - when: "{{workflow.status}} != Succeeded" template: echo arguments: {parameters: [{name: "message", value: "ERROR!"}]}
None
ϦτϥΠɾλΠϜΞτͳͲ templates: - name: entrypoint # ϦτϥΠճͳͲΛઃఆ retryStrategy: limit: 2
# λΠϜΞτΛઃఆ (Pod ͷه๏ͱಉ͡) activeDeadlineSeconds: 28800 # ϊʔυͷࢦఆ (Pod ͷه๏ͱಉ͡) nodeSelector: cloud.google.com/gke-nodepool: highmem-pool # Ϧιʔε੍ݶ (Pod ͷه๏ͱಉ͡) container: resources: limits: memory: "32Gi"
None
ͦͷଞ • ฒྻ࣮ߦ࣌ͷฒྻ্ݶΛઃఆ • ϘϦϡʔϜʹΑΔσʔλͷड͚͠ • ิॿίϯςφͷར༻ (Sidecar, Daemon, ...)
• ֎෦ετϨʔδͷར༻ • etc. ৄ͘͠ެࣜͷ example Λࢀর https://github.com/argoproj/argo/tree/master/examples
ϦϒηϯεͰͷ Argo Workflow ӡ༻
MLγεςϜͷ࣮ߦج൫ GCP্ͰGKE Λத৺ͱ͢Δػցֶशج൫Λߏங • ෳͷMLγεςϜΛ୯ҰͷGKEΫϥελʹू • όονॲཧ͚ͩͰͳ͘WebΞϓϦಉ͡ΫϥελͰӡ༻ Argo Workflow ͷར༻
• ίϯςφίϯϙʔωϯτGCBͰϏϧυ͠GCR ʹొ • ϫʔΫϑϩʔఆٛଞͷ manifest ͱಉ͡ϨϙδτϦͰཧ • ఆظ࣮ߦ͢ΔϫʔΫϑϩʔ CronJob Ͱ argo submit
GCP, GKE, Argo Workflow ͷߏਤ
ӡ༻ࢦ όονॲཧͱΓ͋͑ͣ Workflow ͱͯ͠ఆٛ • खݩͰ docker run ͚ͩͰࢼݧ࣮ߦͰ͖ΔΑ͏γεςϜΛ࣮ •
·ͣ୯Ұεςοϓͷ Workflow ͱͯ͠ӡ༻ʹࡌͤΔ ӡ༻͠ͳ͕ΒίϯϙʔωϯτԽΛਐΊͯຊମΛεϦϜԽ • DBIO௨ͳͲͷڞ௨ॲཧΛஈ֊తʹΓग़͍ͯ͘͠ • ฒྻԽɾϦτϥΠͳͲͳΔ͘ Workflow ଆͷػೳͰ࣮ݱ ҎԼɺࣄྫͱӡ༻ϊϋΛհ
CASE: ίϯϙʔωϯτͷΈ߹Θͤ • ϨίϝϯυΤϯδϯಛʹίϯϙʔωϯτԽ͕ਐΜͰ͍Δ • SQLઃఆϑΝΠϧͻͱͭͷίϯςφʹ·ͱΊͯ࠷ॳʹల։ templates: - name: entrypoint
steps: - - name: load-config - - name: sqlkit withItems: - sqlfile: /workspace/sql/ratings.sql - sqlfile: /workspace/sql/contents.sql - - name: nlpkit - - name: recommender
CASE: ϝΠϯͷόονॲཧͷεϦϜԽ • ਪఆɾ༧ଌϞσϧDBIO௨ͳͲΛΓग़ͯ͠ϝΠϯͷόονॲཧΛεϦϜԽ • MLΤϯδχΞɾMLج൫ΤϯδχΞͰͷ୲Λ͍ͯ͘͢͠͠Δ onExit: exit-handler templates: -
name: entrypoint steps: - - name: train-predict # MLΤϯδχΞ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLج൫ΤϯδχΞ͕࣮ - name: exit-handler # MLج൫ΤϯδχΞ͕࣮ steps: - - when: "{{workflow.status}} != Succeeded" name: notify-error
CASE: MLϞσϧͷ؆қతͳCD • ਪఆɾ༧ଌϞσϧͷ݁ՌϏϡʔϫ Deployment ͱͯ͠ӡ༻ • ਪఆॲཧྃ࣌ʹ kubectl set
env ͰϏϡʔϫʹ৽͍͠ϞσϧΛಡΈࠐ·ͤΔ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ templates: - name: entrypoint steps: - - name: train-predict - - name: import-to-db - - name: update-viewer - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
CASE: ॏ͍ɾෆ҆ఆͳMLॲཧΛѻ͏ • ਪఆɾ༧ଌϞσϧͳͲͰ Stan Λଟ༻ • ϝϞϦɾCPUΛେྔʹফඅ͢Δ߹ઐ༻ͷϊʔυͰ࣮ߦ • αϯϓϦϯά͕֬తʹࣦഊ͢ΔͷͰϦτϥΠɾλΠϜΞτ͕ඞཁ
- name: train-predict activeDeadlineSeconds: 28800 # 8h retryStrategy: limit: 2 nodeSelector: cloud.google.com/gke-nodepool: highmem-pool container: resources: limits: memory: "32Gi"
CASE: Ϟσϧਪఆͷಈతͳฒྻ࣮ߦ • όϯσΟοτπʔϧͰ࣮ࢪதͷςετ͝ͱʹਪఆॲཧ͕ඞཁ • ֤ςετͷਪఆॲཧΛಈతʹฒྻ࣮ߦ templates: - name: entrypoint
steps: # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ # લͷεςοϓͷग़ྗ͔ΒύϥϝʔλͷϦετΛಡΈࠐΈ - - withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷཁૉ͝ͱʹޙଓͷεςοϓΛ࣮ߦ name: calc-weights arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ӡ༻ TIPS Argo Workflow ͷ Web UI ͷΞΫηε • σϑΥϧτͰ
kubectl port-forward ͰΞΫηε͢Δඞཁ͕͋Δ • ΠϯλʔωοτΞΫηεΛՄೳʹ͢Δʹ Ingress ͰϩʔυόϥϯαΛཱͯΔ • GCP ͷ Identity-Aware Proxy Λ͏ͱϩʔυόϥϯαଆͰೝূΛ͔͚ΒΕΔ ݹ͍ϫʔΫϑϩʔͷΫϦʔϯΞοϓ • ࣮ߦࡁΈͷ Workflow ͱͦͷཧ͢Δ Pod Successful ͷ··Γଓ͚Δ • ఆظతʹݹ͍ Workflow Λআ͢Δ CronJob Λཱ͍ͯͯΔ • argo delete --older Φϓγϣϯ͕ศར
Pros, Cons, ·ͱΊ
Argo Workflow - Pros ଞͷϫʔΫϑϩʔΤϯδϯͱൺϩοΫΠϯ͞Εʹ͍͘ • ίϯςφԽ͞Ε͍ͯΕԿͰಈ͔ͤΔ • ࠓޙଞͷϫʔΫϑϩʔΤϯδϯ͕ग़͖ͯͯΓ͍͑͢ όονॲཧͱWebΞϓϦΛಉ͡ΫϥελͰཧͰ͖Δ
• σϓϩΠɾϩΪϯάɾϞχλϦϯάɾΤϥʔϨϙʔτͳͲΛҰݩԽ • ΦʔτεέʔϦϯάͳͲͱΈ߹ΘͤͯϦιʔεར༻ΛޮԽ
Argo Workflow - Cons ଞͷϫʔΫϑϩʔΤϯδϯ΄ͲϓϩάϥϚϒϧͰͳ͍ • Airflow, Luigi ͷΑ͏ʹ Python
DSL ͕ॻ͚ͨΓ͠ͳ͍ • ֤ΫϥυαʔϏεઐ༻ͷΦϖϨʔλ༻ҙ͞Ε͍ͯͳ͍ ࡞͞ΕͨϫʔΫϑϩʔΛଈ࣮࣌ߦ͢ΔҎ֎ͷػೳͨͳ͍ • ఆظ࣮ߦʹ CronJob ͳͲΛ͏ඞཁ͕͋Δ • Web UI ϞχλϦϯάͷΈͰϦτϥΠͳͲͷૢ࡞Ͱ͖ͳ͍ • ϫʔΫϑϩʔࣗମͷςϯϓϨʔτԽɾ࠶ར༻͕͠ʹ͍͘ • WorkflowTemplate ͕ఏҊ͞Ε͍ͯΔͷͰظ
·ͱΊ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ෳͷαʔϏεͰMLγεςϜΛར༻ • ଟ͘ͷεςοϓ͔ΒͳΔόονॲཧ͕ෳଘࡏ •
։ൃɾӡ༻ΛޮԽ͢ΔͨΊίϯϙʔωϯτΛׂͯ͠ίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • ίϯςφίϯϙʔωϯτΛΈ߹ΘͤͯϫʔΫϑϩʔΛߏங • MLγεςϜͷ։ൃɾӡ༻্ͷ߹ʹ߹Θ֤ͤͯछػೳΛ׆༻