Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Argo Workflow によるMLジョブ管理
Search
Livesense Inc.
PRO
March 27, 2019
Technology
2
720
Argo Workflow によるMLジョブ管理
MACHINE LEARNING Meetup KANSAI #4
2019/3/27
Livesense Inc.
PRO
March 27, 2019
Tweet
Share
More Decks by Livesense Inc.
See All by Livesense Inc.
株式会社リブセンス会社紹介資料 / Invent the next common.
livesense
PRO
1
4.2k
26新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
1
1.6k
中途セールス職_会社説明資料
livesense
PRO
0
88
EM候補者向け転職会議説明資料
livesense
PRO
0
41
コロナで失われたノベルティ作成ノウハウを復活させた話
livesense
PRO
0
140
株式会社リブセンス 会社説明資料(報道関係者様向け)
livesense
PRO
0
580
転職会議でGPT-3を活用した企業口コミ要約機能をリリースした話
livesense
PRO
0
1.2k
株式会社リブセンス マッハバイト_プレイブック
livesense
PRO
0
580
Tech Award 2021 選出方法
livesense
PRO
0
1.1k
Other Decks in Technology
See All in Technology
AWS版GitHub?Amazon CodeCatalystの全体像をまとめてみた
oshanqq
1
3.2k
デジタル化・DX推進あるある
y150saya
0
220
疎通2024
sadnessojisan
4
490
[RSJ24] Object Segmentation from Open-Vocabulary Manipulation Instructions Based on Optimal Transport Polygon Matching with Foundation Models
keio_smilab
PRO
0
120
すぐに始めるAWSコスト削減。短期でできる改善策と長期的な運用負荷軽減への取り組み方を解説
ncdc
1
120
いまからでも遅くない! コンテナでWebアプリケーションを 動かしてみよう(1)Gitハンズオン
nomu
0
280
Hyperledger Fabricの成長、成熟を振り返る / Looking back history of Hyperledger Fabric
gakumura
0
180
Practical GenAI with Go - Elastic and Golang Sydney
adriancole
0
130
スーパーマリオRPGのリメイク版の変更点からみるUX
nishiharatsubasa
1
110
LLMに日本語テキストを学習させる意義
ksaito
13
3.4k
EitherT_with_Future
aoiroaoino
0
380
20分で分かるIAM全機能 (拡大版) / 20240903-jawsug-yokohama-iam
opelab
3
120
Featured
See All Featured
Designing the Hi-DPI Web
ddemaree
278
34k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
45
4.8k
4 Signs Your Business is Dying
shpigford
179
21k
Designing Experiences People Love
moore
138
23k
Imperfection Machines: The Place of Print at Facebook
scottboms
263
13k
Navigating Team Friction
lara
183
13k
Unsuck your backbone
ammeep
667
57k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
325
21k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.4k
Optimising Largest Contentful Paint
csswizardry
28
2.7k
The Mythical Team-Month
searls
218
43k
GraphQLとの向き合い方2022年版
quramy
43
13k
Transcript
Argo Workflow ʹΑΔMLδϣϒཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε (ژΦϑΟε)
MACHINE LEARNING Meetup KANSAI #4 LT
͜Εͷհ͠·͢
https://argoproj.github.io/
Կ͕Ͱ͖Δͷ͔ "Container native workflow engine for Kubernetes" • ෳͷίϯςφΛྻ/ฒྻ࣮ߦ͢ΔϫʔΫϑϩʔΛఆٛͰ͖Δ •
σʔλύΠϓϥΠϯ, CI/CD ͳͲͷར༻Λఆ • ৽όʔδϣϯͰ DAG αϙʔτ • Argo ϕʔεͷ༷ʑͳϓϩμΫτ • Argo CD: GitOps ʹΑΔ CD Λ࣮ݱ • Argo Event: ϫʔΫϑϩʔͷτϦΨ
apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: ml-workflow- spec: entrypoint: main
templates: - name: main steps: - - name: load-dataset template: load-dataset - - name: train-model-1 template: train-model arguments: parameters: [{name: model, value: model1}] - name: train-model-2 template: train-model arguments: parameters: [{name: model, value: model2}] ...
... - name: load-dataset container: image: postgres:latest command: [sh, -c]
args: ["psql db -c 'SELECT * FROM dataset' -A -F, > dataset.csv"] - name: train-model inputs: parameters: [{name: model}] container: image: train-model command: [sh -c] args: ["python train_model.py --model={{inputs.parameters.model}}"]
None
ͳͥ͏ͷ͔ ʮϞσϧ͕Ͱ͖ͨͷͰɺαΫοͱӡ༻ʹ͍ͤͨʯ • MLϞσϧͷ։ൃऀ • SQL Ͱσʔλऔಘ ʙ Ϟσϧ༧ଌΛϑΝΠϧʹग़ྗ •
Docker Ͱಈ͘Α͏ʹ͓ͯ͘͠ • MLγεςϜͷ։ൃऀ • DBIO Ϟσϧɾ༧ଌ݁ՌͷσϦόϦॲཧΛ࣮ • Argo Ͱͯ͢ΛΈ߹ΘͤͨϫʔΫϑϩʔΛ࡞Δ → ίϯςφ୯ҐͰׂ୲
ϦϒηϯεͰͷར༻ྫ • ग़ྗͷDBॻ͖ࠐΈॲཧͷ • Ϟσϧͷ Continuous Delivery • ฒߦॲཧ
ग़ྗͷDBॻ͖ࠐΈॲཧͷ • ٻਓαΠτͷݕࡧॱҐ੍ޚ༻༧ଌϞσϧ • όονͰֶशɾ༧ଌ͠ग़ྗΛDBʹॻ͖ࠐΈ • Ϟσϧͷ։ൃऀCSVग़ྗ·Ͱ࣮ͯ͠ Docker Խ͓ͯ͘͠ •
ॻ͖ࠐΈॲཧΫϨσϯγϟϧཧγεςϜͷ։ൃऀ͕࣮ steps: - - name: train-model # MLϞσϧͷ։ൃऀ͕࣮ - - name: predict-rates # MLϞσϧͷ։ൃऀ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLγεςϜͷ։ൃऀ͕࣮ # ※ग़ྗϑΝΠϧڞ༗ϘϦϡʔϜͰड͚͠
Ϟσϧͷ Continuous Delivery • Ӧۀઓུɾࠂग़ߘΛఆͨ͠ٻਓޮՌਪఆϞσϧ • ϚʔέςΟϯά୲ऀ͚ͷϏϡʔϫΛ R-Shiny Ͱ։ൃɾӡ༻ •
ਪఆॲཧ͕ྃ͢ΔͨͼʹϏϡʔϫΛσϓϩΠͯ͠ϞσϧΛߋ৽ steps: - - name: estimate # ਪఆॲཧ - - name: upload-model # ࡞͞ΕͨϞσϧΛετϨʔδʹอଘ - - name: update-viewer # ϏϡʔϫΛσϓϩΠ͢͠
Ϟσϧͷ Continuous Delivery (ଓ͖) • Ϗϡʔϫಉ͡ Kubernetes ΫϥελͰ Deployment ͱ͍ͯಈ͍͍ͯΔ
• kubectl set env Ͱ Deployment Λߋ৽͢Δ͜ͱͰ৽͍͠ϞσϧΛಡΈࠐΉ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
ฒߦॲཧ • WebςετͷଟόϯσΟοτ࠷దԽͷॏΈߋ৽δϣϒ • ෳͷςετ͕͓ͬͯΓɺ֤ςετͷਪఆॲཧฒߦ࣮ߦ͍ͨ͠ steps: - - name: list-experiments
# ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: calc-weights # ͜ΕΛϦετΞοϓ͞Εͨςετͷ͚ͩฒߦ࣮ߦ͢Δ # ग़ྗύϥϝʔλͷϦετΛ͢ͱͦͷ͚ͩίϯςφ্ཱ͕͕ͪΔ # Ϧετ [{"experimentId": 1}, {"experimentId": 2}] ͷΑ͏ͳ JSON withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷ֤ΞΠςϜ͔ΒύϥϝʔλΛऔΓग़ͯ͢͠ arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ฒߦॲཧ (ଓ͖) templates: - name: list-experiments container: ... outputs: parameters:
- name: experiments # ग़ྗύϥϝʔλͷϦετΛϑΝΠϧࢦఆ valueFrom: {path: /output/experiments.json} - name: calc-weights container: ... inputs: parameters: # ύϥϝʔλΛೖྗͱͯ͠ड͚औΔ - name: experimentId
None
·ͱΊ • ෳίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ؆୯ʹΊΔ • ͭͬͨ͘MLϞσϧΛ͘͢ӡ༻͍ͨ͠ͱ͖ʹศར هࣄ͋Γ·͢: Argo ʹΑΔίϯςφωΠςΟϒͳσʔλύΠϓϥΠϯͷϫʔΫϑϩʔཧ