ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

Kubernetes Meetup Tokyo #55 （https://k8sjp.connpass.com/event/267620/）にて発表した資料です。 引き続き、Kubeflow Pipelinesについて取り上げています。

Transcript

1. Keita Watanabe 2022/12/22 Kubernetes Meetup Tokyo # 55 LT KubeflowͰ͸͡ΊΔ
 ML on Kubernetes #4 Kube fl ow Pipelines: Part 2
2. Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning
 ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠
 ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
3. ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢ KubeflowͰ͸͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ • #2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines: Part1 (Kube fl ow Pipelinesͷಋೖฤ) • #4 Kube fl ow Pipelines: Part2 (Kube fl ow Pipelinesͷ࣮ફฤ) ࠓճʂ
4. લճ·Ͱͷ෮श: Kubeflow Pipelines (KFP)ͷಋೖ Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ લճͷ಺༰: KFPͷಋೖͱαϯϓϧύΠϓϥΠϯͷ࣮ߦ
5. SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ (1) ίϯϙʔωϯτͷఆٛ
 ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ
 ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔ܎Λఆٛ͢Δ (3) ύΠϓϥΠϯͷ࣮ߦ
 ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)
6. ೖग़ྗͷܕʹ͍ͭͯ • ೖग़ྗ͸ɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ • Parameters • Int, double, fl oat, str౳ɻ௨ৗύΠϓϥΠϯͷৼΔ෣͍Λ੍ޚ͢Δͷʹ༻͍ Δɻ-> ෳ਺ͷParametersΛฦ͍ͨ͠৔߹͸ʁ • Artifact • σʔλͷ΍ΓऔΓʹ༻͍Δɻ • ϑΝΠϧೖग़ྗͷྫ • ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
7. ෳ਺ͷValueΛฦ͍ͨ͠৔߹
8. ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ • Input/Output • δΣωϦοΫͳܕώϯτ • InputBinaryFile/OutputBinaryFile • ೖग़ྗ͕io.ByteIOͷͱ͖ • InputPath/OutputPath • ೖग़ྗ͕ϑΝΠϧ΁ͷύεͷͱ ͖ • InputTextFile/OutputTextFile • ೖग़ྗ͕io.TextIOWrapperͷͱ͖ ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
9. Componentͷߏஙํ๏ ͦͷ2: YamlΛॻ͘ ύΠϓϥΠϯͷఆٛ ίϯϙʔωϯτͷఆٛ
10. Component ಡΈࠐΈํ๏ ύΠϓϥΠϯͷఆٛ • load_component_from_ fi le
 ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url
 URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_text
 YamlϑΝΠϧΛString͔ΒಡΈࠐΉ
11. Component ಡΈࠐΈํ๏ • load_component_from_ fi le
 ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url
 URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_text
 YamlϑΝΠϧΛString͔ΒಡΈࠐΉ
12. ύΠϓϥΠϯͷ෼ذɾϧʔϓ
13. ࣍ճ͸… • Kube fl ow্Ͱͷ࣮ࡍͷML Model TrainingΛऔΓ্͍͛ͨͱࢥ͍·͢
14. ࢀߟࢿྉ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/component- development/ • https://medium.datadriveninvestor.com/conditional-kube fl ow-pipeline-for- dummies-972fd81f9de2 • https://github.com/kube fl ow/pipelines/blob/ d5b428416871fdc17325b1f04674b5ac62f17894/samples/core/ train_until_good/train_until_good.py