Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ML on Kubernetes with Kubeflow #3 Kubeflow Pipe...

ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1

Kubernetes Meetup Tokyo #54 https://k8sjp.connpass.com/event/264501/ で発表したLT資料

Keita Watanabe

November 24, 2022
Tweet

More Decks by Keita Watanabe

Other Decks in Technology

Transcript

  1. Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning


    ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠
 ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
  2. Kubeflow Pipelines ʢKFPʣͱ͸ KFPͷߏ੒ཁૉ • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ

    Python SDK • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ Notebook αϙʔτ Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ UI: pipeline graph view Ұ෦ൈਮ
  3. KFP SDKͷόʔδϣϯʹ͍ͭͯ • 2022/11/24ݱࡏɺKFP SDKʹ͸ҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ
 v1ʢStable Statusʣ/ v2ʢBeta Statusʣ •

    KFP SDK v2͸ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ
 KFP v1্Ͱಈ࡞Մೳͱͨ͠΋ͷ • ͜͜Ͱ͸KFP SDK v2Λ༻͍Δ • KFP SDK v2ͷར༻ʹ͸KFP 1.6Ҏ߱
 ͕ඞཁʢࠓճ͸1.8.4Λ࢖༻ʣ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
  4. ͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ… • “/var/run/secrets/kube fl ow/pipelines/token” ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ • ࣮͸ϚϧνϢʔβʔ؀ڥͰSDKΛ࣮ߦ͢Δࡍ ʹ͸ɺServiceAccount token͕


    ”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢ Δඞཁ͕͋Δ
 ʢ͜ͷ؀ڥม਺ͷσϑΥϧτͷύε͕
 “/var/run/secrets/kube fl ow/pipelines/token”ʣ • σϑΥϧτͷNotebook؀ڥʹ͸͜ͷτʔΫϯ ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube fl ow-subfrom-inside-clustersub
  5. PodDefault • Kubernetesʹ͸Podʹରͯ͠ɺ ؀ڥม਺΍Volumeͷ৘ใͳͲ Λ͋ͱ͔Β௥Ճ͢ΔPodPreset ͱ͍͏Ϧιʔεʢݱࡏ
 Alpha stageʣ͕ଘࡏ͢Δ • PodDefault͸PodPreset૬౰ͷ

    ػೳΛKube fl owଆͰ
 ࣮૷ͨ͠΋ͷ https://github.com/kube fl ow/kube fl ow/blob/master/components/admission-webhook/README.md
  6. ࢀߟࢿྉ • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube fl ow-Pipelines • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube

    fl ow-subfrom-inside-clustersub • https://www.kube fl ow.org/docs/components/notebooks/overview • https://github.com/aws-samples/aws-do-kube fl ow • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2- compatibility/ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube fl ow-subfrom-inside-clustersub