Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ML on Kubernetes with Kubeflow #4 Kubeflow Pipe...

Keita Watanabe
December 22, 2022

ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

Kubernetes Meetup Tokyo #55 (https://k8sjp.connpass.com/event/267620/)にて発表した資料です。 引き続き、Kubeflow Pipelinesについて取り上げています。

Keita Watanabe

December 22, 2022
Tweet

More Decks by Keita Watanabe

Other Decks in Technology

Transcript

  1. Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning


    ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠
 ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
  2. ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢ KubeflowͰ͸͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ •

    #2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines: Part1 (Kube fl ow Pipelinesͷಋೖฤ) • #4 Kube fl ow Pipelines: Part2 (Kube fl ow Pipelinesͷ࣮ફฤ) ࠓճʂ
  3. ೖग़ྗͷܕʹ͍ͭͯ • ೖग़ྗ͸ɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ • Parameters • Int, double, fl oat,

    str౳ɻ௨ৗύΠϓϥΠϯͷৼΔ෣͍Λ੍ޚ͢Δͷʹ༻͍ Δɻ-> ෳ਺ͷParametersΛฦ͍ͨ͠৔߹͸ʁ • Artifact • σʔλͷ΍ΓऔΓʹ༻͍Δɻ • ϑΝΠϧೖग़ྗͷྫ • ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
  4. ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ • Input/Output • δΣωϦοΫͳܕώϯτ • InputBinaryFile/OutputBinaryFile • ೖग़ྗ͕io.ByteIOͷͱ͖ •

    InputPath/OutputPath • ೖग़ྗ͕ϑΝΠϧ΁ͷύεͷͱ ͖ • InputTextFile/OutputTextFile • ೖग़ྗ͕io.TextIOWrapperͷͱ͖ ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
  5. Component ಡΈࠐΈํ๏ ύΠϓϥΠϯͷఆٛ • load_component_from_ fi le
 ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url


    URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_text
 YamlϑΝΠϧΛString͔ΒಡΈࠐΉ
  6. ࢀߟࢿྉ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/component- development/ • https://medium.datadriveninvestor.com/conditional-kube fl ow-pipeline-for-

    dummies-972fd81f9de2 • https://github.com/kube fl ow/pipelines/blob/ d5b428416871fdc17325b1f04674b5ac62f17894/samples/core/ train_until_good/train_until_good.py