Share
Embed
WantedlyでFeature Storeを導入する際に考えたこと
第36回 MLOps 勉強会
https://mlops.connpass.com/event/300857/
Higuchi kokoro
November 29, 2023
Transcript
45 slides
-
©2023 Wantedly, Inc. 'FBUVSF4UPSFͱͳʹ͔ 8BOUFEMZʹಋೖͯ͠Ͳ͏͔ͩͬͨ ຊͷ͓͢͠Δ͜ͱ Part1 w 'FBUVSF4UPSFͱͳʹ͔ w
'FBUVSF4UPSFͷඪ४తͳ࣮ߏ w 'FBUVSF4UPSFϓϩμΫτͷհ 8BOUFEMZͰ'FBUVSF4UPSFΛಋೖͨ͠എܠ Part2 w 8BOUFEMZͷਪનγεςϜͷཁ݅ w ͜Ε·Ͱͷ՝ w 'FBUVSF4UPSFͷࣗ࡞ʹࢸͬͨഎܠ ࣗ࡞ͨ͠'FBUVSF4UPSFͷߏ Part3 w શମߏ w ֤ػೳͷ࣮ߏ Part4 w ࡞ͬͯྑ͔ͬͨ͜ͱ w ࣗ࡞ͯ͠ྑ͔ͬͨ͜ͱ w ͬͱΑ͍ͨ͘͠ͱ͜Ζ ࡞ͬͨॴײ 'FBUVSF4UPSF֓ཁ -
©2023 Wantedly, Inc. Ṥޱ৺ )JHVDIJ,PLPSP ॴଐ8BOUFEMZ JOD 9![FSFCPN@ σʔλαΠΤϯςΟετͱͯ͠ਪનγεςϜͷਫ਼վળɾධՁɾ
.-0QT·ΘΓͷ։ൃΛ͍ͯ͠·͢ ਖ਼͘͠πʔϧϓϩάϥϛϯάݴޠΛ͍ɺ ϩόετʹϓϩμΫτ։ൃ͢Δ͜ͱʹؔ৺͕͋Γ·͢ ࣗݾհ -
©2023 Wantedly, Inc. Create A World Where Work Drives Passion.
ڀۃͷదࡐదॴʹΑΓ γΰτͰίίϩΦυϧͻͱΛ;͢ ձࣾհ ࢲͨͪͷϛογϣϯ ©2023 Wantedly, Inc. -
©2023 Wantedly, Inc. ձࣾʹ༡ͼʹߦ͜͏ ňΧδϡΞϧ໘ஊʯͱ͍͏ࣄ୳͠ͷ৽ج४ • ਓͱνʔϜΛʮ͍ʯͰϚονϯά • ňΛฉ͖ʹߦ͘ʯମݧΛ௨ͯ͡ϑϥοτͳग़ձ͍Λग़ ਐԽΛଓ͚ΔUIͱج൫
• ίϯςϯπ͕ө͑ΔϞμϯͰγϯϓϧͳUIදݱ • ΑΓຊ࣭తͳϚονϯάΛՄೳʹ͢ΔਪનΞϧΰϦζϜ ఏڙϓϩμΫτ -
©2023 Wantedly, Inc. Կ͕Ͱ͖Δ Ұ؏ͨ͠ํ๏ͰσʔλΛॲཧ͠ɺ ෳͷϞσϧؒͰڞ༗Մೳͳ ಛྔΛੜɾཧͰ͖Δ 'FBUVSF4UPSFͱ
ͲΜͳγεςϜ ಛྔσʔλࣗମɺ ͦͷϝλσʔλΛޮతʹอଘɺ ΞΫηεɺ࠶ར༻͢ΔͨΊʹઃܭ ͞ΕͨγεςϜ ͏ͱͲ͏ͳΔ ػցֶशϞσϧ։ൃͷεϐʔυ ΞοϓɺσʔλͷҰ؏ੑͷ֬อɺ ͓ΑͼϞσϧੑೳͷվળʹد༩ ػցֶशͷͨΊͷಛྔσʔλͷதԝཧɾڞ༗γεςϜ -
©2023 Wantedly, Inc. 'FBUVSF4UPSFͷඪ४తͳ࣮ߏ ओཁͳͭͷίϯϙʔωϯτ 'FBUVSF3FHJTUSZ ಛྔϝλใͳͲΛҰཡͰ͖Δ ར༻ऀ͚ͷΠϯλʔϑΣʔεͷఏڙ 0QFSBUJPOBM.POJUPSJOH $PODFQU%SJGUͳͲϞσϧͷܦ࣌ྼԽ
Λݕ͢ΔಛྔͷϞχλϦϯάػߏ ref: https://neptune.ai/blog/feature-stores-components-of-a-data-science-factory-guide 5SBOTGPSN ੜσʔλΛಛྔʹม͢Δػߏ ཁ݅ʹΑΓมํ๏͕ҟͳΔ 4USFBNPS#BUDIͳͲ 4UPSBHF ಛྔΛอ͢Δػߏ#BUDIͳΒ %8) ϦΞϧλΠϜͳΒ3FEJTͳͲ 4FSWJOH ಛྔΛఏڙ͢ΔػߏϦʔΫ͕ ى͖ͳ͍Α͏ʹ࣌ܥྻΛ ࢀর͢Δͷ͋Δ QPJOUJOUJNF -
©2023 Wantedly, Inc. 044 w ʹ(PKFLͱ(PPHMF$MPVE͕ϦϦʔε .-ͱ%BUB*OGSBͷ w
σʔλΠϯϑϥΛநԽҙͷΠϯϑϥ ($1 "84 4OPX fl BLF Ͱར༻Մೳ σʔλϦʔΫΛճආՄೳ w QPJOUJOUJNFKPJO ޙड़ Λαϙʔτ ֶशͱαʔϏϯάͷҰ؏ੑ୲อ w ֶशɾਪ࣌ʹಉ͡ΠϯλʔϑΣʔεͰར༻Մೳ 'FBUVSF4UPSFϓϩμΫτհd'FBTUd 3FGIUUQTEPDTGFBTUEFWWNBTUFSXIBUJTGFBTU -
©2023 Wantedly, Inc. ϑϧϚωʔδυαʔϏε w #2ͷσʔλΞΫηεɾΨόφϯεΛ׆༻Ͱ͖Δ ϨΠςϯγͷαʔϏϯά w
ಛྔͷϦΫΤετͷ͕NTͰྃ .-ɾੜ"*ͷ྆ํʹରԠ w ಛྔͱಉ͡Α͏ʹϕΫτϧຒΊࠐΈ֨ೲՄೳ σʔλϦʔΫΛճආՄೳ w QPJOUJOUJNFKPJO ޙड़ Λαϙʔτ 'FBUVSF4UPSFϓϩμΫτհd7FSUFY"*d 3FGIUUQTDMPVEHPPHMFDPNWFSUFYBJEPDTGFBUVSFTUPSF IMKBWBJGT IUUQTDMPVEHPPHMFDPNCMPHKBQSPEVDUTBJNBDIJOFMFBSOJOHOFXWFSUFYBJGFBUVSFTUPSFCJHRVFSZQPXFSFEHFOBJSFBEZ -
©2023 Wantedly, Inc. ิQPJOUJOUJNFKPJO w ֶशɾਪ༻Ϩίʔυͷ༧ଌλΠϛϯάʹԊͬͯ ࠷৽͔ͭɺϦʔΫͷ͠ͳ͍λΠϛϯάͷಛྔΛ ར༻͢Δػೳ
w Ұൠతʹಛྔͱ+PJOରͷͲͪΒʹ UJNFTUBNQྻΛ࣋ͨͤͯࢀর͢Δ -
©2023 Wantedly, Inc. ิಛྔͷάϧʔϐϯά w ݕࡧੑϦιʔεཧͷͨΊɺಛྔ͍͔ͭ͘ͷ؍ Ͱάϧʔϐϯά͠͞ΕΔ͜ͱ͕ଟ͍ 7FSUFY"*ʹ͓͚Δάϧʔϐϯάྫ w &OUJUZ'FBUVSFΛඥ͚Δ୯Ґ
w FY Ϣʔβʔ ΞΠςϜ w 'FBUVSF(SPVQσʔλιʔεͷϩέʔγϣϯʹ جͮ͘άϧʔϓ w 'FBUVSF7JFXҙຯͷ୯Ґʹجͮ͘άϧʔϓ 3FGIUUQTDMPVEHPPHMFDPNWFSUFYBJEPDTGFBUVSFTUPSFMBUFTUPWFSWJFX -
©2023 Wantedly, Inc. ఏڙ͢ΔϢʔβʔମݧ w ϢʔβʹืूΛɺϦΫϧʔλʔʹީิऀΛਪન͢Δ w ީิͱͳΔΞΠςϜʹείΞ͚ͮ͠ɺϥϯΩϯάͱͯ͠ఏڙ͢Δ
νʔϜߏ w .-1E.ਓ %4ਓ ਪનج൫ΤϯδχΞਓ ݱঢ়ͷγεςϜߏ w 1ZUIPOͷKPCΛఆظ࣮ߦ͠ɺਪ݁ՌΛ#JH2VFSZʹ֨ೲ ओʹ%4 w #2ͷ݁ՌΛ3FEJTܦ༝ͰɺϢʔβʹ৴ ओʹਪનج൫ΤϯδχΞ ࣮࣌ͷཁٻ w 'FBUVSF4UPSFՃਫ਼վળΛؚΊϲ݄ ਓͰϦϦʔε w ಛྔ࡞Λߦ͏%4͕ࣗϝϯςͰ͖ΔγεςϜ͕·͍͠ 'FBUVSF4UPSFͷಋೖରͱͳͬͨਪનγεςϜͷཁ݅ɾલఏ -
©2023 Wantedly, Inc. ՝ͦͷಛྔʹ༧ଌલͷσʔλΛར༻Ͱ͖ͳ͍ w ैདྷͷ༧ଌλεΫ5SBJO7BMJE5FTUΛ࣌ܥྻׂ͠ɺ ༧ଌσʔλऩूظؒΛઃ͚͍ͯͨ w ༧ଌσʔλऩूظ͕ؒ͋ΔͨΊɺલͷߦಈ·ͰΛ
ೖྗʹؚΊΔ͜ͱ͕ग़དྷͳ͍ w ਫ਼վળͷͨΊʹɺϢʔβͷΠϕϯτൃੜλΠϛϯάͷ લ·ͰΛೖྗσʔλͷऩूظؒͱ͍ͨ͠ w ಛྔΛఆظతʹอଘ͠ɺϢʔβຖʹར༻͢Δ ೖྗظؒΛม͑Δඞཁ͕ग़͖ͯͨ -
©2023 Wantedly, Inc. 'FBUVSF4UPSFͷػೳͱ8BOUFEMZͷ՝ͱඞཁੑ ػೳ ݱঢ়ͷ՝ ඞཁੑ Feature Registry Ұݩཧ͍ͯ͠ͳ͍ͨΊɺ
Ͳ͜ͰͲͷಛྔ͕ΘΕ͍ͯΔͷ͔୳ͮ͠Β͍&࠶ར༻͠ʹ͍͘ ߴ Operational Monitoring ܽଛͳͲͷσάϨʹؾ͚ͮͳ͍ ͨͩ͠ΦϑϥΠϯςετͰ͋ΔఔΧόʔͰ͖Δ Storage ಛྔ͕ΫϥυετϨʔδʹ͔͓ͬͯ͠Βͣɺ σόοά࠶ར༻ɺϢʔβຖʹऔಘ࣌ظΛม͑Δ͜ͱ͕͍͠ ߴ Transform PythonϨΠϠʔͰBatchॲཧ͍ͯ͠ΔͷͰ͍ த Serving Servingػߏطʹ͋ΔͷͰඞཁͳ͍ ແ Ұൠతͳ'FBUVSF4UPSFͷػೳ͕શͯඞཁͰ͋ΔΘ͚Ͱແ͍ͱΘ͔ͬͨ💡 -
©2023 Wantedly, Inc. WT֎෦πʔϧ VertexAI Feast or ͦͷଞOSS Pros
ΧελϚΠζ͍͢͠ ඞཁͳ෦͚࣮ͩͰ͖Δ εέʔϥϒϧ ϦΞϧλΠϜͳಛྔͷαʔϏϯά ແྉɻίʔυΛશͯಡΊΔ Cons ࣮ίετ͕͔͔ΔɻυΩϡϝϯτશͯ ࣗͰ༻ҙ͢Δඞཁ͕͋Δ 2023/10ϦϦʔεͰ ใ͕গͳ͍ ྉ͕͔͔ۚΔ Transformػߏ݁ہ࣮͕ඞཁɻ ґଘϥΠϒϥϦͷߋ৽͕ ͍ ࠃͷར༻ࣄྫ͕͋·Γͳ͍ -
©2023 Wantedly, Inc. ࣗ࡞͕ಋೖอक ࠷ίετ͕খͦ͞͏ ࣗ࡞͢Δ͜ͱʹͨࠜ͠ڌ ࣗ࡞ˠطଘͷ Ҡߦͦ͢͠͏ w
طଘେنαʔϏεʹదԠͰ͖ΔΑ͏ʹػೳ͕ଟ͍ w FY ಛྔͷάϧʔϐϯάϦΞϧλΠϜαʔϏϯά w ϝϯςφϯε͞Ε͍ͯΔϥΠϒϥϦ͚ͩΛબΜͰಋೖͰ͖Δ w $IBU(15ʹΑΓɺ࣮ίετ͕Լ͕͍ͬͯΔ w 7FSUFY"*ಛྔ͕#2ʹΞοϓϩʔυ͞Ε͍ͯΔඞཁ͋Γ w ͲͷΈͪɺˢΛߦ͏ػߏඞཁ w ΑΓଞࣾࣄྫ͕ग़͖͔ͯͯΒஈ֊తʹҠߦՄೳ -
©2023 Wantedly, Inc. ࣗ࡞ͯ͠ྑ͔ͬͨ͜ͱ w ઃܭΛܦͯɺࣗࣾͰٻΊ͍ͯΔͷͷཁ͕݅໌֬ʹͳͬͨ w 1ZUIPOͷΈͰ࣮͞Ε͍ͯΔͷͰɺ%4͕ࣗཉ͍͠ػೳΛؾܰʹ13ͰՃͰ͖ΔΑ͏ʹͳͬͨ w
ಛྔੜʹؔ͢Δ։ൃੜ࢈ੑͷ্ͷऔΓΈ͕ҰΧॴʹूத͠ɺεέʔϧ͢ΔΑ͏ʹͳͬͨ w େنͳνʔϜߏΛఆ͍ͯ͠Δπʔϧ͕ଟ͍ͳ͔ɺඞཁेͳػೳΛબͯ͠ಋೖ͢Δ͜ͱ͕ग़དྷͨ w $IBU(15ͷ಄ʹΑΓɺίʔυΛॻ͘ίετ͕֨ஈʹԼ͕͍ͬͯΔͨΊɺૉૣ࣮͘Ͱ͖ͨ w 1BOEBT⁶#JH2VFSZ⁶ZBNMؒͷܕΛࣄલʹόϦσʔγϣϯͰ͖ΔͨΊɺ&SSPSܽଛ͕ݮͬͨ ࣗ࡞͍ͯ͠͠ͱײͨ͜͡ͱ w ಛྔΛϦΞϧλΠϜͰαʔϏϯά͢Δػߏͷࣗ࡞͔ͳΓͦ͠͏͓ͦΒ͘7FSUFY"*ʹͬͨํ͕Α͍ w ੜͨ͠ಛྔಉ࢜ΛKPJO͢ΔΫΤϦΛಈతʹ࡞ΔػߏෳࡶԽ͍͢͠ɻKJOKBUFNQMBUFͷ׆༻ͳͲ͕ඞཁͦ͏ ࣗ࡞ͨ͠ॴײ -
©2023 Wantedly, Inc. ·ͱΊ 'FBUVSF4UPSFʹؚ·ΕΔػೳؔ࿈ϓϩμΫτʹ͍ͭͯհͨ͠ w ಛྔσʔλࣗମɺϝλσʔλΛޮతʹอଘɺΞΫηεɺ࠶ར༻͢ΔͨΊʹઃܭ͞ΕͨγεςϜ w Ұ؏ͨ͠ํ๏ͰσʔλΛॲཧ͠ɺෳͷϞσϧؒͰڞ༗ՄೳͳಛྔΛੜͰ͖Δ 8BOUFEMZͰൺֱݕ౼ͨ݁͠Ռ'FBUVSF4UPSFΛͰ։ൃͨ͠
w ؔ࿈ϓϩμΫτͷશͯͷػೳ͕ඞཁͰແ͍͜ͱ͕Θ͔ͬͨͨΊ w ࣗ࡞͢Δ͜ͱͰඞཁेͳػೳ͚ͩΛ࣮Ͱ͖ͨ 'FBUVSF4UPSFΛಋೖ͢Δ͜ͱͰߴͳ։ൃͱਫ਼ͷվળͰ͖ͨ w σόοάϞχλϦϯάɺϨϏϡʔͷ͢͠͞ͳͲɺӡ༻্ͷϖΠϯ͕େ෯ʹվળ͞Εͨ w Ϣʔβຖʹ࠷৽ͷσʔλΛඥ͚Δ͜ͱ͕ग़དྷΔͨΊɺਫ਼͕վળ͞Εͨ