推薦のためのdata pipelineのモニタリングとロギング

Transcript

1. ©2019 Wantedly, Inc. ਪનͷͨΊͷdata pipelineͷ ϞχλϦϯάͱϩΪϯά Apr 16, 2019 - Hazumi Ichijo Data Pipeline Casual Talk #2
2. ©2018 Wantedly, Inc. ࣗݾ঺հ Ұᑍ
   ୺੅
   !IB[VNJSS!SFSPTU
   %BUB
   &OHJOFFS
   JO
   8BOUFEMZ
   (P
   3BJMT
   3FBDUʜ
   ๺ւಓग़਎
3. ©2019 Wantedly, Inc.  ͳͥϞχλϦϯάɾϩΪϯά͢Δ͔
    ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔
    ͲΕ͘Β͍कΔ΂͖͔
    ԿΛ͢΂͖͔
4. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    σʔλ
   
   σʔλ
    σʔλ
    αʔϏε
5. ©2019 Wantedly, Inc. ໰୊͕ൃੜͨ͠ͱ͖Ͳ͏͠·͢ʁ
6. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    +PC
    +PC
    +PC
    σʔλ
   
   σʔλ
    σʔλ
   
7. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    +PC
    +PC
    +PC
    σʔλ
   
   σʔλ
    σʔλ
   
8. ©2019 Wantedly, Inc. +PC
    +PC
    +PC
   
   +PC
    +PC
    +PC
    +PC
    +PC
    σʔλ
   
   σʔλ
    σʔλ
    ʁ ʁ ʁ
9. ©2019 Wantedly, Inc.  ͦ΋ͦ΋ͳ͍
    ζϨΔʢྫ
   ݹ͍σʔλΛݩʹͨ͠ਪન͕ग़Δʣ ग़ͯ͘Δ঱ঢ় ਪનͷͨΊͷσʔλ͕
10. ©2019 Wantedly, Inc.  ͦ΋ͦ΋ͳ͍
     ζϨΔʢྫ
    ݹ͍σʔλΛݩʹͨ͠ਪન͕ग़Δʣ ग़ͯ͘Δ঱ঢ় ਪનͷͨΊͷσʔλ͕ 
    ؾ͖ͮʹ͍͘
11. ©2019 Wantedly, Inc.  ໰୊͕ൃੜͨ͠ࡍʹؾ͖ͮʹ͍͘
     ໰୊͕ൃੜͨ࣌͠ͷௐࠪํ๏͕໘౗ ͳͥϞχλϦϯάɾϩΪϯάΛ͢Δ͔ ਪનͷͨΊͷσʔλ͸
12. ©2019 Wantedly, Inc.  ໰୊͕ൃੜͨ͠ࡍʹؾ͖ͮʹ͍͘
     ໰୊͕ൃੜͨ࣌͠ͷௐࠪํ๏͕໘౗ ͳͥϞχλϦϯάɾϩΪϯάΛ͢Δ͔ ਪનͷͨΊͷσʔλ͸ 
    ϞχλϦϯάɾϩΪϯάΛͯ͠໰୊ൃੜ࣌ʹඋ͑Δ
13. ©2019 Wantedly, Inc. ԿΛϞχλϦϯάɾϩΪϯά͢Δʁ
14. ©2019 Wantedly, Inc.  ຖճܾΊΒΕͨ৔ॴʹσʔλ͕ग़ྗ͞ΕΔ΂͖
     ग़ྗ͞ΕΔσʔλ͕ਖ਼͍͠ඞཁ͕͋Δ
    w ґଘ͢Δσʔλ͕࠷৽ͷঢ়ଶ
    w ܾΊΒΕͨ࣌ࠁʹ࣮ߦ͞Εͯ΄͍͠
    w FUD
    ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ਪનͷͨΊͷσʔλ͸
15. ©2019 Wantedly, Inc.  ຖճܾΊΒΕͨ৔ॴʹσʔλ͕ग़ྗ͞ΕΔ΂͖
     ग़ྗ͞ΕΔσʔλ͕ਖ਼͍͠ඞཁ͕͋Δ
    w ґଘ͢Δσʔλ͕࠷৽ͷঢ়ଶ
    w ܾΊΒΕͨ࣌ࠁʹ࣮ߦ͞Εͯ΄͍͠
    w FUD
    ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ਪનͷͨΊͷσʔλ͸ 
    ϞχλϦϯάɾϩΪϯάΛͯ͠कΕΔΑ͏ʹ͢Δ
16. ©2019 Wantedly, Inc.  ΞϓϦέʔγϣϯ
    w δϣϒ࣮ߦ࣌ͷ࣮ߦঢ়گͷ஋ʢྫ
    ֶश཰ʣ
    w ΦϯϥΠϯධՁͷ஋
    w ॻ͖ग़͠ઌʹσʔλ͕ॻ͖ࠐ·Ε͔ͨ
    w Τϥʔ௨஌
    w Կʹ͕͔͔࣌ؒΔ͔
     Πϯϑϥ
    w δϣϒ͕੒ޭ͔ͨ͠ɾࣦഊ͔ͨ͠ɾ࣮ߦ͞Ε͔ͨ
    w ઃఆ͍ͯ͠Δ࣌ؒʹ࣮ߦ͞Ε͔ͨ ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ϞχλϦϯά
17. ©2019 Wantedly, Inc. w ॲཧͷ಺༰͕Θ͔Δ΋ͷʢྫ
    42-ʣ
    w δϣϒͷ։࢝ɾऴྃ
    w ॻ͖ग़ͨ͠σʔλ
    w FUD ԿΛϞχλϦϯάɾϩΪϯά͢΂͖͔ ϩΪϯά
18. ©2019 Wantedly, Inc. ͲΕ͘Β͍कΒΕΔ΂͖΋ͷʁ
19. ©2019 Wantedly, Inc.  αʔϏεશମ͕࢖͑ͳ͘ͳΔ
     ,1*ʹ99Өڹ͕ͰΔ
     Ұ෦ͷػೳ͕࢖͑ͳ͘ͳΔ ͲΕ͘Β͍कΔ΂͖͔ ͋Δਪનʹ໰୊͕͋Δͱɾɾɾ
20. ©2019 Wantedly, Inc.  αʔϏεશମ͕࢖͑ͳ͘ͳΔ
     ,1*ʹ99Өڹ͕ͰΔ
     Ұ෦ͷػೳ͕࢖͑ͳ͘ͳΔ ͲΕ͘Β͍कΔ΂͖͔ ͋Δਪનʹ໰୊͕͋Δͱɾɾɾ 
    ͦ΋ͦ΋ͦͷσʔλͷॏཁ౓͕Θ͔Δඞཁ͕͋Δ
21. ©2019 Wantedly, Inc. ԿΛ͢΂͖͔
22. ©2019 Wantedly, Inc.  ॏཁ౓ΛͰ͖Δ͚ͩ໌֬ʹ͓ͯ͘͠
     ϞχλϦϯάɾϩΪϯάͷ߲໨ચ͍ग़͠
     ΠϯϑϥνʔϜͱڠྗͯ͠ɺ༧Ί
    w ॏཁ౓Λ఻͑Δ
    w ϞχλϦϯάɾϩΪϯάͷλεΫͷׂΓৼΓΛߦ͏
    w ໰୊͕ൃੜͨ࣌͠ɺͲ͏͍ͬͨϑϩʔͰ֬ೝ͢Δ͔ɺͲ͏ରॲ ͢Δ͔Λ఻͑Δ
    
    ඞཁͰ͋Ε͹࣮ࡍʹϦϋʔαϧΛߦ͏ ԿΛ͢΂͖͔