Kubernetes (EKS) で構築する スケーラブルなジョブ実行基盤

Transcript

1. Kubernetes (EKS) Ͱߏங͢Δ
 εέʔϥϒϧͳδϣϒ࣮ߦج൫ Kenta Suzuki (twitter: @kekekenta) CTO, primeNumber inc 2019/8/28 Kubernetes Meetup Tokyo #22
2. ࣗݾ঺հ • Kenta Suzuki • twitter: @kekekenta • CTO, primeNumber inc.
 εϥΠυͷ࠷ޙʹձࣾ঺հࡌͤ·ͨ͠ͷͰɺ͝ڵຯ͋Γ·ͨ͠Βޚཡ͍ͩ͘͞ •෼ੳج൫ߏஙɾӡ༻ࢧԉSaaSʮtrocco®ʯ
3. .Z42- 4 4BMFTGPSDF #JH2VFSZ ෼ੳج൫ߏஙɾӡ༻ σʔλసૹ αϚϦʔςʔϒϧ࡞੒
   σʔλ؂ࢹ
   ֎෦࿈ܞ
4. .Z42- 4 4BMFTGPSDF #JH2VFSZ σʔλసૹ αϚϦʔςʔϒϧ࡞੒
   σʔλ؂ࢹ
   ֎෦࿈ܞ ෼ੳج൫ͷߏஙɾӡ༻Λ·ͱΊͯΧόʔ
5. troccoͷσʔλసૹδϣϒ • σʔλιʔε • MySQLɺPostgreSQLɺSQLServerͳͲ • εέδϡʔϧ࣮ߦ • ࠷୹1෼ִؒ ೚ҙͷσʔλιʔεͷςʔϒϧ౳ͷ಺༰ΛDWHʹసૹ troccoͷσʔλసૹ෦෼͸OSSͷembulkΛଟ༻͍ͯ͠·͢ɻ
 ͜ͷ৔ΛआΓͯɺίϛολʔͷํʑɺembulkίϛϡχςΟͷํʑʹײँਃ্͛͠·͢ɻ ࠓޙ͸embulkͷpluginެ։౳Ͱߩݙ͍ͯ͘͠༧ఆͰ͢ʂ
6. troccoͷసૹδϣϒ • δϣϒ࣮ߦ਺ • ʙ10,000/೔ • δϣϒ͝ͱͷ࣮ߦ࣌ؒ • min: 5s ʙ max: 5h • ࣮ߦ࣌ؒଳ • ਺࣌ؒʹ1౓ͷεύΠΫ
7. Fan out on ECS چΞʔΩςΫνϟ
8. Fan out on ECSαʔϏε SQS εέδϡʔϥʔ 8PSLFS
   &$4
   UBTL 8PSLFS
   &$4
   UBTL 8PSLFS
   &$4
   UBTL ECS ECS NODE δϣϒ polling
9. ໰୊఺
10. λεΫεέʔϧΞ΢τͷλΠϛϯά೉͍͠ • SQSͷΩϡʔͷαΠζΛ؂ࢹ • CloudwatchϝτϦΫε͕࠷୹1෼͝ͱ • εέʔϧ·Ͱɺ1෼଴ͨͳ͍ͱߦ͚ͳ͍
11. ϗετεέʔϧΞ΢τͷλΠϛϯά೉͍͠ • λεΫ͕΋͏௥ՃͰ্ཱ͕ͪΔϦιʔε͕ ͳ͘ͳͬͨΒεέʔϧ • ىಈ଴ͪͷλεΫ͕ग़དྷͨλΠϛϯάͰϗ ετεέʔϧͯ͠΄͍͠ɻɻ
12. λεΫεέʔϧΠϯ೉͍͠ • ECSλεΫΛ҆શʹऴྃͤ͞Δඞཁ͕͋Δ • సૹδϣϒ͸5hʹٴͿ৔߹΋͋Δ • TERMγάφϧͷτϥοϓͯ͠graceful shutdownΛ ࣮૷ • ͨͩ͠ɺECSͷίϯιʔϧ্ͷ࣮ߦதλεΫҰཡ͔ΒɺTERMΛૹͬͯ͠͹Β͘͢Δ ͱফ͑ͯ͠·͏
13. ϗετͷεέʔϧΠϯ΋೉͍͠ • Auto Scaling Group ͷ Life cycle hook Λ࢖ͬͯɺΠϯ ελϯεఀࢭલʹDRAININGΛ࣮ࢪ • ىಈதͷλεΫ͕ͳΔ·Ͱ؂ࢹ͠ɺͳ͘ͳͬͨΒΠϯε λϯεఀࢭΛ͢͢ΊΔ • લड़ͷ௨ΓɺىಈதͷλεΫҰཡ͔Βফ͑ΔͨΊ͝ʹΐ ͝ʹΐ΍ͬͯͨ
14. Ϧιʔεׂ౰ • సૹߴ଎ԽΦϓγϣϯͱͯ͠ɺCPU/ϝϞϦ্ݶΛઃఆ Ͱ͖ΔΑ͏ʹ͍ͨ͠ • ECSͰ΋λεΫ͝ͱʹϦιʔε্ݶΛઃఆͰ͖Δ͕ɺى ಈ࣌ͷΈ • Ϧιʔε্ݶ͝ͱʹαʔϏεΛ෼͚Δඞཁ͕͋Γ
15. ೉͍͠
16. ͦ͜Ͱɺk8s
17. k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ
18. k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ
19. k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ statusͷભҠ "Pending" -> "Running" -> "Succeeded"
20. k8sͷpodεέδϡʔϥʔ • nodeϦιʔεͷۭ͖ঢ়گʹԠͯ͡ɺεέδϡʔϥʔ͕ॱ࣍podΛ nodeʹׂΓ౰ͯͯ͘ΕΔ • podΛ࡞੒͓͚ͯ͠͹ɺ͋ͱ͸࣮ߦ͞ΕΔͷΛ଴͚ͭͩ ,VCFSOFUFT QPE /0%& QPE QPE pod্ཱͪ͛ ࣮ߦ଴ͪpod create pod
21. ͱΓ͋͑ͣࡶʹpodΛcreate͢Ε ͹ɺ͋ͱ͸k8s͕࣮ߦ·Ͱ໘౗ݟ ͯ͘ΕΔ
22. ϊʔυͷεέʔϧΞ΢τ/Πϯ • EKSͷ৔߹ɺcluster autoscalerΛ࢖͏ͷ͕σϑΝΫτʢͱ ࢥΘΕΔʣ
 https://github.com/kubernetes/autoscaler/tree/master/ cluster-autoscaler/cloudprovider/aws • podεέδϡʔϦϯάͷঢ়گʹԠͯ͡ɺEC2ϊʔυͷASG ܦ༝ͰεέʔϧΠϯɾΞ΢τΛ΍ͬͯ͘ΕΔ
23. ϊʔυͷεέʔϧΞ΢τ QPE &$
    /0%& QPE QPE
    1FOEJOH ࠓͷnodeϦιʔεͰ͸ɺ
 podΛ্ཱͪ͛Δ͜ͱ͕ग़དྷͳ͍
24. ϊʔυͷεέʔϧΞ΢τ QPE &$
    /0%& QPE &$
    /0%& QPE ClusterAutoscaler͕ASGܦ༝ͰnodeΛεέʔϧΞ΢τ
25. ϊʔυεέʔϧΠϯ QPE &$
    /0%& QPE &$
    /0%& QPE
26. ϊʔυεέʔϧΠϯ QPE &$
    /0%& QPE &$
    /0%& Ϧιʔεʹ༨༟͕͋Δ
27. ϊʔυεέʔϧΠϯ QPE &$4
    /0%& QPE εέʔϧΠϯ
28. 1. create pod ࣮ͯ͠ߦΛ଴ͭ 2. ඞཁʹԠͯ͡ɺnode͕εέʔϧΞ΢ τ͢ΔͷΛ଴ͭ εέʔϥϒϧδϣϒج൫ര஀
29. k8sʹΑΔδϣϒج൫
30. ݱࡏͷΞʔΩςΫνϟ SQS εέδϡʔϥʔ 8PSLFS
    1PE +PC
    %JTQBUDIFS Kubernetes δϣϒ create pod schuedule pod &,4
    /0%& 8PSLFS
    1PE $MVTUFS
    "VUPTDBMFS Scale in/out
31. Job Dispatcher SQS εέδϡʔϥʔ 8PSLFS
    1PE Kubernetes δϣϒ create pod schuedule pod &,4
    /0%& 8PSLFS
    1PE $MVTUFS
    "VUPTDBMFS Scale in/out +PC
    %JTQBUDIFS
32. Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒
33. Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ LT
    "1*ʹϦΫΤετ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒
34. Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ EPDLFSΠϝʔ δͷλάࢦఆ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒
35. Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ QPEͰ࣮ߦ͢Δ ίϚϯυΛࢦఆ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒
36. Job Dispatcher • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒ ˞ίʔυ͸ΠϝʔδͰ͢ $16ϝϞϦࢦఆ
37. Worker pod SQS εέδϡʔϥʔ +PC
    %JTQBUDIFS Kubernetes δϣϒ create pod schuedule pod &,4
    /0%& $MVTUFS
    "VUPTDBMFS Scale in/out 8PSLFS
    1PE 8PSLFS
    1PE
38. Worker pod • ࣮ࡍͷసૹΛ࣮ߦ • సૹ։࢝Ͱ্ཱ͕ͪΓɺసૹ͕ऴྃͨ͠Βpod΋ऴྃ ˞ίʔυ͸ΠϝʔδͰ͢
39. Cluster autoscaler SQS εέδϡʔϥʔ +PC
    %JTQBUDIFS Kubernetes δϣϒ create pod schuedule pod &,4
    /0%& Scale in/out 8PSLFS
    1PE 8PSLFS
    1PE $MVTUFS
    "VUPTDBMFS
40. Cluster autoscaler ˞ίʔυ͸ΠϝʔδͰ͢
41. k8sδϣϒج൫ಋೖͷ੒Ռ • ѹ౗తίετ࡟ݮʹ੒ޭ • εέʔϧͷλΠϛϯά͕೉͍͠໰୊͕͍͍ ײ͡ʹղܾͰ͖ͨ • cluster autoscaler༷༷ • ։ൃ؀ڥͱຊ൪؀ڥͷಉҰԽ
42. ؂ࢹʹ͍ͭͯ • create podͰ͖ͳ͍ • pod͕pendingͷ··্ཱ͕ͪΒͳ͍ • pod͕runningதʹಥવࢮ ຊ൪ӡ༻࣮੷͕ແ͍ͷͰɺຊ౰ʹظ଴௨Γʹ ಈ͔͘ͱ͍͏ҙຯͰ؂ࢹΛ࣮ࢪ
43. troccoͷδϣϒεςʔλεભҠ • DB্ʹδϣϒ͝ͱͷεςʔλεΛอ࣋ • dispatcher΍podͳͲ͕ͦΕͧΕߋ৽͢Δ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE SQSʹΩϡʔΠϯά create pod ׬ྃ pod্ཱ͕ͪΓ·Γ pod͕࣮ߦத సૹ׬ྃ
44. redashͰͷ؂ࢹ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE ֤εςʔλεͰ଺ཹ͍ͯͨ͠ΒΞϥʔτ
45. ·ͱΊ • γϯϓϧ • k8sͷݞʹ৐Ε͹ɺεέʔϥϒϧͳδϣϒج൫ߏங΋γϯϓ ϧʹͳΔ • ؂ࢹ • ӡ༻࣮੷͕ແ͍ͷͰɺͱΓ͋͑ͣऔΕΔσʔλ͸ͱ͓ͬͯ͘ • k8s΁ͷҠߦΛৼΓฦͬͯ • όον͔Βk8sಋೖ͢Δͷ͸ׂͱ΍Γ΍͍͢ͷͰ͓͢͢Ί • কདྷతʹ • কདྷతʹ͸ɺϗετͷ؅ཧͱ͔΍Βͳͯ͘ྑ͘ͳΔͱخ͍͠
46. ͝ਗ਼ௌ ͋Γ͕ͱ͏͍͟͝·ͨ͠ :bow Α͚Ε͹ձࣾ঺հ΋ޚཡ͍ͩ͘͞
47. גࣜձࣾ primeNumber • 2015೥11݄ ઃཱ • ࣾһ9໊ʢ͏ͪΤϯδχΞ8໊ʣ
 ύʔτφʔΤϯδχΞ30໊Ҏ্ • 2018೥11݄ trocco®ϦϦʔε • 2019೥7݄ γϦʔζAࢿۚௐୡʢ໿1.3ԯԁʣ࣮ࢪ https://primenumber.co.jp
48. ಇ͘؀ڥ • @த໨ࠇͷ઒Ԋ͍ • ίΞλΠϜ੍ • 11ʙ15࣌ • िத೔ɺ2೔·ͰϦϞʔτՄ ※ࢼ༻த • δϜશֹิॿ
49. We are hiring • ιϑτ΢ΣΞΤϯδχΞ • ΠϯϑϥΤϯδχΞ • ͳͲͳͲɺଟํ໘ઈࢍืूத