Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes (EKS) で構築する スケーラブルなジョブ実行基盤
Search
Kenta Suzuki
August 28, 2019
Technology
7
6.3k
Kubernetes (EKS) で構築する スケーラブルなジョブ実行基盤
Kenta Suzuki
August 28, 2019
Tweet
Share
More Decks by Kenta Suzuki
See All by Kenta Suzuki
trocco®での インシデント重大度表(SEVレベル表)運用
kekekenta
0
3.2k
Ruby はじめての型 Gotanda.rb 2023/04/20
kekekenta
0
280
trocco® の品質を守る、とても普通な取り組み
kekekenta
0
5.3k
Embulkを利用したデータ統合SaaSの構築と運用
kekekenta
2
4.9k
Other Decks in Technology
See All in Technology
Backlog ユーザー棚卸しRTA、多分これが一番早いと思います
__allllllllez__
1
130
Zephyr RTOSを使った開発コンペに参加した件
iotengineer22
1
200
Model Mondays S2E03: SLMs & Reasoning
nitya
0
350
Beyond Kaniko: Navigating Unprivileged Container Image Creation
f30
0
130
開発生産性を測る前にやるべきこと - 組織改善の実践 / Before Measuring Dev Productivity
kaonavi
0
190
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
2
7.7k
品質と速度の両立:生成AI時代の品質保証アプローチ
odasho
1
200
怖くない!はじめてのClaude Code
shinya337
0
380
asken AI勉強会(Android)
tadashi_sato
0
180
OPENLOGI Company Profile
hr01
0
67k
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
2
310
B2C&B2B&社内向けサービスを抱える開発組織におけるサービス価値を最大化するイニシアチブ管理
belongadmin
1
5.8k
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
PRO
54
11k
The Pragmatic Product Professional
lauravandoore
35
6.7k
Thoughts on Productivity
jonyablonski
69
4.7k
Making Projects Easy
brettharned
116
6.3k
Rebuilding a faster, lazier Slack
samanthasiow
82
9.1k
Automating Front-end Workflow
addyosmani
1370
200k
Rails Girls Zürich Keynote
gr2m
94
14k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Designing for humans not robots
tammielis
253
25k
How to train your dragon (web standard)
notwaldorf
94
6.1k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Transcript
Kubernetes (EKS) Ͱߏங͢Δ εέʔϥϒϧͳδϣϒ࣮ߦج൫ Kenta Suzuki (twitter: @kekekenta) CTO, primeNumber
inc 2019/8/28 Kubernetes Meetup Tokyo #22
ࣗݾհ • Kenta Suzuki • twitter: @kekekenta • CTO, primeNumber
inc. εϥΠυͷ࠷ޙʹձࣾհࡌͤ·ͨ͠ͷͰɺ͝ڵຯ͋Γ·ͨ͠Βޚཡ͍ͩ͘͞ •ੳج൫ߏஙɾӡ༻ࢧԉSaaSʮtrocco®ʯ
.Z42- 4 4BMFTGPSDF #JH2VFSZ ੳج൫ߏஙɾӡ༻ σʔλసૹ αϚϦʔςʔϒϧ࡞ σʔλࢹ ֎෦࿈ܞ
.Z42- 4 4BMFTGPSDF #JH2VFSZ σʔλసૹ αϚϦʔςʔϒϧ࡞ σʔλࢹ ֎෦࿈ܞ ੳج൫ͷߏஙɾӡ༻Λ·ͱΊͯΧόʔ
troccoͷσʔλసૹδϣϒ • σʔλιʔε • MySQLɺPostgreSQLɺSQLServerͳͲ • εέδϡʔϧ࣮ߦ • ࠷1ִؒ ҙͷσʔλιʔεͷςʔϒϧͷ༰ΛDWHʹసૹ
troccoͷσʔλసૹ෦OSSͷembulkΛଟ༻͍ͯ͠·͢ɻ ͜ͷΛआΓͯɺίϛολʔͷํʑɺembulkίϛϡχςΟͷํʑʹײँਃ্͛͠·͢ɻ ࠓޙembulkͷpluginެ։Ͱߩݙ͍ͯ͘͠༧ఆͰ͢ʂ
troccoͷసૹδϣϒ • δϣϒ࣮ߦ • ʙ10,000/ • δϣϒ͝ͱͷ࣮ߦ࣌ؒ • min: 5s
ʙ max: 5h • ࣮ߦ࣌ؒଳ • ࣌ؒʹ1ͷεύΠΫ
Fan out on ECS چΞʔΩςΫνϟ
Fan out on ECSαʔϏε SQS εέδϡʔϥʔ 8PSLFS &$4UBTL 8PSLFS &$4UBTL
8PSLFS &$4UBTL ECS ECS NODE δϣϒ polling
λεΫεέʔϧΞτͷλΠϛϯά͍͠ • SQSͷΩϡʔͷαΠζΛࢹ • CloudwatchϝτϦΫε͕࠷1͝ͱ • εέʔϧ·Ͱɺ1ͨͳ͍ͱߦ͚ͳ͍
ϗετεέʔϧΞτͷλΠϛϯά͍͠ • λεΫ͕͏ՃͰ্ཱ͕ͪΔϦιʔε͕ ͳ͘ͳͬͨΒεέʔϧ • ىಈͪͷλεΫ͕ग़དྷͨλΠϛϯάͰϗ ετεέʔϧͯ͠΄͍͠ɻɻ
λεΫεέʔϧΠϯ͍͠ • ECSλεΫΛ҆શʹऴྃͤ͞Δඞཁ͕͋Δ • సૹδϣϒ5hʹٴͿ߹͋Δ • TERMγάφϧͷτϥοϓͯ͠graceful shutdownΛ ࣮ •
ͨͩ͠ɺECSͷίϯιʔϧ্ͷ࣮ߦதλεΫҰཡ͔ΒɺTERMΛૹͬͯ͠Β͘͢Δ ͱফ͑ͯ͠·͏
ϗετͷεέʔϧΠϯ͍͠ • Auto Scaling Group ͷ Life cycle hook ΛͬͯɺΠϯ
ελϯεఀࢭલʹDRAININGΛ࣮ࢪ • ىಈதͷλεΫ͕ͳΔ·Ͱࢹ͠ɺͳ͘ͳͬͨΒΠϯε λϯεఀࢭΛ͢͢ΊΔ • લड़ͷ௨ΓɺىಈதͷλεΫҰཡ͔Βফ͑ΔͨΊ͝ʹΐ ͝ʹΐͬͯͨ
Ϧιʔεׂ • సૹߴԽΦϓγϣϯͱͯ͠ɺCPU/ϝϞϦ্ݶΛઃఆ Ͱ͖ΔΑ͏ʹ͍ͨ͠ • ECSͰλεΫ͝ͱʹϦιʔε্ݶΛઃఆͰ͖Δ͕ɺى ಈ࣌ͷΈ • Ϧιʔε্ݶ͝ͱʹαʔϏεΛ͚Δඞཁ͕͋Γ
͍͠
ͦ͜Ͱɺk8s
k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ
k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ
k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ statusͷભҠ "Pending" -> "Running" -> "Succeeded"
k8sͷpodεέδϡʔϥʔ • nodeϦιʔεͷۭ͖ঢ়گʹԠͯ͡ɺεέδϡʔϥʔ͕ॱ࣍podΛ nodeʹׂΓͯͯ͘ΕΔ • podΛ࡞͓͚ͯ͠ɺ͋ͱ࣮ߦ͞ΕΔͷΛ͚ͭͩ ,VCFSOFUFT QPE /0%& QPE
QPE pod্ཱͪ͛ ࣮ߦͪpod create pod
ͱΓ͋͑ͣࡶʹpodΛcreate͢Ε ɺ͋ͱk8s͕࣮ߦ·Ͱ໘ݟ ͯ͘ΕΔ
ϊʔυͷεέʔϧΞτ/Πϯ • EKSͷ߹ɺcluster autoscalerΛ͏ͷ͕σϑΝΫτʢͱ ࢥΘΕΔʣ https://github.com/kubernetes/autoscaler/tree/master/ cluster-autoscaler/cloudprovider/aws • podεέδϡʔϦϯάͷঢ়گʹԠͯ͡ɺEC2ϊʔυͷASG ܦ༝ͰεέʔϧΠϯɾΞτΛͬͯ͘ΕΔ
ϊʔυͷεέʔϧΞτ QPE &$/0%& QPE QPE 1FOEJOH ࠓͷnodeϦιʔεͰɺ podΛ্ཱͪ͛Δ͜ͱ͕ग़དྷͳ͍
ϊʔυͷεέʔϧΞτ QPE &$/0%& QPE &$/0%& QPE ClusterAutoscaler͕ASGܦ༝ͰnodeΛεέʔϧΞτ
ϊʔυεέʔϧΠϯ QPE &$/0%& QPE &$/0%& QPE
ϊʔυεέʔϧΠϯ QPE &$/0%& QPE &$/0%& Ϧιʔεʹ༨༟͕͋Δ
ϊʔυεέʔϧΠϯ QPE &$4/0%& QPE εέʔϧΠϯ
1. create pod ࣮ͯ͠ߦΛͭ 2. ඞཁʹԠͯ͡ɺnode͕εέʔϧΞ τ͢ΔͷΛͭ εέʔϥϒϧδϣϒج൫ര
k8sʹΑΔδϣϒج൫
ݱࡏͷΞʔΩςΫνϟ SQS εέδϡʔϥʔ 8PSLFS 1PE +PC %JTQBUDIFS Kubernetes δϣϒ create
pod schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out
Job Dispatcher SQS εέδϡʔϥʔ 8PSLFS 1PE Kubernetes δϣϒ create pod
schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out +PC %JTQBUDIFS
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ • SQS Λ polling • k8s API
ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ LT"1*ʹϦΫΤετ • SQS Λ polling • k8s
API ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ EPDLFSΠϝʔ δͷλάࢦఆ • SQS Λ polling •
k8s API ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher ˞ίʔυΠϝʔδͰ͢ QPEͰ࣮ߦ͢Δ ίϚϯυΛࢦఆ • SQS Λ polling •
k8s API ʹϦΫΤετΛ͛ɺ pod Λ࡞
Job Dispatcher • SQS Λ polling • k8s API ʹϦΫΤετΛ͛ɺ
pod Λ࡞ ˞ίʔυΠϝʔδͰ͢ $16ϝϞϦࢦఆ
Worker pod SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod
schuedule pod &,4/0%& $MVTUFS "VUPTDBMFS Scale in/out 8PSLFS 1PE 8PSLFS 1PE
Worker pod • ࣮ࡍͷసૹΛ࣮ߦ • సૹ։࢝Ͱ্ཱ͕ͪΓɺసૹ͕ऴྃͨ͠Βpodऴྃ ˞ίʔυΠϝʔδͰ͢
Cluster autoscaler SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod
schuedule pod &,4/0%& Scale in/out 8PSLFS 1PE 8PSLFS 1PE $MVTUFS "VUPTDBMFS
Cluster autoscaler ˞ίʔυΠϝʔδͰ͢
k8sδϣϒج൫ಋೖͷՌ • ѹతίετݮʹޭ • εέʔϧͷλΠϛϯά͕͍͕͍͍͠ ײ͡ʹղܾͰ͖ͨ • cluster autoscaler༷༷ •
։ൃڥͱຊ൪ڥͷಉҰԽ
ࢹʹ͍ͭͯ • create podͰ͖ͳ͍ • pod͕pendingͷ··্ཱ͕ͪΒͳ͍ • pod͕runningதʹಥવࢮ ຊ൪ӡ༻࣮͕ແ͍ͷͰɺຊʹظ௨Γʹ ಈ͔͘ͱ͍͏ҙຯͰࢹΛ࣮ࢪ
troccoͷδϣϒεςʔλεભҠ • DB্ʹδϣϒ͝ͱͷεςʔλεΛอ࣋ • dispatcherpodͳͲ͕ͦΕͧΕߋ৽͢Δ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE SQSʹΩϡʔΠϯά
create pod ྃ pod্ཱ͕ͪΓ·Γ pod͕࣮ߦத సૹྃ
redashͰͷࢹ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE ֤εςʔλεͰཹ͍ͯͨ͠ΒΞϥʔτ
·ͱΊ • γϯϓϧ • k8sͷݞʹΕɺεέʔϥϒϧͳδϣϒج൫ߏஙγϯϓ ϧʹͳΔ • ࢹ • ӡ༻࣮͕ແ͍ͷͰɺͱΓ͋͑ͣऔΕΔσʔλͱ͓ͬͯ͘
• k8sͷҠߦΛৼΓฦͬͯ • όον͔Βk8sಋೖ͢ΔͷׂͱΓ͍͢ͷͰ͓͢͢Ί • কདྷతʹ • কདྷతʹɺϗετͷཧͱ͔Βͳͯ͘ྑ͘ͳΔͱخ͍͠
͝ਗ਼ௌ ͋Γ͕ͱ͏͍͟͝·ͨ͠ :bow Α͚Εձࣾհޚཡ͍ͩ͘͞
גࣜձࣾ primeNumber • 201511݄ ઃཱ • ࣾһ9໊ʢ͏ͪΤϯδχΞ8໊ʣ ύʔτφʔΤϯδχΞ30໊Ҏ্ • 201811݄
trocco®ϦϦʔε • 20197݄ γϦʔζAࢿۚௐୡʢ1.3ԯԁʣ࣮ࢪ https://primenumber.co.jp
ಇ͘ڥ • @தࠇͷԊ͍ • ίΞλΠϜ੍ • 11ʙ15࣌ • िதɺ2·ͰϦϞʔτՄ ※ࢼ༻த
• δϜશֹิॿ
We are hiring • ιϑτΣΞΤϯδχΞ • ΠϯϑϥΤϯδχΞ • ͳͲͳͲɺଟํ໘ઈࢍืूத