Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
チャットワークにおけるKubernetesOnAWS.pdf
Search
Ryo Sakamoto
May 28, 2018
0
85
チャットワークにおけるKubernetesOnAWS.pdf
Ryo Sakamoto
May 28, 2018
Tweet
Share
More Decks by Ryo Sakamoto
See All by Ryo Sakamoto
いろいろなAWSアカウントのArgo CDを統合した話
cwsakamoto
1
770
ArgoCDとGitHub Self Hosted Runnerを使って リリース時間を1/4にした話
cwsakamoto
0
2k
Adventure around Kubernetes at Chatwork
cwsakamoto
6
7.6k
チャットワークにおけるKubernetesOnAWS.pdf
cwsakamoto
0
76
Kubernetes on AWS at Chatwork
cwsakamoto
0
1.6k
Featured
See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
720
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
109
49k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
6.8k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.3k
Bash Introduction
62gerente
608
210k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
Gamification - CAS2011
davidbonilla
80
5k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
8.2k
How to Ace a Technical Interview
jacobian
276
23k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
16
2.1k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Transcript
νϟοτϫʔΫʹ͓͚Δ Kubernetes on AWS SRE Ryo Sakamoto
© ChatWork ▸ ຊൃϏδωενϟοτ ▸ λεΫཧϏσΦ௨͕Մೳ ▸ ಋೖاۀ174,000ࣾҎ্ʢ※20184݄࣌ʣ
© ChatWork ΞδΣϯμ ▸ Kubernetesͷڥಈ͍͍ͯΔΞϓϦ ▸ KubernetesͰར༻͍ͯ͠Δπʔϧ ▸ KubernetesͷࢹɺϩΪϯά ▸
Kubernetesͷversion up ▸ ·ͱΊ
© ChatWork Kubernetesͷར༻ ▸ ϝοηʔδॲཧ෦ͷϦϓϨΠε(201612݄) ▸ backen appΛkubernetes Ͱಈ͔͢ ▸
ࡉ͔͍ࡢͷAWS summitͰ…
© ChatWork Kubernetesͷڥ ▸ ڥ AWS ▸ AWSͳͲطଘͷࢿݯΛར༻͔ͨͬͨͨ͠Ί ▸ ߏஙπʔϧ
kube-aws ▸ https://github.com/kubernetes-incubator/kube-aws ▸ ϝΠϯϝϯςφmumoshu (chatwork kubernetes ސ) ▸ cloudformation(ͱcloud-init)Ͱ·ΔͬͱߏஙͰ͖Δ
© ChatWork KubernetesͰಈ͍͍ͯΔͷ ▸ backend ▸ ࡢͷAWS summitͰmessage backendΛϦϓϨΠεͨ͠ ▸
͜ͷϓϩδΣΫτҎ߱(webhook, oauthͳͲ)ͯ͢kubernetes ▸ ϊʔυm4.2xlarge * 10ఔ ▸ CDڥ(concourse) ▸ spot instance ͳnodepoolΛར༻
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(1) ▸ cluster-autoscaler ▸ podͷauto scaleͰͳ͘ɺnodeͷauto scaleͯ͘͠ΕΔ ▸
schedulerΛࢹͯ͠ɺϦιʔε͕Γͳ͍pod͕͍ΔͱASGΛૢ࡞ ▸ nodeͷݮʹେ͖͘ߩݙ ▸ σϓϩΠ࣌ͷpodͷೖସ͑ͳͲͰҰ࣌తʹϊʔυ͕Γͳ͘ͳΔͱ͖ ͞ΒͬͱରԠͯ͘͠ΕΔ
© ChatWork cluster-autoscalerͷಈ͖(scale out) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (2) “fails to be scheduled due to insufficient” (4)scale out (3) set-desired-capacity ྫ
© ChatWork cluster-autoscalerͷಈ͖(scale in) controller nodepool api-server scheduler cluster- autoscaler
pod (1) watch (apiܦ༝) nodeͷ༻ (3) set-desired-capacity ྫ a b a b nodeͷ༻ a b a b (4) scale in (2) evict
© ChatWork KubernetesͰར༻͍ͯ͠Δπʔϧ(2) ▸ kube2iam ▸ podຖʹroleͷ༩ ▸ ௨ৗΠϯελϯεͷϩʔϧΛར༻ ▸
ෆཁͳpolicy͕͘ & Γͳ͍ͷAPIKEYΛͨͤΔ͜ͱʹͳΔ ▸ secretbase64ͳ͚ͩ ▸ एׯෆ҆ఆͰɺkiamʹஔ͖͑༧ఆ
© ChatWork kube2iam ▸ annotationʹroleΛهࡌ ▸ roleworkerͷroleΛ৴པ͓ͯ͘͠ ▸ worker͕asuumeͰ͖ΔΑ͏ʹ͓ͯ͘͠ ▸
pod͕ɺAWSͷAPIΛར༻͠Α͏ͱ͢ΔͱɺmetadataʹΞΫηε͢Δ ▸ metadataͷΞΫηεΛiptablesͰkube2iamʹసૹ ▸ kube2iam͕annotationͷroleͷΫϨσϯγϟϧΛൃߦ
© ChatWork kube2iam app kube2iam 1. credentialͷൃߦ(ec2-metadata) 2. iptablesͰkube2iamͷpodʹϦΫΤετ͕సૹ 3.
credentialͷൃߦ pod ྫ
© ChatWork Kubernetesͷࢹ ▸ datadog only ▸ daemonsetͰஔ ▸ not
k8sͳڥͷࢹͱ౷Ұ͍ͨ͠ & prometheusͷཧΛͨ͘͠ͳ͍ ▸ prometheusͷΑ͏ʹΤϯυϙΠϯτΛੜ͢ͷͰͳ͘ɺ֤ϗετͷ statsdʹૹ৴ ▸ version 6Λར༻ ▸ v5ͰϝτϦΫε͕͚͍ܽͯͨ(ϝτϦΫεᷓΕ)͕ɺv6Ͱ͚ܽͳ͘ͳͬͨ
© ChatWork datadogͷlive container monitoring
© ChatWork KubernetesͷϩΪϯά ▸ fluentd + stackdriver ▸ fluentdΛdaemonsetͰஔ ▸
֤ίϯςφϗετͷಛఆͷॴʹstdoutΛు͖ग़͍ͯ͠Δ ▸ audit-logfluentdͰstackdriverʹૹ৴ ▸ S3ͰΑ͔͕ͬͨɺKubernetesΛಋೖͨ࣌͠ʹAthenaग़͔ͨΓ ▸ stackdriver + bigqueryҰ෦ͷϩάͰಋೖ
© ChatWork Kubernetesͷversion up ▸ kube-awsͰཧ͍ͯ͠ΔҎ্ɺϚωʔδυͳversion upͰ͖ͳ͍ ▸ version upkubernetes
౷߹Λߦͬͨ ▸ version up 1.7 -> 1.8, 1.5 -> 1.8 && 1.8ԽͷλΠϛϯάͰΫϥελ౷߹ ▸ νϟοτϫʔΫͰ·ͩingressར༻Ͱ͖͍ͯͳͯ͘ɺELB + NodePort ▸ ͳͷͰɺversion upELBʹ৽چ྆ํΛͿΒԼ͛ͯɺݹ͍ํΛޙୀ ▸ ࠓͷΞϓϦέʔγϣϯͱͯ͠onlineͰversion upྃ
© ChatWork version up old k8s pod ྫ app chatwork
web NodePort
© ChatWork version up old k8s pod ྫ app app
chatwork web NodePort NodePort new k8s
© ChatWork version up pod ྫ app chatwork web NodePort
new k8s
© ChatWork KubernetesͰࠓޙΓ͍ͨ(1) ▸ EKS ▸ kube-awsʹΈࠐ·ΕΔ༧ఆ ▸ ͬͺΓϩʔϦϯάΞοϓσʔτ͍ͨ͠ ▸
service mesh ▸ envoyͷಋೖ ▸ istio, linkerd ▸ grpc loadbalancer -> envoy, nginx-ingress-controller
© ChatWork KubernetesͰࠓޙΓ͍ͨ(2) ▸ prometheusͷಋೖ ▸ hpaσϑΥϧτͰcpu͔͠ͳ͍ͷͰ͔ͭʹ͍͘ ▸ cpuͰͳ͘kafkaͷeventͷ٧·Γ۩߹Ͱscale in/out͍ͨ͠
▸ datadogͰapiΛͬͯͰ͖Δ͚Ͳɺdatadogͱͷଓෆ҆ ▸ ϓϥοτϑΥʔϜԽ ▸ openFaaSͳͲ
© ChatWork EKSͷظ ▸ preview൛Λར༻͍͍ͤͯͨͩͨ͞ ▸ workerͷՃ͕͕͕͕͕….configmapܦ༝Ͱొ͢Δɺͱ͍͏ํ๏ͩͬͨ ▸ ͜Εͩͱkubernetesͷ֎ͰɺΫϥελߏங͕ด͡ͳ͍ ▸
AWSͷϦιʔε׆༻(IAMɺVPC)ͳͲظ ▸ fargateͰnodeͦͷͷΛҙࣝ͠ͳ͍ͷ͍͍͕ɺloggingࢹ…
© ChatWork ·ͱΊ ▸ νϟοτϫʔΫͷKubernetesڥʹ͍ͭͯͷ ▸ ͍Ζ͍ΖΓ͍ͨ͜ͱ͋Δ ▸ EKSʹظ ▸
controll plane͕Ϛωʔδυ͞ΕΔ҆৺ײ
© ChatWork ΤϯδχΞืूத http://corp.chatwork.com/ja/recruit/ ▸ ओମੑΛ࣋ͪɺࣗΒߦಈͰ͖Δ ▸ ଞऀΛೝΊɺଚॏͰ͖Δ ▸ ใΛूΊɺڞ༗Ͱ͖Δ
ͱ͍͏ํΛܴ͠·͢ʂ