Adventure around Kubernetes at Chatwork

1. ,VCFSOFUFTΛΊ͙Δ๯ݥ 
   +"846(ίϯςφࢧ෦
   
   $IBUXPSL
   43&νʔϜ
   ࡔຊ
2. © Chatwork ࣗݾ঺հ 2 ▸ ࡔຊ ྒ (͔͞΋ͱɹΓΐ͏) ▸ Chatwork SREΤϯδχΞ ▸ ϥϯχϯάΤϯδχΞ ▸ ϑϧϚϥιϯ͕Α͏΍͘3࣌ؒ17෼͙Β͍(2019೥3݄) ▸ 2019೥ͷ૸ߦڑ཭͸2700km͙Β͍ ▸ NY- LA͸4000kmͳͷͰ, ࢒೦ͳ͕ΒΞϝϦΧԣஅ͸Ͱ͖ͣ
3. ▸ ϏδωενϟοταʔϏε ▸ άϧʔϓνϟοτɺλεΫ؅ཧɺ
 ϑΝΠϧڞ༗ɺϏσΦɾԻ੠௨࿩ ▸ ಋೖاۀ 246,000 ࣾ
 (2019೥12݄ݱࡏ)
4. © Chatwork ΞδΣϯμ 4 ▸ ChatworkͷKubernetesͷྺ࢙ΛৼΓฦΓͳ͕Βݱࡏͷӡ༻ʹ͍ͭͯ ▸ ྺ࢙ ▸ ӡ༻πʔϧ ▸ ChatworkͷKubernetes CI/CD
5. © Chatwork ஫ҙ 5 Fargateͷ࿩͸͋Γ·ͤΜ
6. © Chatwork લఏ
   $IBUXPSLͷ,VCFSOFUFTར༻ܗଶ 6 ▸ Ϛϧνςφϯτ ▸ Ϋϥελࣗମ͸SRE͕࡞Δ ▸ ؅ཧܥΞϓϦέʔγϣϯ(Datadog, Fluentd ͳͲ) -> SRE ▸ αʔϏεΞϓϦέʔγϣϯ -> DevνʔϜ
7. © Chatwork $IBUXPSLͷ,VCFSOFUFTͷྺ࢙ 7 ▸ ಋೖ͸2016೥ ▸ ࠷ॳͷProductionͷόʔδϣϯ͸1.5 ▸ ϝοηʔδϯάγεςϜͷ࡮৽ϓϩδΣΫτ ▸ AWS Summit Tokyo 2017 Ͱൃද ▸ ͦͷޙԿ౓͔ΫϥελҠߦ(όʔδϣϯΞοϓ)Λͭͭ͠ɺݱࡏʹࢸΔ ▸ ࣗલKubernetes, EKSͷࠞ߹ঢ়ଶ
8. © Chatwork &,4ͷಋೖ 8 ▸ EKSར༻લ͸kube-awsͱ͍͏ͷΛར༻ͯ͠, ࣗલͰϗεςΟϯά ▸ https://github.com/kubernetes-incubator/kube-aws ▸ EKSͷొ৔ʹΑΓkube-awsͷϢʔβݮ… ▸ ࡉ͔͍ػೳʹόά͕ଟ͘, దٓPRରԠͭͭ͠΋ࣗલύονӡ༻ ▸ EKSࣗମ͸ίϯτϩʔϥʔ෦෼͚ͩͰ, ࢖͍ʹ͍͘ͳ͊…ͱϓϨϏϡʔͰ ࢥ͍͕ͬͯͨeksctlʹΑΓͦ͏͍͏ͱ͜Ζ͕·Δͬͱָʹ
9. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 9 ▸ eksctl ▸ ։ൃ͕ૣ͍ ▸ جຊతʹ͸Ϋϥελͷߏ੒͸updateͰ͖ͳ͍ ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺ৽ن࡞੒ɾچϊʔυά ϧʔϓ࡟আ ▸ ϊʔυάϧʔϓͷߏ੒Λࢼ͍ͯ͠Δͱ͖͸ͪΐͬͱखؒ
10. ΦϑΟγϟϧͳͷͰɺEKS৽͍͠ػೳ͸େମAWSͷൃදͱಉ࣌ʹରԠ͞Ε͍ͯΔ
11. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 11 ▸ kube-aws ▸ Ϋϥελͷߏ੒͸updateՄೳ ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺچϊʔυάϧʔϓ͔Β ϩʔϦϯάΞοϓσʔτ(CloudFormationͷupdateͳͷͰ...) ▸ EKSΛར༻͢Δ΋ͷͰ͸ͳ͘, controller, etcdؚΊͯ࡞੒ ▸ ੜKubernetesͳͷͰɺΧελϚΠζੑ͸͔ͳΓߴ͍ ▸ KubernetesͷઃఆϑΝΠϧ͕௕େͳyamlʹͳΔ(Productionͷ686ߦ)
12. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 12 ▸ ௕ظతʹݟΔͱ, eksctlͷ΄͏͕ϝϦοτ͸େ͖͍ ▸ Ϣʔβೝূ͕iamͰͰ͖ͨΓ, podʹiam roleΛ෇͚ΒΕͨΓ, eksͦͷ΋ͷ ͷϝϦοτ΋େ͖͍ ▸ iam role for pod͸ͱͯ΋خ͍͠ ▸ kube2iam, kiam, kube-aws-iam-controller͢΂ͯӡ༻͖ͯͨ͠ܦݧ ▸ ͜Ε͸͜ΕͰखܰ͞΋͋Δ͚ΕͲ
13. ΫϥελΛ࡞Δ͚ͩͳΒ͜ΕͰOK
 ͔͠͠ɾɾ
14. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 14 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
15. ͜ΕΛ৐Γӽ͑Δӡ༻ɾ࢓૊Έ͕ඞཁ
16. © Chatwork ,VCFSOFUFTͷӡ༻Λࢧ͑Δπʔϧͨͪ 16 ▸ eksctl (kube-aws) ▸ Ϋϥελࣗମ͸͜ΕͰ࡞੒ ▸ Variant ▸ https://github.com/mumoshu/variant ▸ helm, helmfile ▸ https://github.com/roboll/helmfile
17. © Chatwork 7BSJBOU 17 ▸ task runner ▸ eksctlͳͲͷίϚϯυΛϥοϓ͢Δ ▸ wrapper shellͰ΋͍͍͕ɺͪ͜Β͸yamlͰॻ͚ͯɺύϥϝʔλͳͲͷࢦ ఆ΋΍Γ΍͍͢ ▸ dockerϕʔεͷtask runner͕ఆٛͰ͖ɺ؀ڥґଘΛݮΒͤΔ ▸ ؀ڥͱίϚϯυΛηοτʹͰ͖Δ
18. © Chatwork FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔ 18 ▸ eksctl͚ͩͰ΋े෼ͳ৔߹΋ଟ͍ ▸ ͱΓ͋͑ͣࢼݧ؀ڥΛىಈͤ͞Δʹ͸ίϚϯυ͚ͩͰे෼ ▸ ͨͩ͠ࡉ͔͍ઃఆΛ͠Α͏ͱ͢Δͱ, ϑΝΠϧͰͷઃఆͷ΄͏ָ͕ ▸ sampleҎ֎ͷυΩϡϝϯτ͸ͳ͍ͷͰ, ஫ҙ ▸ https://github.com/weaveworks/eksctl/blob/master/pkg/apis/eksctl.io/ v1alpha5/types.go ͔Β୳͔͢͠ͳ͍ͱ͖΋͋Δ ▸ ઃఆΛ౉ͤΔVariantͰϥοϓͯ͠, eksctl༻ઃఆϑΝΠϧͷ஋Λஔ׵
19. # Ϋϥελ໊ # αϒωοτ # cluster-autoscalerͷiam roleͭ͘Δ # eksຊମ(ϊʔυάϧʔϓআ͘)ͷeksctlͷઃఆϑΝΠϧ
20. © Chatwork FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔ 20 ▸ Ϋϥελ࡞Δ͚ͩͳΒ, ୯ମར༻Ͱ΋͍͚ͳ͘͸ͳ͍ ▸ FluxͩΕ͕ೖΕΔͷ໰୊ɺ΁ͷରԠ ▸ Flux ... GitOps tool ޙड़͠·͢ ▸ GitOpsͰઃఆͷ൓ө΍ΞϓϦέʔγϣϯͷσϓϩΠΛ͢Δ ▸ Ͱ͸, GitOpsΛ࣮ݱ͢ΔͨΊͷπʔϧ͸Ұମ୭͕ೖΕΔͷ͔...! ▸ GitOps ͸Kubernetes಺෦ʹ࣮ݱ͢ΔͨΊͷΞϓϦέʔγϣϯ͕͍Δ
21. © Chatwork 21 ▸ eksctlίϚϯυͰFluxΛ͍ΕΔ͜ͱ͸Մೳ ▸ υΩϡϝϯτϕʔεͰ΋ରԠͰ͖Δ͕ɺԼهͷϑϩʔΛ·ͱΊ͍ͨ ▸ eksctlͷઃఆϑΝΠϧ࡞੒ ▸ eksctl create cluster ▸ eksctl enable repo ... (Fluxͷinstall) ▸ ͜ΕΒΛvariantͰλεΫ੍ޚ(Infrastructure as codeͷҰ؀) FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔
22. © Chatwork 22 FLTDUMΛ7BSJBOUͰXSBQͨ݁͠Ռ ͜Ε͚ͩͰEKS؀ڥOK https://en.wikipedia.org/wiki/Achlis EKS࡞੒ίϚϯυΛachlisͱ໋໊
23. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 23 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
24. ͜͜·Ͱ͸Kubernetesࣗମͷߏங ͔͜͜Β͸Kubernetes಺෦ͷߏங
25. © Chatwork 25 ▸ https://toris.io/2019/12/what-i-think-about-when-i-think-about-kubernetes- and-ecs/ ▸ (ͳΜ͔ଜ্य़थͷλΠτϧͬΆ͍...) ▸ EKS͸ϚωʔδυͰ͕͢ɺϚωʔδϝϯτίϯιʔϧͰ࡞੒͚ͨͩ͠Ͱ͸ ΄΅Կ΋Ͱ͖·ͤΜ ▸ ϊʔυάϧʔϓ(࣮ࡍʹίϯςφ͕ಈ͘؀ڥ)Λ࡞ͬͯ΋ɺ·ͩϩά΋ૹΕ ͣɺϝτϦΫε΋ݟΕͣɺͰ͢ ▸ ͳͷͰɺEKSͰ΋؅ཧܥͷΞϓϦέʔγϣϯ(ex. Datadog)ͷӡ༻͕ඞਢ &$4ͱͷҧ͍
26. © Chatwork IFMN
    IFMNpMF 26 ▸ helm ▸ The package manager for Kubernetes ▸ KubernetesʹΞϓϦέʔγϣϯΛೖΕΔࡍʹɺඞཁͳ΋ͷΛύοέʔδ Խͯ͠ɺೖΕͯ͘ΕΔπʔϧ ▸ ChatworkͰ͸΄ͱΜͲΞϓϦέʔγϣϯΛHelmΛར༻ͯ͠σϓϩΠ ▸ Chartࣗମ͸؀ڥʹґଘ͠ͳ͍ʂ ▸ σϓϩΠͷ࠶ݱੑ(ࣗલKubernetes(kube-aws) -> EKSͰ΋ͦͷ··)
27. © Chatwork IFMN
    IFMNpMF 27 ▸ helmfile ▸ helm chartΛͲ͏΍ͬͯద༻͢Δͷ͔Λએݴతʹॻ͚Δ ▸ helmͰσϓϩΠ͢ΔࡍͷΞϓϦέʔγϣϯͷઃఆ஋΍ɺhelmίϚϯυ ͷΦϓγϣϯͳͲΛҰׅͰهࡌ͠ɺhelmͰσϓϩΠͰ͖Δπʔϧ ▸ https://github.com/helm/charts/tree/master/stable ▸ ͍ΖΜͳΞϓϦέʔγϣϯͷhelm chart ▸ https://developers.freee.co.jp/entry/2019/12/03/122657
28. © Chatwork ͳͥIFMNpMF͕ඞཁͳͷ͔ 28 ▸ ΞϓϦέʔγϣϯ͝ͱʹ࢖͍͍ͨΦϓγϣϯ(΍લॲཧ)͕มΘΔ ▸ ྫ͑͹)ΞϓϦέʔγϣϯಉ࢜Ͱґଘؔ܎ͷ͋Δ΋ͷ, ͳͲ ▸ helmͷઃఆϑΝΠϧ͕؀ڥ͝ͱ(test, stg, prod)ͰมΘΔ ▸ ྫ͑͹)AWSͷΞΧ΢ϯτ͝ͱʹҧ͏, ϩάͷు͖ग़͠ઌ͕ҧ͏ ▸ go template ΋࢖͑ΔͷͰ, helmͷઃఆ஋Λಈతʹੜ੒͢Δ͜ͱ΋Մೳ ▸ ChatworkͰ͸ɺ͞ΒʹVariantͰhelmfileΛแΈࠐΈ
29. © Chatwork 29 IFMNpMF IFMN Λ7BSJBOUͰXSBQͨ݁͠Ռ ͜Ε͚ͩͰKubernetes಺෦΋OK
30. © Chatwork 30 ͍Ζ͍Ζ7BSJBOUͰXSBQͨ݁͠Ռ
31. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 31 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
32. ͜͜·Ͱ͸Kubernetesӡ༻ ͔͜͜Β͸CI/CD
33. © Chatwork ,VCFSOFUFTͷӡ༻ͰҰ൪େมͳ͜ͱ 33 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
34. © Chatwork ,VCFSOFUFT؀ڥͷ$*$% 34 ▸ KubernetesͰͷCI/CD ▸ αʔϏεͷΞϓϦέʔγϣϯ ▸ Concourse CIΛར༻ͯ͠CI/CD ▸ ͢΂ͯΛyamlܗࣜͷpipelineͰఆٛͰ͖Δπʔϧ
35. © Chatwork ,VCFSOFUFT؀ڥͷ$*$% 35 ▸ ࠔͬͨ ▸ CIͳΓϩʔΧϧ͔ΒσϓϩΠ͢Δ৔߹, ςϯϓϨʔτԽ͍ͯ͠Δͱͯ͠ ΋ɺΫϥελ͕Ͱ͖Δͨͼʹ޲͖ઌΛม͑Δඞཁ͕͋Δ ▸ ͦ΋ͦ΋αʔϏεΞϓϦͷσϓϩΠΛߦ͏Concourse CIࣗ਎ΛͲ͜Ͱӡ ༻͢Δͷ͔໰୊
36. © Chatwork (JU0QTʂ 36 https://www.weave.works/technologies/gitops/
37. © Chatwork (JU0QTʂ!$IBUXPSL 37 Argoͷ෦෼͸·ͩରԠͰ͖͍ͯͳ͍
38. © Chatwork $IBUXPSLͷ(JU0QTߏ૝ 38 ▸ Flux + Argo CDͷϋΠϒϦουߏ੒ ▸ Flux ▸ ѻ͑Δͷ͸ݪଇmanifestͷΈͰ, 1flux - 1repo - 1branch ▸ Argo CDࣗ਎ͷΠϯετʔϧ΍namespace, aws-auth(eksಠࣗ)ͳͲ manifestద༻Ͱ͋·Γมߋ͕ͳ͍΋ͷ ▸ Argo CD ▸ GUI͕͋ͬͨΓ, helmʹରԠ͍ͯͨ͠Γɺͱػೳ͕ଟ͘࢖͍΍͍͢
39. © Chatwork $IBUXPSLͷ(JU0QTߏ૝ 39 ▸ αʔϏεͷΞϓϦέʔγϣϯ͚ͩͰ͸ͳ͘ɺ؅ཧܥͷΞϓϦͷΞϓϦέʔ γϣϯ΋GitOpsͰద༻͢Δํࣜʹมߋ༧ఆ ▸ Kubernetes͕Ͱ͖ΔͱɺඞཁͳΞϓϦέʔγϣϯ͸͢΂ͯউखʹೖ͍ͬͯ Δঢ়ଶ ▸ αʔϏεͷΞϓϦέʔγϣϯʹؔͯ͠͸΋ͪΖΜ੾Γସ͑ͳͲ͕ඞཁ
40. © Chatwork (JU0QTΛΊ͙Δ؀ڥ 40
41. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 41 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
42. © Chatwork ·ͱΊ 42 ▸ Chatworkʹ͓͚ΔKubernetesͷӡ༻(ߏங) ▸ eksctl, variant, helm, helmfile ▸ GitOpsߏ૝ ▸ EKSɺ൒ֹʂ