Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Adventure around Kubernetes at Chatwork

Ryo Sakamoto
January 23, 2020

Adventure around Kubernetes at Chatwork

Ryo Sakamoto

January 23, 2020
Tweet

More Decks by Ryo Sakamoto

Other Decks in Technology

Transcript

  1. © Chatwork ࣗݾ঺հ 2 ▸ ࡔຊ ྒ (͔͞΋ͱɹΓΐ͏) ▸ Chatwork

    SREΤϯδχΞ ▸ ϥϯχϯάΤϯδχΞ ▸ ϑϧϚϥιϯ͕Α͏΍͘3࣌ؒ17෼͙Β͍(2019೥3݄) ▸ 2019೥ͷ૸ߦڑ཭͸2700km͙Β͍ ▸ NY- LA͸4000kmͳͷͰ, ࢒೦ͳ͕ΒΞϝϦΧԣஅ͸Ͱ͖ͣ
  2. © Chatwork $IBUXPSLͷ,VCFSOFUFTͷྺ࢙ 7 ▸ ಋೖ͸2016೥ ▸ ࠷ॳͷProductionͷόʔδϣϯ͸1.5 ▸ ϝοηʔδϯάγεςϜͷ࡮৽ϓϩδΣΫτ

    ▸ AWS Summit Tokyo 2017 Ͱൃද ▸ ͦͷޙԿ౓͔ΫϥελҠߦ(όʔδϣϯΞοϓ)Λͭͭ͠ɺݱࡏʹࢸΔ ▸ ࣗલKubernetes, EKSͷࠞ߹ঢ়ଶ
  3. © Chatwork &,4ͷಋೖ 8 ▸ EKSར༻લ͸kube-awsͱ͍͏ͷΛར༻ͯ͠, ࣗલͰϗεςΟϯά ▸ https://github.com/kubernetes-incubator/kube-aws ▸

    EKSͷొ৔ʹΑΓkube-awsͷϢʔβݮ… ▸ ࡉ͔͍ػೳʹόά͕ଟ͘, దٓPRରԠͭͭ͠΋ࣗલύονӡ༻ ▸ EKSࣗମ͸ίϯτϩʔϥʔ෦෼͚ͩͰ, ࢖͍ʹ͍͘ͳ͊…ͱϓϨϏϡʔͰ ࢥ͍͕ͬͯͨeksctlʹΑΓͦ͏͍͏ͱ͜Ζ͕·Δͬͱָʹ
  4. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 9 ▸ eksctl ▸ ։ൃ͕ૣ͍ ▸ جຊతʹ͸Ϋϥελͷߏ੒͸updateͰ͖ͳ͍

    ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺ৽ن࡞੒ɾچϊʔυά ϧʔϓ࡟আ ▸ ϊʔυάϧʔϓͷߏ੒Λࢼ͍ͯ͠Δͱ͖͸ͪΐͬͱखؒ
  5. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 11 ▸ kube-aws ▸ Ϋϥελͷߏ੒͸updateՄೳ ▸ ϊʔυάϧʔϓͷઃఆมߋΛ͍ͨ͠৔߹͸ɺچϊʔυάϧʔϓ͔Β

    ϩʔϦϯάΞοϓσʔτ(CloudFormationͷupdateͳͷͰ...) ▸ EKSΛར༻͢Δ΋ͷͰ͸ͳ͘, controller, etcdؚΊͯ࡞੒ ▸ ੜKubernetesͳͷͰɺΧελϚΠζੑ͸͔ͳΓߴ͍ ▸ KubernetesͷઃఆϑΝΠϧ͕௕େͳyamlʹͳΔ(Productionͷ686ߦ)
  6. © Chatwork FLTDUMͱLVCFBXTͷൺֱ 12 ▸ ௕ظతʹݟΔͱ, eksctlͷ΄͏͕ϝϦοτ͸େ͖͍ ▸ Ϣʔβೝূ͕iamͰͰ͖ͨΓ, podʹiam

    roleΛ෇͚ΒΕͨΓ, eksͦͷ΋ͷ ͷϝϦοτ΋େ͖͍ ▸ iam role for pod͸ͱͯ΋خ͍͠ ▸ kube2iam, kiam, kube-aws-iam-controller͢΂ͯӡ༻͖ͯͨ͠ܦݧ ▸ ͜Ε͸͜ΕͰखܰ͞΋͋Δ͚ΕͲ
  7. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 14 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
  8. © Chatwork ,VCFSOFUFTͷӡ༻Λࢧ͑Δπʔϧͨͪ 16 ▸ eksctl (kube-aws) ▸ Ϋϥελࣗମ͸͜ΕͰ࡞੒ ▸

    Variant ▸ https://github.com/mumoshu/variant ▸ helm, helmfile ▸ https://github.com/roboll/helmfile
  9. © Chatwork 7BSJBOU 17 ▸ task runner ▸ eksctlͳͲͷίϚϯυΛϥοϓ͢Δ ▸

    wrapper shellͰ΋͍͍͕ɺͪ͜Β͸yamlͰॻ͚ͯɺύϥϝʔλͳͲͷࢦ ఆ΋΍Γ΍͍͢ ▸ dockerϕʔεͷtask runner͕ఆٛͰ͖ɺ؀ڥґଘΛݮΒͤΔ ▸ ؀ڥͱίϚϯυΛηοτʹͰ͖Δ
  10. © Chatwork FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔ 18 ▸ eksctl͚ͩͰ΋े෼ͳ৔߹΋ଟ͍ ▸ ͱΓ͋͑ͣࢼݧ؀ڥΛىಈͤ͞Δʹ͸ίϚϯυ͚ͩͰे෼ ▸ ͨͩ͠ࡉ͔͍ઃఆΛ͠Α͏ͱ͢Δͱ,

    ϑΝΠϧͰͷઃఆͷ΄͏ָ͕ ▸ sampleҎ֎ͷυΩϡϝϯτ͸ͳ͍ͷͰ, ஫ҙ ▸ https://github.com/weaveworks/eksctl/blob/master/pkg/apis/eksctl.io/ v1alpha5/types.go ͔Β୳͔͢͠ͳ͍ͱ͖΋͋Δ ▸ ઃఆΛ౉ͤΔVariantͰϥοϓͯ͠, eksctl༻ઃఆϑΝΠϧͷ஋Λஔ׵
  11. © Chatwork FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔ 20 ▸ Ϋϥελ࡞Δ͚ͩͳΒ, ୯ମར༻Ͱ΋͍͚ͳ͘͸ͳ͍ ▸ FluxͩΕ͕ೖΕΔͷ໰୊ɺ΁ͷରԠ ▸

    Flux ... GitOps tool ޙड़͠·͢ ▸ GitOpsͰઃఆͷ൓ө΍ΞϓϦέʔγϣϯͷσϓϩΠΛ͢Δ ▸ Ͱ͸, GitOpsΛ࣮ݱ͢ΔͨΊͷπʔϧ͸Ұମ୭͕ೖΕΔͷ͔...! ▸ GitOps ͸Kubernetes಺෦ʹ࣮ݱ͢ΔͨΊͷΞϓϦέʔγϣϯ͕͍Δ
  12. © Chatwork 21 ▸ eksctlίϚϯυͰFluxΛ͍ΕΔ͜ͱ͸Մೳ ▸ υΩϡϝϯτϕʔεͰ΋ରԠͰ͖Δ͕ɺԼهͷϑϩʔΛ·ͱΊ͍ͨ ▸ eksctlͷઃఆϑΝΠϧ࡞੒ ▸

    eksctl create cluster ▸ eksctl enable repo ... (Fluxͷinstall) ▸ ͜ΕΒΛvariantͰλεΫ੍ޚ(Infrastructure as codeͷҰ؀) FLTDUM͚ͩͰ͸ͳͥͩΊͳͷ͔
  13. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 23 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
  14. © Chatwork 25 ▸ https://toris.io/2019/12/what-i-think-about-when-i-think-about-kubernetes- and-ecs/ ▸ (ͳΜ͔ଜ্य़थͷλΠτϧͬΆ͍...) ▸ EKS͸ϚωʔδυͰ͕͢ɺϚωʔδϝϯτίϯιʔϧͰ࡞੒͚ͨͩ͠Ͱ͸

    ΄΅Կ΋Ͱ͖·ͤΜ ▸ ϊʔυάϧʔϓ(࣮ࡍʹίϯςφ͕ಈ͘؀ڥ)Λ࡞ͬͯ΋ɺ·ͩϩά΋ૹΕ ͣɺϝτϦΫε΋ݟΕͣɺͰ͢ ▸ ͳͷͰɺEKSͰ΋؅ཧܥͷΞϓϦέʔγϣϯ(ex. Datadog)ͷӡ༻͕ඞਢ &$4ͱͷҧ͍
  15. © Chatwork IFMN IFMNpMF 26 ▸ helm ▸ The package

    manager for Kubernetes ▸ KubernetesʹΞϓϦέʔγϣϯΛೖΕΔࡍʹɺඞཁͳ΋ͷΛύοέʔδ Խͯ͠ɺೖΕͯ͘ΕΔπʔϧ ▸ ChatworkͰ͸΄ͱΜͲΞϓϦέʔγϣϯΛHelmΛར༻ͯ͠σϓϩΠ ▸ Chartࣗମ͸؀ڥʹґଘ͠ͳ͍ʂ ▸ σϓϩΠͷ࠶ݱੑ(ࣗલKubernetes(kube-aws) -> EKSͰ΋ͦͷ··)
  16. © Chatwork IFMN IFMNpMF 27 ▸ helmfile ▸ helm chartΛͲ͏΍ͬͯద༻͢Δͷ͔Λએݴతʹॻ͚Δ

    ▸ helmͰσϓϩΠ͢ΔࡍͷΞϓϦέʔγϣϯͷઃఆ஋΍ɺhelmίϚϯυ ͷΦϓγϣϯͳͲΛҰׅͰهࡌ͠ɺhelmͰσϓϩΠͰ͖Δπʔϧ ▸ https://github.com/helm/charts/tree/master/stable ▸ ͍ΖΜͳΞϓϦέʔγϣϯͷhelm chart ▸ https://developers.freee.co.jp/entry/2019/12/03/122657
  17. © Chatwork ͳͥIFMNpMF͕ඞཁͳͷ͔ 28 ▸ ΞϓϦέʔγϣϯ͝ͱʹ࢖͍͍ͨΦϓγϣϯ(΍લॲཧ)͕มΘΔ ▸ ྫ͑͹)ΞϓϦέʔγϣϯಉ࢜Ͱґଘؔ܎ͷ͋Δ΋ͷ, ͳͲ ▸

    helmͷઃఆϑΝΠϧ͕؀ڥ͝ͱ(test, stg, prod)ͰมΘΔ ▸ ྫ͑͹)AWSͷΞΧ΢ϯτ͝ͱʹҧ͏, ϩάͷు͖ग़͠ઌ͕ҧ͏ ▸ go template ΋࢖͑ΔͷͰ, helmͷઃఆ஋Λಈతʹੜ੒͢Δ͜ͱ΋Մೳ ▸ ChatworkͰ͸ɺ͞ΒʹVariantͰhelmfileΛแΈࠐΈ
  18. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 31 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
  19. © Chatwork ,VCFSOFUFTͷӡ༻ͰҰ൪େมͳ͜ͱ 33 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ
  20. © Chatwork ,VCFSOFUFT؀ڥͷ$*$% 34 ▸ KubernetesͰͷCI/CD ▸ αʔϏεͷΞϓϦέʔγϣϯ ▸ Concourse

    CIΛར༻ͯ͠CI/CD ▸ ͢΂ͯΛyamlܗࣜͷpipelineͰఆٛͰ͖Δπʔϧ
  21. © Chatwork $IBUXPSLͷ(JU0QTߏ૝ 38 ▸ Flux + Argo CDͷϋΠϒϦουߏ੒ ▸

    Flux ▸ ѻ͑Δͷ͸ݪଇmanifestͷΈͰ, 1flux - 1repo - 1branch ▸ Argo CDࣗ਎ͷΠϯετʔϧ΍namespace, aws-auth(eksಠࣗ)ͳͲ manifestద༻Ͱ͋·Γมߋ͕ͳ͍΋ͷ ▸ Argo CD ▸ GUI͕͋ͬͨΓ, helmʹରԠ͍ͯͨ͠Γɺͱػೳ͕ଟ͘࢖͍΍͍͢
  22. © Chatwork ,VCFSOFUFTͷӡ༻Ͱେมͳ͜ͱ 41 ▸ Kubernetes version up ͸ 3ϲ݄ʹ1ճ

    ▸ ͢΂ͯʹ௥ै͠ͳ͍ͱͯ͠΋, ൒೥ʹ1ճ͙Β͍͸version up ▸ test, stg, production؀ڥͷ࡞Γม͑ɺσϓϩΠ͕සൃʂ ▸ EKSʹ͸ϩʔϦϯάΞοϓάϨʔυ͋Δ͚ΕͲ.... ▸ खಈ(υΩϡϝϯτରԠ)Ͱ͸ӡ༻ෛՙ͕ߴ͗͢Δ