Upgrade to Pro — share decks privately, control downloads, hide ads and more …

k8s - Kubernetes 8 Factors

k8s - Kubernetes 8 Factors

https://www.youtube.com/watch?v=OB2IbY9QJzA
アプリケーション開発においては Twelve-Factor App のようなスケーラブルな開発を助ける方法論が多数存在しています。しかしながら、Kubernetes などのインフラレイヤーにはよく知られた方法論や原則がないという印象を持っています。この知識の欠落のため、Wantedly では3年間の Kubernetes クラスタの運用の中で、次々に運用上の問題に直面してきました。そのなかで逐次的な問題解決に限界を感じ、そもそも問題の起きにくいクラスタの性質を検討しました。この結果としてクラスタが持つべき性質はポータビリティであり、これを保つためにクラスタアドミンが守るべき8つの原則があるという結論に至りました。

Avatar for Shimpei Otsubo

Shimpei Otsubo

July 22, 2019
Tweet

More Decks by Shimpei Otsubo

Other Decks in Technology

Transcript

  1. ©2019 Wantedly, Inc. Shimpei Otsubo @potsbo Visit Recommendation Squad, Wantedly,

    Inc. 2018 New Grads Kubernetes Dev Tools CI / CD AuthN / Z Productivity Dvorak
  2. ©2019 Wantedly, Inc. લఏͱͳΔ Wantedly ͷ؀ڥ Mainly AWS 40 Engineers

    6 Infrastructure Engineers 5 Kubernetes Clusters 300 namespaces in total Partially GCP No On-Premises
  3. ©2019 Wantedly, Inc. ϙʔλϏϦςΟͱ͸ લఏͱͳΔ Wantedly ͷ؀ڥ αʔόʔαΠυΤϯδχΞͱૄ݁߹ 8ݪଇ ࣮ࡍͷҠߦ

    ݪଇΛ࡞Δʹࢸͬͨഎܠ Ϋϥελ࠶࡞੒ͷඞཁʹഭΒΕΔ ΫϥελΞυϛϯ͕ΫϥελΛஔ͘ΠϯϑϥΛ؆୯ʹม͑ΒΕΔঢ়ଶ
  4. ©2019 Wantedly, Inc. … … … Ϋϥελͷ࢓༷ cluster-specs αʔϏε༻Ϧιʔε service-resources

    ʮϙʔλϏϦςΟΛ΋ͭʯͱ͸ ΫϥελΞυϛϯ͕ΫϥελΛஔ͘ΠϯϑϥΛ؆୯ʹม͑ΒΕΔঢ়ଶ
  5. ©2019 Wantedly, Inc. … … … Ϋϥελͷ࢓༷ cluster-specs αʔϏε༻Ϧιʔε service-resources

    ίίʹڱ͕ؒ͋Δ ʮϙʔλϏϦςΟΛ΋ͭʯͱ͸ ΫϥελΞυϛϯ͕ΫϥελΛஔ͘ΠϯϑϥΛ؆୯ʹม͑ΒΕΔঢ়ଶ
  6. ©2019 Wantedly, Inc. … … … Ϋϥελͷ࢓༷ Ξυϛϯ༻Ϧιʔε αʔϏε༻Ϧιʔε cluster-addons

    cluster-specs service-resources ʮϙʔλϏϦςΟΛ΋ͭʯͱ͸ ΫϥελΞυϛϯ͕ΫϥελΛஔ͘ΠϯϑϥΛ؆୯ʹม͑ΒΕΔঢ়ଶ
  7. ©2019 Wantedly, Inc. … … … Ϋϥελͷ࢓༷ Ξυϛϯ༻Ϧιʔε αʔϏε༻Ϧιʔε cluster-addons

    cluster-specs service-resources ʮϙʔλϏϦςΟΛ΋ͭʯͱ͸ ΫϥελΞυϛϯ͕ΫϥελΛஔ͘ΠϯϑϥΛ؆୯ʹม͑ΒΕΔঢ়ଶ
  8. ©2019 Wantedly, Inc. … … … Ϋϥελͷ࢓༷ Ξυϛϯ༻Ϧιʔε αʔϏε༻Ϧιʔε cluster-addons

    cluster-specs service-resources ʮϙʔλϏϦςΟΛ΋ͭʯͱ͸ ΫϥελΞυϛϯ͕ΫϥελΛஔ͘ΠϯϑϥΛ؆୯ʹม͑ΒΕΔঢ়ଶ ͜ͷ෦෼ʹϑΥʔΧε
  9. ©2019 Wantedly, Inc. എܠ Wantedly ʹ͓͚Δ Kubernetes App Engineer ͱͷૄ݁߹Λ໨ࢦ͍ͯ͠Δ

    ΋ͪΖΜݱ࣌఺Ͱ͸ίϛϡχέʔγϣϯ͸ॏཁ Ͱ΋ࣾ಺ Heroku ʹͳΓ͍ͨͷͰ஌Βͳͯ͘΋ྑ͍ͱ͜Ζ͸໨ࢦ͍ͨ͠ ͲΜͳίϯςφΛಈ͔͍ͯ͠Δͷ͔ΠϯϑϥνʔϜ͕஌Βͳͯ͘ྑ͍ঢ়ଶ αʔϏε༻Ϧιʔε͸είʔϓ֎ ΋ͪΖΜΠϯϑϥνʔϜͰਪ঑/ېࢭͳͲͷضৼΓ͸͢Δ Ͱ΋Ϋϥελ؅ཧʹ͓͍ͯ͸ࣗ༝ͱͯ͠࢒͢෦෼ αʔϏε༻ʹϦιʔε؅ཧ͸ΞϓϦέʔγϣϯΤϯδχΞʹ೚ͤΔ
  10. ©2019 Wantedly, Inc. implement configure ask / help This is

    NOT how we do App Engineers Products infrastructure Infra Engineers
  11. ©2019 Wantedly, Inc. App Engineers Infra Engineers implement Access through

    tools and APIs Products infrastructure This is how we do Provide tools automate
  12. ©2019 Wantedly, Inc. എܠ 2018೥ʹͳΓ etcd ͷ version Λ্͛Δඞཁੑ͕ग़͖ͯͨ 2015೥͔Β

    ͏ͪΠϯϑϥ͸6ਓ Wantedly Ͱىͬͨ͜͜ͱ k8s Λॳظ͔Βݕূ/ӡ༻͖ͯͨ͠ v1.1ͩͬͨ ݱࡏ͸ 40ਓͷΤϯδχΞͰ߹ܭ 300 ͷ namespace Λ؅ཧ͍ͯ͠Δ ͜ͷͨΊॳظͷ component ͕࢒ͬͯ͠·͍ͬͯΔ in-place ͷ upgrade ͸ϦεΫ͕େ͖͍ ࠶࡞੒ͩʂ ʮ৽͍͠ͷ࡞͔ͬͨΒͦΕͧΕҠߦ͠ͱ͍ͯʯ͸ඇݱ࣮త
  13. ©2019 Wantedly, Inc. https://12factor.net/ja/ https://content.pivotal.io/ebooks/beyond-the-12-factor-app THE TWELVE FACTORS CloudNative ࣌୅ͷվྑ൛

    Application αΠυͰ͸ʮϙʔλϏϦςΟΛอͭݪଇ͕஌ΒΕ͍ͯΔʯ ͔͠͠Πϯϑϥ͕࣋ͭ΂͖ੑ࣭͸Α͘·ͱΊΒΕͯ͸͍ͳ͍ ࡞ͬͯ͠·͓͏ʂ
  14. ©2019 Wantedly, Inc. ίʔυϕʔε ґଘએݴ ίʔυϕʔε಺֎ʹอ؅ͯ͠஫ೖ͢Δ ୯Ұํ޲Ϗϧυ ؀ڥҰக ඇ໊લ؅ཧ όοΫΞοϓ

    ઃఆ͸͢΂ͯίʔυԽ͢Δ ґଘ͢ΔίϯϙʔωϯτͷWFSTJPOએݴ͢Δ ઃఆ෼཭ ίʔυϕʔε͔ΒΫϥελ΁ͷҰํ޲ͷΈͷ൓ө ࠩҟ͸ઃఆʹཹΊΔ ϥϕϧΤΠϦΞεΛ࢖͏ ϦιʔεఆٛΛ෮ݩՄೳͳঢ়ଶͰอ؅͢Δ 12-factor app ͱڞ௨ New!! ॏෳܽଛ؅ཧ ॏෳͱܽଛͷ͍ͣΕ͔ͷΈΛकΔ
  15. ©2019 Wantedly, Inc. ίʔυϕʔε ࠓͷΫϥελͷઃఆ͸repositoryΛݟΕ͹ྑ͍ঢ়ଶʹ͢Δ ઃఆ͸͢΂ͯίʔυԽ͢Δ ܗࣜ͸ԿͰ΋ྑ͍ terraform / kops

    5IF5XFMWF'BDUPS"QQͰ͸ʮ*ίʔυϕʔεʯʹ૬౰ ࠓΫϥελΛࣦͬͯ΋ಉ͡ΫϥελΛ͙͢ʹͭ͘ΕΔ͔ʁ कΓ͍ͨ͜ͱ ؅ཧ͢Δ΋ͷ ؅ཧ͠ͳ͍΋ͷ αʔϏε༻ͷ …
  16. ©2019 Wantedly, Inc. શ͘ಉ͡ΫϥελΛ࡞ΔͨΊʹ͸ґଘΛ໌֬ʹએݴ͢Δ͜ͱ͕ॏཁ ར༻͢ΔΦʔέετϨʔγϣϯπʔϧͰՄೳͳݶΓWFSTJPOΛݻఆ͢Δ - etcdMembers: - instanceGroup: master-ap-northeast-1c

    name: c - instanceGroup: master-ap-northeast-1b name: b - instanceGroup: master-ap-northeast-1c-2 name: c2 name: main version: 3.0.17 ґଘએݴ ґଘ͢ΔίϯϙʔωϯτͷWFSTJPOએݴ͢Δ 5IF5XFMWF'BDUPS"QQͰ͸ʮ**ґଘؔ܎ʯʹ૬౰ ؅ཧ͢Δ΋ͷ ؅ཧ͠ͳ͍΋ͷ αʔϏε༻ͷ … ࠓΫϥελΛࣦͬͯ΋ಉ͡ΫϥελΛ͙͢ʹͭ͘ΕΔ͔ʁ कΓ͍ͨ͜ͱ
  17. ©2019 Wantedly, Inc. Code Config Deploy + = THE TWELVE

    FACTOR APP Code Config Cluster + = Kubernetes 8 factors ઃఆ෼཭ ίʔυϕʔε಺֎ʹอ؅ͯ͠஫ೖ͢Δ 5IF5XFMWF'BDUPS"QQͰ͸ʮ***ઃఆʯʹ૬౰ ؀ڥࠩҟΛ໌ࣔతʹએݴ͢Δ ྫ͑͹ઃఆϑΝΠϧͷHFOFSBUPSΛ࡞Δ wantedly Ͱ͸࣮͸ఘΊ͍ͯΔ $POGJH͕ߴʑ਺ඦߦ 3FWJFXͰಉҰมߋͰ͋Δ͜ͱΛอূ ಉҰεϖοΫͷݕূ؀ڥΛ؆୯ʹݐͯΒΕΔ͔ʁ कΓ͍ͨ͜ͱ কདྷతʹਐΈ͍ͨಓ
  18. ©2019 Wantedly, Inc. શͯͷมߋʹSFWJFXΛཁٻ͢Δ Ϋϥελͱઃఆʹࠩ෼͕ͳ͍͔ʁ Ϋϥελͷঢ়ଶ͔ΒίʔυϕʔεʹࣗಈͰ13Λ࡞੒͢ΔΞϓϩʔν΋͋Δ ٞ࿦͕෼͔ΕΔͱ͜Ζ कΓ͍ͨ͜ͱ ࣗಈEFQMPZͩͱߋʹྑ͍ Code

    Cluster ୯Ұํ޲Ϗϧυ 5IF5XFMWF'BDUPS"QQͰ͸ʮ7ϏϧυɺϦϦʔεɺ࣮ߦʯʹ૬౰ ίʔυϕʔε͔ΒΫϥελ΁ͷҰํ޲ͷΈͷ൓ө $MVTUFSΛJNNVUBCMFʹ͢Δ ίʔυϕʔεͷมߋͳ͠ʹΫϥελΛม͑ͳ͍ XBOUFEMZͰ͸·ͩख࡞ۀ github.com/fluxcd/flux
  19. ©2019 Wantedly, Inc. 8BOUFEMZͰ͸ԼͷΑ͏ͳΫϥελ໊ʹ͍ͯͨͨ͠Ίେมͩͬͨ ඇ໊લ؅ཧ ϥϕϧΤΠϦΞεΛ࢖͏ prod.example.com api.prod.example.com dev.example.com api.dev.example.com

    stage.example.com api.stage.example.com ࠓޙ΋มΘΒͳͦ͞͏ͳงғؾ͕͋Δ Τϥʔݩ͕QSPEFYBNQMFDPNͷ৔߹͸QBHFSEVUZ ໊લʹґଘͨ͠ϩδοΫΛ࡞ͬͯ͠·ͬͨ Τϥʔݩ͕TUBHFFYBNQMFDPNͷ৔߹͸໷ؒ͸ແࢹ ΦʔέετϨʔγϣϯπʔϧ͕%/4Ϩίʔυ·Ͱ࡞੒ͯ͠͠·͏ Ҡߦ࣌ʹਖ਼͍͠BQJαʔόʔ͕ߦํෆ໌ʹͳΔ ڞ௨ͷূ໌ॻΛ༻͍ͳ͍ͱҠߦ࣌ʹπʔϧ͕յΕͯ͠·͏ Ϋϥελ໊ BQJαʔόʔͷ%/4Ϩίʔυ
  20. ©2019 Wantedly, Inc. ґଘͰ͖ͳͦ͞͏ͳ໊લϨίʔυʹมߋ ඇ໊લ؅ཧ ϥϕϧΤΠϦΞεΛ࢖͏ prod201907.example.com prod201907.api.example.com dev201907.example.com dev201907.api.example.com

    stage201907.example.com stage201907.api.example.com ௚઀ͷࢀরΛආ͚Δ ϩδοΫͷ۠ผ͸ϥϕϧΛ༻͍Δ ໊લʹҙຯΛ࣋ͨͤͳ͍ FOWQSPEVDUJPOTUBHFEFW *.api.example.com ͷূ໌ॻΛڞ௨ͯ͠ར༻Ͱ͖Δ ґଘͰ͖ͳͦ͞͏ͳΞυϨεΛ࠾༻ FOWBQJFYBNQMFDPN͔Β$/".&ΛுΔ͜ͱͰґଘπʔϧͷมߋΛෆཁʹͰ͖Δ Ϋϥελ໊ BQJαʔόʔͷ%/4Ϩίʔυ
  21. ©2019 Wantedly, Inc. ґଘͰ͖ͳͦ͞͏ͳ໊લϨίʔυʹมߋ ඇ໊લ؅ཧ ϥϕϧΤΠϦΞεΛ࢖͏ prod201907.example.com prod201907.api.example.com dev201907.example.com dev201907.api.example.com

    stage201907.example.com stage201907.api.example.com ௚઀ͷࢀরΛආ͚Δ ΫογϣϯͱͳΔϨΠϠʔΛඞͣڬΉ Ωʔ͸௚઀ࢀর͠ͳ͍͜ͱ (BUFXBZͱ͍͏ղܾࡦ΋͋Δ *.api.example.com ͷূ໌ॻΛڞ௨ͯ͠ར༻Ͱ͖Δ ґଘͰ͖ͳͦ͞͏ͳΞυϨεΛ࠾༻ {env}.api.example.com ͔Β CNAME ΛுΔ͜ͱͰґଘπʔϧͷมߋΛෆཁʹͰ͖Δ Ϋϥελ໊ BQJαʔόʔͷ%/4Ϩίʔυ ͜ͷߟ͑͸LT௨ͯ͠ڞ௨Ͱ͋Δͱߟ͍͑ͯΔ
  22. ©2019 Wantedly, Inc. όοΫΞοϓ ϦιʔεఆٛΛ෮ݩՄೳͳঢ়ଶͰอ؅͢Δ ͦΕͧΕʹҠߦͯ͠΋Β͏ϑϩʔΛऔΕͳ͍͜ͱ͕͋Δ ํ๏͸໰Θͳ͍ Ұׅͯ͠Ϛωδϝϯτͯ͋͛͠Δ͜ͱΛ೦಄ʹஔ͘ ΞϓϦέʔγϣϯΤϯδχΞͷෛ୲ͳ͠ʹҠߦ …

    … … ίί 8BOUFEMZͰ͸IFQUJPWFMFSPΛ࠾༻ EFQMPZͷͨͼʹZBNMΛߋ৽BQQMZ͢Δϑϩʔ΋औΕΔ 8BOUFEMZͰ͸ࢿ࢈తʹZBNMͱຊ൪ঢ়ଶʹࠩ෼͕ग़Δ͜ͱΛڐ༰ THE TWELVE FACTOR APP Ͱ͸ѻΘͳ͍ϨΠϠ ҰൠͷΞϓϦέʔγϣϯͰ͸%#ͷόοΫΞοϓʹ૬౰͢Δ Ϋϥελ؅ཧͰ͸ආ͚ΒΕͳ͍෦෼
  23. ©2019 Wantedly, Inc. Ϋϥελ಺ͷεςʔτϑϧͳ΋ͷ͕ଘࡏ͢ΔͱҠߦ͕ඇৗʹେม SFBESFQMJDBΛ৽نΫϥελͰཱͯͯঢ֨ εςʔτϑϧͱ͸ʁ +PC΍$SPO+PC΋͋Δҙຯεςʔτϑϧ 4UBUFGVMTFU͸΋ͪΖΜεςʔτϑϧ ࣮ߦ͔ͨ͠ʁͷTUBUF͕ඇৗʹॏཁͰ͋ΔͨΊ ॏෳܽଛ؅ཧ

    ॏෳͱܽଛͷ͍ͣΕ͔ͷΈΛकΔ 8BOUFEMZͰ͸εςʔτϑϧηοτͷຊ൪ར༻Λجຊతʹېࢭ͍ͯ͠Δ ࠷ѱσʔλ͕ඈͿ͜ͱΛڐ༰Ͱ͖Δ৔߹͸PL ॏෳͱܽଛͷ͍ͣΕ͔ͷΈΛकΔ Ҡߦ࣌ʹEFMFUFͱDSFBUFΛͲͷॱͰߦ͏͔ʁ ͍ͣΕͷॱͰ΋QPEͷॏෳܽଛͷ͍ͣΕ͔͸ൃੜ͠͏Δ
  24. ©2019 Wantedly, Inc. ॏෳܽଛ؅ཧ ॏෳͱܽଛͷ͍ͣΕ͔ͷΈΛकΔ at-most-one at-least-one ࠷େ1͔ͭ͠ଘࡏͯ͠͸ߦ͚ͳ͍ resource ࠷௿1ͭҎ্͸ଘࡏ͠ͳ͍ͱ͍͚ͳ͍

    resource Muti-Cluster Strategy Λఆٛ ͜ͷ strategy ͝ͱʹҠߦ࣌ͷѻ͍Λม͑ΔΑ͏ʹͨ͠ ෳ਺Ϋϥελ͕ଘࡏ͢Δͱ͖ʹ Value ྫ Deployment(worker), CronJob, StatefulSet Deployment (web), ConfigMap, Service at-most-one delete and create at-least-one create and delete ΑΓཧ૝ͷੈք શͯͷ resource ͰॏෳΛڐ༰͢Δ શͯͷ Job Ͱႈ౳ੑΛ୲อ͢Δ Wantedly Ͱ͸ະ࣮ݱ Kubernetes Ͱ State Λ࣋ͬͨ resource Λ্ख͘ѻ͏ख๏΋͋Δ ͨͩʮ୭͕ͲΜͳίϯςφΛཱͯͯ΋͍͍ʯΛอͭʹ͸͋Δఔ౓ͷׂ੾΋ඞཁ
  25. ©2019 Wantedly, Inc. ॏෳഉআ ΫϥελͷҠߦ࣌ʹ ॏෳͱܽଛͷ͍ͣΕ͔ͷΈΛकΔ SFBESFQMJDBΛ৽نΫϥελͰཱͯͯঢ֨ εςʔτϑϧͱ͸ʁ +PC΍$SPO+PC΋͋Δҙຯεςʔτϑϧ 4UBUFGVMTFU͸΋ͪΖΜεςʔτϑϧ

    ෳ਺ΫϥελͰಉ࣌ʹଘࡏͯ͠Α͍͔ʁ͕Ұͭͷ൑அج४ ࣮ߦ͔ͨ͠ʁͷTUBUF͕ඇৗʹॏཁͰ͋ΔͨΊ BUNPTUPOF BUMFBTUPOF ෳ਺Ϋϥελ͕ଘࡏͯ͠΋͔ͭ͠ଘࡏͯ͠͸ߦ͚ͳ͍SFTPVSDF ෳ਺Ϋϥελ͕ଘࡏͯ͠΋ͭҎ্͸ଘࡏ͠ͳ͍ͱ͍͚ͳ͍SFTPVSDF .VUJ$MVTUFS4USBUFHZΛఆٛ ͜ͷTUSBUFHZ͝ͱʹҠߦ࣌ͷѻ͍Λม͑ΔΑ͏ʹͨ͠
  26. ©2019 Wantedly, Inc. ίʔυϕʔε ґଘએݴ ίʔυϕʔε಺֎ʹอ؅ͯ͠஫ೖ͢Δ ୯Ұํ޲Ϗϧυ ؀ڥҰக ඇ໊લ؅ཧ όοΫΞοϓ

    ઃఆ͸͢΂ͯίʔυԽ͢Δ ґଘ͢ΔίϯϙʔωϯτͷWFSTJPOએݴ͢Δ ઃఆ෼཭ ίʔυϕʔε͔ΒΫϥελ΁ͷҰํ޲ͷΈͷ൓ө ࠩҟ͸ઃఆʹཹΊΔ ϥϕϧΤΠϦΞεΛ࢖͏ ϦιʔεఆٛΛ෮ݩՄೳͳঢ়ଶͰอ؅͢Δ 12-factor app ͱڞ௨ New!! ॏෳܽଛ؅ཧ ॏෳͱܽଛͷ͍ͣΕ͔ͷΈΛकΔ
  27. ©2019 Wantedly, Inc. heptio/ark(౰࣌) Ͱ backup ࣄલ ಉҰઃఆͰΫϥελΛ্ཱͪ͛Δ at-least-one resource

    ͷΈΛ ৽Ϋϥελʹ restore at-most-one resource ΛݱΫϥελ͔Β delete at-most-one resource Λ৽Ϋϥελʹ restore DNS Λ੾Γସ͑ ౰೔ ؂ࢹΛશͯϥϕϧ؅ཧʹม͑Δ ֤ॴʹΤΠϦΞεΛࠩ͠ࠐΉ ༨ஊ ࣮ࡍͷҠߦϑϩʔ Ops πʔϧ༻ͷূ໌ॻࠩ͠ସ͑ 1ϲ݄ఔ౓ 1೔
  28. ©2019 Wantedly, Inc. ίʔυϕʔε ґଘએݴ ίʔυϕʔε಺֎ʹอ؅ͯ͠஫ೖ͢Δ ୯Ұํ޲Ϗϧυ ؀ڥҰக ඇ໊લ؅ཧ όοΫΞοϓ

    ઃఆ͸͢΂ͯίʔυԽ͢Δ ґଘ͢ΔίϯϙʔωϯτͷWFSTJPOએݴ͢Δ ઃఆ෼཭ ίʔυϕʔε͔ΒΫϥελ΁ͷҰํ޲ͷΈͷ൓ө ࠩҟ͸ઃఆʹཹΊΔ ϥϕϧΤΠϦΞεΛ࢖͏ ϦιʔεఆٛΛ෮ݩՄೳͳঢ়ଶͰอ؅͢Δ 12-factor app ͱڞ௨ New!! ॏෳܽଛ؅ཧ ॏෳͱܽଛͷ͍ͣΕ͔ͷΈΛकΔ