Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leader Election in Kubernetes #k8sjp

d-kuro
January 29, 2020

Leader Election in Kubernetes #k8sjp

d-kuro

January 29, 2020
Tweet

More Decks by d-kuro

Other Decks in Technology

Transcript

  1. Leader Election ͱ͸ • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩

    ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕ ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/
  2. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3
  3. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 API Server Controller Manager Deployment Controller Watch Create ReplicaSet replica: 3
  4. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update
  5. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update API Server Controller Manager Deployment Controller Watch Update ReplicaSet replica: 3 -> 2
  6. Reconciliation Loop Provided in Managing Kubernetes: Operating Kubernetes Clusters in

    the Real World, 23. https://www.oreilly.com/library/view/managing-kubernetes/9781492033905/
  7. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  8. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  9. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  10. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  11. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 Error from server (AlreadyExists): deployments.apps “foo" already exists 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  12. Leader Election in Kubernetes • Kubernetes ͷ Leader Election ͸

    ڝ߹Λආ͚ͭͭߴՄ༻ੑΛอͭͨΊͷ ΢ΥʔϜελϯόΠͷ࢓૊Έͱͯ͠࢖ΘΕ͍ͯΔ • ϥΠϒϥϦͱͯ͠ఏڙ͞Ε͍ͯΔͷͰ ࠓճઆ໌ͨ͠ Controller ͷྫҎ֎Ͱ΋ Kubernetes ্Ͱಈ͘ΞϓϦέʔγϣϯͰ Leader Election ͍ͨ͠ͱ͍͏৔߹ʹ΋࢖༻Ͱ͖Δ
  13. Leader Election ͷ࣮૷ • Leader-for-life • Operator SDK ͕ఏڙ͍ͯ͠Δ࣮૷ •

    github.com/operator-framework/operator-sdk/pkg/leader • Leader-with-lease • controller-runtime(client-go) ͕ఏڙ͍ͯ͠Δ࣮૷ • github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection
  14. Leader Election ͷ࣮૷ • Leader-for-life • NotReady Node ʹ Pod

    ͕͋ΔͱϦʔμʔͷબग़͕஗͘ͳΔ (pod-eviction-timeout, default: 5m ʹґଘ͢Δ) • Split Brain ͷϦεΫͳ͠ • Leader-with-lease • Split Brain ͷϦεΫ͕͋Δ • NotReady Node ʹ Pod ͕͋ͬͯ΋Ϧʔμʔͷ੾Γସ͕͑ૣ͍
  15. Leader-for-life • OwnerReference ͕Ϧʔμʔͷ Pod Ͱ͋Δ ConfigMap Λ࡞੒͠, ϩοΫ͢Δ •

    Pod ͕࡟আ͞ΕΔͱ Kubernetes ͷΨϕʔδίϨΫγϣϯͷ ࢓૊ΈʹΑΓ ConfigMap ΋ࣗಈతʹ࡟আ͞ΕΔͨΊ, ଞͷ Pod ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
  16. API Server Controller Pod A Watch Controller Pod B Leader-for-life

    ConfigMap Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"
  17. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"
  18. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"
  19. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE# Create ConfigMap Watch
  20. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE# Create ConfigMap Watch
  21. Leader-with-lease • ConfigMap ·ͨ͸ Endpoints Λ༻͍ͯϩοΫ͢Δ • Ϧʔεظ͕ؒઃఆ͞Ε͓ͯΓ, Ϧʔμʔ͸ఆظతʹϦʔεͷߋ৽Λߦ͏ •

    ԿΒ͔ͷཧ༝ͰϦʔε͕ظݶ੾Εʹͳͬͨࡍʹ͸ ଴ػ͍ͯͨ͠ Controller ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
  22. API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap

    Lock Watch Update lease record Check lease record
  23. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Update lease record observedTime: 2020-01-28T05:33:18Z holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  24. observedTime: 2020-01-28T05:33:18Z Check lease record API Server Controller Pod A

    Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record Controller Pod A ConfigMap Lock Update lease record Ϧʔμʔ͸ఆظతʹϦʔεΛߋ৽͢Δ (ߋ৽͢Δͱ renewTime ͕ॻ͖׵ΘΔ) holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  25. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Update lease record PCTFSWFE5JNF5; Controller Pod A ConfigMap Lock Update lease record observedTime: 2020-01-28T05:33:18Z Controller ͸಺෦Ͱ observedTime ͱ͍͏஋Λ͓࣋ͬͯΓ ൑ఆʹ࢖༻͢Δ holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  26. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  27. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch IPMEFS*EFOUJUZ1PE" MFBTF%VSBUJPO4FDPOET BDRVJSF5JNF5; SFOFX5JNF5; MFBEFS5SBOTJUJPOT Check lease record Controller Pod B ConfigMap Lock holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80 PCTFSWFE5JNF5; observedTime + leaseDurationSeconds < now ͳΒ ԿΒ͔ͷཧ༝ͰϦʔε͕ߋ৽͞Ε͍ͯͳ͍ͨΊ ࣗ෼͕ϦʔμʔʹͳΖ͏ͱߋ৽ΛࢼΈΔ
  28. Update lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  29. Update lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Check lease record holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  30. ·ͱΊ • Kubernetes Ͱ͸ Object Λ෼ࢄϩοΫʹ࢖༻͢Δ͜ͱͰ Leader Election Λ࣮ݱ͍ͯ͠Δ •

    ࡉ͔͍ίʔυͷ࿩ͱ͔͸͜͜ʹॻ͖·ͨ͠ • https://d-kuro.github.io/posts/kubernetes-leader-election/
  31. Reference • Configuring leader election - Operator SDK • https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-leader-election.html

    • େମ஻ͬͨ͜ͱ͕ॻ͍ͯ͋Δ • Gopherize.me - A Gopher pic that's as unique as you • https://gopherize.me/ • Gopher ܅͕࡞ΕΔαΠτ • community/icons at master · kubernetes/community • https://github.com/kubernetes/community/tree/master/icons • Kubernetes ͷΞΠίϯηοτ