kube-system落としてみました

kube-system 意識していますか？ (私は意識してませんでした)
kube-dns が死んだりすると、サービス停止しちゃいます (経験談)
せっかくなのでkube-systemをそれぞれ落として、影響を調べてみました。 (反省を兼ねて)
今回はGKEで試しています

Transcript

1. Kubernetes Meetup Tokyo #16 LT 1 
   
   
   4IVOUBSP
   4BJCB
   kube-system
   མͱͯ͠Έ·ͨ͠
   
2. ຊ೔͓఻͍͑ͨ͜͠ͱ
   ˔  LVCFTZTUFN
   ҙ͍ࣝͯ͠·͔͢ʁ
   ࢲ͸ҙࣝͯ͠·ͤΜͰͨ͠
   ˔  LVCFEOT
   ͕ࢮΜͩΓ͢ΔͱɺαʔϏεఀࢭͪ͠Ό͍·͢
   ܦݧஊ
   ˔  ͔ͤͬ͘ͳͷͰLVCFTZTUFNΛͦΕͧΕམͱͯ͠ɺ
   ӨڹΛௐ΂ͯΈ·ͨ͠ɻ
   ൓লΛ݉Ͷͯ
   ˔  ࠓճ͸(,&Ͱࢼ͍ͯ͠·͢
   
   2
3. @tarosaiba ˔  'VUVSF
   "SDIJUFDU
   *OD
   
   ˔  5FDIOPMPHZ
   *OOPWBUJPO
   (SPVQ
   ˔  *OGSB
   
   .8
   
   $*
   
   पΓ΍͖ͬͯ·ͨ͠
   ˔  ΤϯϓϥͰ"84
   ࠷ۙ͸($1
   
   4IVOUBSP
   4BJCB
   3
4. 4 ࢖ͬͨ؀ڥ
   Microservices Kubernetes Engine Nodes ˔  (,&
   Λ
   /PEFͰཱͯ·ͨ͠ɻ EFGBVMUͷBEEPOΛ͢΂ͯ༗ޮ
   ˓  WFSTJPO
   
   (,&
   HLF
   
   
5. 5 ࢖ͬͨ؀ڥ
   ˔  ,VCFSOFUFTͷαϯϓϧΞϓϦ
   lHVFTUCPPLz
   ΛσϓϩΠ
   
   https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
6. ɹɹɹɹࠓճݟΔ΋ͷ
   6
7. 7 # Get kube-system pods > kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE event-exporter-v0.2.3-85644fcdf-bzddn 2/2 Running 0 3h fluentd-gcp-scaler-8b674f786-9tp2z 1/1 Running 0 3h fluentd-gcp-v3.2.0-6gq6r 2/2 Running 0 3h fluentd-gcp-v3.2.0-8h5hp 2/2 Running 0 3h fluentd-gcp-v3.2.0-pb44f 2/2 Running 0 2h heapster-v1.6.0-beta.1-74f4bbc55c-52khj 3/3 Running 0 3h kube-dns-7df4cb66cb-9ft98 4/4 Running 0 3h kube-dns-7df4cb66cb-wvhpz 4/4 Running 0 2m kube-dns-autoscaler-67c97c87fb-xgrx6 1/1 Running 0 3h kube-proxy-gke-demo-default-pool-0dacde9e-dvbd 1/1 Running 0 3h kube-proxy-gke-demo-default-pool-0dacde9e-gc4m 1/1 Running 0 3h kube-proxy-gke-demo-default-pool-0dacde9e-lr6s 1/1 Running 0 2h l7-default-backend-7ff48cffd7-vfmxz 1/1 Running 0 3h metrics-server-v0.2.1-fd596d746-xz8s2 2/2 Running 0 3h OBNFTQBDFʹzLVCFTZTUFNz
   Λࢦఆ
   
8. k8sΛࢧ͑Δ
   ొ৔ਓ෺
   ͓͞Β͍
   8 kube-apiserver etcd kube-controller manager kube-scheduler kubelet kube-proxy container runtime ... ... Master Node GCP Managed Our Project kubelet kube-dns
9. k8sΛࢧ͑Δ
   ొ৔ਓ෺
   ͓͞Β͍
   9 kube-apiserver etcd kube-controller manager kube-scheduler kubelet kube-proxy container runtime ... ... Master Node GCP Managed kubelet kube-dns
10. k8sΛࢧ͑Δ
    ొ৔ਓ෺
    ͓͞Β͍
    10 kube-apiserver etcd kube-controller manager kube-scheduler kubelet kube-proxy container runtime ... ... Master Node GCP Managed Our Project kubelet kube-dns
11. k8sΛࢧ͑Δ
    ొ৔ਓ෺
    ͓͞Β͍
    11 kube-apiserver etcd kube-controller manager kube-scheduler kubelet kube-proxy container runtime ... ... Master Node GCP Managed Our Project kubelet kube-dns ͪΌΜͱҙࣝͰ͖ͯͳ͔ͬͨ
    ൓ল
    
12. 12 Name ໾ׂ kube-apiserver ˔  LTͷϦιʔεͷૢ࡞Λߦ͏ͨΊͷ
    ,VCFSOFUFT
    "1*
    Λఏڙ
    etcd ˔  ෼ࢄ
    ,FZ7BMVF
    4UPSF
    ˔  LTΫϥελͷ৘ใΛอଘ
    kube-scheduler ˔  ֤/PEFͷঢ়ଶΛݟͯɺ1PEͷׂ౰Λߦ͏εέδϡʔϥ
    kube-controller-manager ˔  ֤ίϯτϩʔϥʔΛ࣮ߦ͢Δ
    ˔  ྫ
    3FQMJDB
    4FUͷ1PE਺Λ؂ࢹͯ͠ɺ1PEͷొ࿥ΛߦͬͨΓ
    ɹɹɹɹɾɾɾ ɹଞ
    GCP Managed ($1ϚωʔδυͰΑΖ͘͠ʹ
    
13. Name Type ਺ ໾ׂ kubelet systemd 3 •  kube-apiserverͱ࿈ܞɺίϯςφىಈΛ͢Δ kube-proxy-* ୯ମ Pod 3 •  NWτϥϑΟοΫΛ؅ཧ kube-dns Deployment 2 •  Ϋϥελ಺ͷDNS kube-dns-autoscaler Deployment 1 •  kube-dnsͷεέʔϦϯάΛߦ͏ fluentd-gcp-v* DaemonSet 3 •  Stackdriver LoggingʹContainerͷϩάΛૹ͍ͬͯΔ fluentd-gcp-scaler Deployment 1 •  fluentdͷεέʔϦϯάΛߦ͏ event-exporter-v* Deployment 1 •  Stackdriver Loggingʹkube-apiserverͷϩάΛૹ͍ͬͯΔ metrics-server-v** Deployment 1 •  HPA༻ʹϝτϦΫεΛఏڙ heapster-v** Deployment 1 •  Stackdriver MonitoringʹϝτϦΫεΛૹ৴ 13 Our Project ˞
    (,&
    HLF
    ͷ
    /PEF
    ୆
    ͷ৔߹
    ˞
    MEFGBVMUCBDLFOE
    ͸ࠓճলུ
    
14. Our Project ˞
    (,&
    HLF
    ͷ
    /PEF
    ୆
    ͷ৔߹
    ˞
    MEFGBVMUCBDLFOE
    ͸ࠓճলུ
    Name Type ਺ ໾ׂ kubelet systemd 3 •  kube-apiserverͱ࿈ܞɺίϯςφىಈΛ͢Δ kube-proxy-* ୯ମ Pod 3 •  NWτϥϑΟοΫΛ؅ཧ kube-dns Deployment 2 •  Ϋϥελ಺ͷDNS kube-dns-autoscaler Deployment 1 •  kube-dnsͷεέʔϦϯάΛߦ͏ fluentd-gcp-v* DaemonSet 3 •  Stackdriver LoggingʹContainerͷϩάΛૹ͍ͬͯΔ fluentd-gcp-scaler Deployment 1 •  fluentdͷεέʔϦϯάΛߦ͏ event-exporter-v* Deployment 1 •  Stackdriver Loggingʹkube-apiserverͷϩάΛૹ͍ͬͯΔ metrics-server-v** Deployment 1 •  HPA༻ʹϝτϦΫεΛఏڙ heapster-v** Deployment 1 •  Stackdriver MonitoringʹϝτϦΫεΛૹ৴ 14 ݁ߏ͍Ζ͍Ζಈ͍ͯͨΜͩʂ
    ஌Βͳ͔ͬͨɻ ൓ল
    
15. ࢖͍Α͏ʹΑͬͯ͸ɺ
    མͪͯ͠·͏͜ͱ΋͋Δ͔΋
    15
16. ͳͷͰɺͻͱͭͣͭམͱͯ͠
    ӨڹΛ֬ೝ͠·ͨ͠
    16
17. 17 Our Project ˞
    (,&
    HLF
    ͷ
    /PEF
    ୆
    ͷ৔߹
    ˞
    ͸ͣ
    
    ະ֬ೝ
    ো֐࠶ݱଌఆ͕೉͔ͬͨ͠
    Name Type ਺ ॏཁ౓ ࢭΊͨͱ͖ͷڍಈ kubelet systemd 3 •  αʔϏε͸Քಇ͢Δ͕ɺPodͷ࡟আ/ىಈ͕ෆՄʹͳͬͨ kube-proxy-* Pod 3 •  ଈ࣌ʹ෮چͨ͠ͷͰ֬ೝͰ͖ͳ͔͕ͬͨɺ ίϯςφؒ௨৴͕ෆՄʹͳΔ?ͨΊαʔϏεো֐ʹͳΔ(͸ͣ) kube-dns Deployment 2 •  Serviceܦ༝ͷAPP → DB ͷૄ௨ෆՄɻαʔϏεো֐ൃੜ kube-dns-autoscaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) fluentd-gcp-v* DaemonSet 3 •  αʔϏεʹӨڹͳ͠ɻϩά͕సૹͰ͖ͳ͘ͳΔ fluentd-gcp-scaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) event-exporter-v* Deployment 1 •  αʔϏεʹӨڹͳ͠ɻΠϕϯτϩά͕సૹͰ͖ͳ͘ͳΔ(͸ͣ) metrics-server-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻHPA͕ػೳ͠ͳ͍ (͸ͣ) heapster-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻϝτϦΫε͕సૹͰ͖ͳ͘ͳΔ(͸ͣ)
18. 18 Our Project ˞
    (,&
    HLF
    ͷ
    /PEF
    ୆
    ͷ৔߹
    ˞
    ͸ͣ
    
    ະ֬ೝ
    ো֐࠶ݱଌఆ͕೉͔ͬͨ͠
    Name Type ਺ ॏཁ౓ ࢭΊͨͱ͖ͷڍಈ kubelet systemd 3 •  αʔϏε͸Քಇ͢Δ͕ɺPodͷ࡟আ/ىಈ͕ෆՄʹͳͬͨ kube-proxy-* Pod 3 •  ଈ࣌ʹ෮چͨ͠ͷͰ֬ೝͰ͖ͳ͔͕ͬͨɺ ίϯςφؒ௨৴͕ෆՄʹͳΔ?ͨΊαʔϏεো֐ʹͳΔ(͸ͣ) kube-dns Deployment 2 •  Serviceܦ༝ͷAPP → DB ͷૄ௨ෆՄɻαʔϏεো֐ൃੜ kube-dns-autoscaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) fluentd-gcp-v* DaemonSet 3 •  αʔϏεʹӨڹͳ͠ɻϩά͕సૹͰ͖ͳ͘ͳΔ fluentd-gcp-scaler Deployment 1 •  αʔϏεʹӨڹͳ͠ɻεέʔϧૢ࡞͸ෆՄʹͳΔ(͸ͣ) event-exporter-v* Deployment 1 •  αʔϏεʹӨڹͳ͠ɻΠϕϯτϩά͕సૹͰ͖ͳ͘ͳΔ(͸ͣ) metrics-server-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻHPA͕ػೳ͠ͳ͍ (͸ͣ) heapster-v** Deployment 1 •  αʔϏεʹӨڹͳ͠ɻϝτϦΫε͕సૹͰ͖ͳ͘ͳΔ(͸ͣ) ݸਓతɺؾΛ෇͚Α͏ϙΠϯτ
    %/4αʔό͕ಈ͍͍ͯΔͷ
    ҙࣝͯ͠ͳ͔ͬͨɻ ൓ল
    
19. kube-system Node Ͱಈ͍͍ͯΔ΍ͭ
    19 kubelet kube-proxy container runtime Node kube-apiserver default APP-B APP-A GCP Service kubernetes Master Namespaces heapstar event exporter metrics server Monitoring API Logging API HPA མͪͯ΋αʔϏεӨڹͳ͠ མͪΔͱαʔϏεӨڹ͋Γ Stackdriver
20. kube-system Node Ͱಈ͍͍ͯΔ΍ͭ
    20 kubelet kube-proxy container runtime Node kube-dns kube-apiserver autoscaler fluentd-gcp autoscaler default APP-B APP-A GCP Service kubernetes Master Namespaces Monitoring API Logging API Stackdriver མͪͯ΋αʔϏεӨڹͳ͠ མͪΔͱαʔϏεӨڹ͋Γ
21. kube-system Node Ͱಈ͍͍ͯΔ΍ͭ
    21 kubelet kube-proxy container runtime Node kube-dns kube-apiserver autoscaler fluentd-gcp autoscaler default APP-B APP-A GCP Service kubernetes Master Namespaces Monitoring API Logging API Stackdriver ໊લղܾ͕Ͱ͖ͳ͘ͳΔ
    མͪͯ΋αʔϏεӨڹͳ͠ མͪΔͱαʔϏεӨڹ͋Γ
22. ˔  LVCFTZTUFNͷ%FQMPZNFOU
    %BFNPOTFUΛEFMFUFͯ͠΋ɺ
    BEEPONBOBHFS
    ͕Ϧιʔε͝ͱউखʹ෮׆ͤ͞ΔΜͰ͢Ͷɻ
    ˓  LVCFDUM
    EFMFUF
    EFQMPZNFOUΛ࿈ଧͯ͠ɺো֐Λ࠶ݱ͠·ͨ͠׼
    
    ͓·͚ؾ͕͍ͭͨ͜ͱ
    22
23. ˔  LVCFTZTUFNͷো֐Ίͬͨʹى͖ͳ͍ͩΖ͏͕ɺҙࣝ͸ඞཁͱ࣮ײ
    ˓  ಛʹLVCFEOT
    ˓  ͪͳΈʹLVCFEOT͸୆Ҏ্Ͱ৑௕Խ͞Ε͍ͯΔ͕Ͱ΋
    
    ˔  ࢓૊Έ͸ͪΌΜͱཧղ͓͔ͯ͠ͳ͍ͱ͍͚ͳ͍ͳͱ࣮ײ
    ˓  ͳΜʹͰ΋ݴ͑Δ͜ͱͰ͕͢
    ˓  LT൒೥ۙ͘࢖͍͕ͬͯͨɺࠓճௐ΂ͯॳΊͯΘ͔ͬͨ͜ͱ͕ͨ͘͞Μ
    ˓  ·ͩ·ͩษڧෆ଍ͳͷͰɺ͜Ε͔ΒͪΌΜͱௐ΂·͢ɻ
    
    ·ͱΊ
    23
24. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    24 ($1 (,&
    Y
    Τϯϓϥ
    
    ΤϯδχΞઈࢍืूதͰ͢