Share
Embed
Transcript
56 slides
-
∁ੴྒ / GMO Pepabo, Inc. 2018-12-04 JapanContainerDays v18.12 ฏ࠷ޙͷ12݄ʂʂ1 "OTJCMFɺ5FSSBGPSNɺ1BDLFS
Ͱ࡞Δ4FMG)PTUFE,VCFSOFUFT -
∁ੴྒ!S@UBLBJTIJ TAKAISHI Ryo Software Engineer @ GMO Pepabo, Inc. Belgian
Beer Go, Ruby Kubernetes, OpenStack https://repl.info/ -
• OpenStackΛ༻͍ͨPrivateCloudΛӡ༻͍ͯ͠Δ • NyahʢIaaSɺLBaaSʣ • AWSͷΑ͏ͳPublicCloud༻ • ϋΠϒϦουΫϥυߏͷαʔϏε͋Δ !4 લఏ
ϖύϘͷϓϥΠϕʔτΫϥυ “Nyah” ͦͷޙ by @tnmt https://speakerdeck.com/tnmt/pepabos-privatecloud-nyah-after-that minne ͰͷϋΠϒϦουΫϥυӡ༻ by @shiro16 https://speakerdeck.com/shiro16/minne-defalsehaiburitudokuraudoyun-yong -
• ੍Λڧ͘͢Δ͜ͱͰΑΓߴ͍ੜ࢈ੑΛಘΔ • Կ͔ΒखΛ͚ͭΔ͔ • => ·ͣCaaS͕Α͔Ζ͏ɻͳΒKubernetes͔ͳʁ • ΑΓखܰʹ͑ΔΠϯϑϥΛࢦͨ͢ΊɺIaaS෦ΛӅṭ͍ͨ͠ •
PaaSFaaSCaaS্ʹߏஙͰ͖Δ͕ٯ͍͠ !6 $BB41BB4'BB4Λఏڙ͢Δ͜ͱͰੜ࢈ੑ61ˢ -
• ͔͠͠ɺKubernetesͷߏஙɾӡ༻ۃྗলྗԽ͍ͨ͠ • Kubernetesͷӡ༻Ͱফ͢Δͷຊస • kubelet / apiserver / controller-manager
/ scheduler / etcd / etc… • 1ΫϥελͱݶΒͳ͍ • શͯࣗͨͪͰཧ͢Δͷେม • ઃఆมߋɺόʔδϣϯΞοϓ • Խ • ো࣌ͷ෮چ࡞ۀ !8 Ͳ͏ͬͯߏஙɾӡ༻͢Δ͔ -
• Ϛωʔδυ Self-Hosted KubernetesͲ͏ͩΖ͏ʁ • KubernetesࣗʹࣗࣗΛཧͤ͞Δߟ͑ํ • Kubernetesͷ࣋ͭߴͳػೳΛ͏͜ͱ͕Ͱ͖Δ • Deployment,
Auto Healing, etc… • Kubernetes as a Serviceͱҧ͏ͷ͔ • ͪ͜ΒɺෳͷΫϥελΛKubernetesͰཧ͢Δͷ !9 ,VCFSOFUFTͷཧΛΑΓָʹ͢Δʹ -
!11 1. Small Dependencies 2. Deployment consistency 3. Introspection 4.
Cluster Upgrades 5. Easier Highly-Available Configurations 1SPQPTBM4FMGIPTUFE$POUSPM1MBOF IUUQTHJUIVCDPNLVCFSOFUFTDPNNVOJUZCMPCNBTUFSDPOUSJCVUPSTEFTJHOQSPQPTBMTDMVTUFSMJGFDZDMFTFMGIPTUFELVCFSOFUFTNE -
• kubeletͰ֤ίϯϙʔωϯτΛಈ͔͢͜ͱͰɺϗετ্ʹඞཁͳί ϯϙʔωϯτΛݮΒ͢͜ͱ͕Ͱ͖Δ • KubernetesͷηοτΞοϓʹ͓͚Δෳࡶͷղফʹܨ͕Δ • ࠓճߏஙͨ͠ڥͷ߹ • DockerͱkubeletͷΈsystemdͰཧ •
ଞͷίϯϙʔωϯτkubeletཧԽ !12 4NBMM%FQFOEFODJFT -
• σΟεΫ্ͷϑΝΠϧSSHΛ༻͍ͨઃఆཧΛݮΒͤΔ • ϗετOSͷґଘΛݮΒ͠ɺσϓϩΠͷҰ؏ੑΛಘΔ • ࠓճߏஙͨ͠ڥͷ߹ • ϑΝΠϧͱͯ͠ஔ͘ඞཁ͕͋ΔϚχϑΣετΛSSHͰஔ • ଞKubernetes
APIܦ༝ !13 %FQMPZNFOUDPOTJTUFODZ -
!17 ηϧϑϗετ͢Δൣғʹ͍ͭͯ 0 1 2 3 4 kubelet etcd apiserver
scheduler controller- manager proxy dns addons 2-4 cluster 1-4 cluster -
• bootkubeͷΑ͏ͳطଘͷߏஙπʔϧΛͬͯͪΖΜΑ͍ • ͜Ε·Ͱ͖ͬͯͨπʔϧͰͲ͜·ͰΕΔͷ͔ɺࢼͯ͠ΈΑ͏ • KubernetesࣗମͷཧղΛਂΊ͍ͨ • Packer / Terraform
/ AnsibleΛ͍ͬͯΕձࣾͷଞϝϯόʔѻ ͍͍͢ͷͰɺͱߟ͑ͨ !19 ࠓճࣗ࡞͢Δ͜ͱʹͨ͠ -
• 2-4Ϋϥελ • جຊతʹ Kubernetes on Kubernetes on OpenStack (Ծ)
ʹ४ڌ • ΊͬͪΌࢀߟʹ͍͍͍ͤͯͨͩͯ͞·͢ :kansya: !21 ࠓճνϟϨϯδͨ͠4FMG)PTUFEʹ͍ͭͯ kubelet SystemdཧԼ etcd StaticPodΛ༻͍ͯkubeletཧԼ apiserver controller-manager scheduler Kubernetes APIΛ༻͍ͯKubernetesཧԼ Kubernetes on Kubernetes on OpenStack (Ծ) by @yuanying https://docs.google.com/presentation/d/1VKk89MaNkGRSlpBsOOHJt8cLD6mpZ5V55GEJqIDu2Sk/edit#slide=id.g28eb8071bc_0_1119 -
• Packer • ϕʔεΠϝʔδʹDockerkubeletɺͦͷଞશϊʔυͰ͏ιϑτΣΞͷΠϯ ετʔϧʹ༻ɻ • Ϋϥελ࡞࣌ɺϊʔυՃ࣌ͷ࣌ؒॖΛݟࠐΜͩ • Terraform •
αʔόʔɺূ໌ॻൃߦʹ༻ • OpenStack Provider, TLS Provider • Ansible • Ϋϥελʔߏஙɺߋ৽ʹ༻ • Dynamic InventoryΛ͏ͱTerraform͔Βద༻ઌIPΞυϨεΛऔಘͰ͖ͯศར !22 1BDLFS5FSSBGPSN"OTJCMFͲ͜Ͱ͔ͬͨ -
1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ
4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted ClusterΓସ͑Δ !23 ߏஙͷྲྀΕ -
1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ
4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted ClusterΓସ͑Δ !24 ߏஙͷྲྀΕ -
• શαʔόʔ͕ඞཁͱ͢ΔίϯϙʔωϯτΛ༧ΊΠϯετʔϧ • Docker, kubeletͳͲ • ΫϥελʔߏஙɺϊʔυՃ࣌ͷ࣌ؒॖ͕త • Πϝʔδߋ৽ͰDockerɾkubeletͷೖΕସ͑Λߦ͏߹͕ඞཁ !25
1BDLFS"OTJCMFͰϕʔεΠϝʔδΛ࡞ Image Server Packer Ansible Install Docker, kubelet, etc Launch Server Save Base Image -
1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ
4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted ClusterΓସ͑Δ !26 ߏஙͷྲྀΕ -
5FSSBGPSNͰαʔόʔΛىಈ͢Δ Docker kubelet master Docker kubelet master Docker kubelet master
Docker kubelet node Docker kubelet node Terraform Launch servers certificates and private key -
5FSSBGPSNͰαʔόʔΛىಈ͢Δ Docker kubelet master Docker kubelet master Docker kubelet master
Docker kubelet node Docker kubelet node Terraform Launch servers certificates and private key • ֤छূ໌ॻɾൿີ伴͜ͷλΠϛϯάͰੜ • TLS Provider • repl.infoɿTerraformͷTLS ProviderΛࢼ͢ • αʔόʔىಈ࣌ʹcloud-configͰͯ͠ஔ • tfstateʹฏจͰอଘ͞ΕΔ(!)ͨΊɺࠓޙํࣜΛม͑Δ༧ఆ -
1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ
4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted ClusterΓସ͑Δ !29 ߏஙͷྲྀΕ -
"OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd master Docker kubelet etcd master Docker
kubelet etcd master Docker kubelet node Docker kubelet node Ansible Launch etcd cluster Static Pod manifest files -
"OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd master Docker kubelet etcd master Docker
kubelet etcd master Docker kubelet node Docker kubelet node • etcd clusterΛߏங͢Δ • masterʹStatic Pod༻ϚχϑΣετϑΝΠϧΛஔ͘ • Static Pod ? Ansible Launch etcd cluster Static Pod manifest via files -
• kubeletࢦఆͨ͠σΟϨΫτϦΛࢹͯ͠ɺஔ͔ΕͨϚχϑΣετ ϑΝΠϧΛݩʹPodΛىಈ͢ΔػೳΛ࣋ͭ • ͜ΕΛStatic PodͱݺͿ !32 4UBUJD1PEʁʁʁ kubelet /etc/kubernetes/manifests
XBUDI ϚχϑΣετϑΝΠϧ͕ஔ͔ΕͨΒ1PEΛ࡞ ফ͞ΕͨΒ1PEΛআ Docker IUUQTLVCFSOFUFTJPEPDTUBTLTBENJOJTUFSDMVTUFSTUBUJDQPE QPENBOJGFTUQBUI NBOJGFTUVSM http://pod.example.com/ -
"OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd bootstrap apiserver bootstrap controller-manager bootstrap scheduler
master Docker kubelet etcd master Docker kubelet etcd master Docker kubelet node Docker kubelet node Ansible Launch bootstrap cluster StaticPod manifest via files apiserverɿ໊લͷ௨ΓɺKubernetes APIͷΤϯυϙΠϯτ schedulerɿPodΛNodeʹׂΓͯΔ controller-managerɿReplicaͳͲΛίϯτϩʔϧ͢Δ -
"OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd bootstrap apiserver bootstrap controller-0manager bootstrap scheduler
master Docker kubelet etcd master Docker kubelet etcd master Docker kubelet node Docker kubelet node • apiserver, controller-manager, schedulerΛStatic PodͰىಈ • ͜ͷ࣌ͰKubernetes API͕ར༻ՄೳͱͳΔ • ֤αʔόʔͷkubelet͕apiserverొɺAPIͰݟ͑ΔΑ͏ʹͳΔ • controller-manager, schedulerಈ͍͍ͯΔͷͰDeploymentͳͲ͑Δ Ansible Launch bootstrap cluster StaticPod manifest via files -
"OTJCMFͰ#PPUTUSBQ$MVTUFSΛىಈ͢Δ Docker kubelet etcd bootstrap apiserver bootstrap controller-manager bootstrap scheduler
master Docker kubelet etcd master Docker kubelet etcd master Docker kubelet node Docker kubelet node register node Ansible Launch bootstrap cluster StaticPod manifest via files -
1. Packer / AnsibleͰϕʔεΠϝʔδΛ࡞ʢDocker, kubeletʣ 2. TerraformͰαʔόʔΛىಈ 3. AnsibleͰBootstrap ClusterΛىಈ
4. AnsibleͰBootstrap Cluster͔ΒSelf-Hosted ClusterΓ ସ͑Δ !36 ߏஙͷྲྀΕ -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd master Docker kubelet etcd master Docker
kubelet etcd master Docker kubelet node Docker kubelet node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd kube-proxy master Docker kubelet etcd kube-proxy
master Docker kubelet etcd kube-proxy master Docker kubelet kube-proxy node Docker kubelet kube-proxy node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd kube-proxy flannel master Docker kubelet etcd
kube-proxy flannel master Docker kubelet etcd kube-proxy flannel master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel master Docker kubelet
etcd apiserver kube-proxy flannel master Docker kubelet etcd apiserver kube-proxy flannel master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API NodeAffinityΛͬͯmasterͷΈىಈ -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker
kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master
Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Launch self-hosted cluster Manifest files via Kubernetes API NodeAffinityΛͬͯmasterͷΈىಈ -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master
Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler • DaemonSetɺDeploymentͰىಈ Ansible Launch self-hosted cluster Manifest files via Kubernetes API -
4FMG)PTUFE$MVTUFS༻ίϯϙʔωϯτΛىಈ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master
Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler • bootstrap apiserverͱapiserverͰ༻ϙʔτ͕ িಥͦ͠͏ • ࣮ࡍɺ͜ͷapiserverىಈʹࣦഊ͢Δ • ͔͠͠ɺ͜ͷޙແࣄʹىಈͯ͘͠Δ Ansible Launch self-hosted cluster Manifest files via Kubernetes API -
#PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFSΓସ͑Δ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master
Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node bootstrap apiserver bootstrap controller-manager bootstrap scheduler Ansible Delete bootstrap cluster Delete StaticPod manifest files -
#PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFSΓସ͑Δ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master
Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node Ansible Delete bootstrap cluster Delete StaticPod manifest files -
#PPUTUSBQ$MVTUFS͔Β4FMG)PTUFE$MVTUFSΓସ͑Δ Docker kubelet etcd apiserver kube-proxy flannel scheduler controller-manager master
Docker kubelet etcd apiserver kube-proxy flannel scheduler master Docker kubelet etcd apiserver kube-proxy flannel controller-manager master Docker kubelet kube-proxy flannel node Docker kubelet kube-proxy flannel node • bootstrap apiserverͷϓϩηε͕͍ͳ͘ͳͬͨ ͷͰɺapiserverىಈϦτϥΠʹΑΓىಈͯ͠ ͘Δ Ansible Delete bootstrap cluster Delete StaticPod manifest files -
!48 )JHI"WBJMBCJMJUZ etcd StaticPodΛ͍ɺmaster3ʙͰΫϥελԽ apiserver DaemonSetΛ͍ɺmasterશͰՔಇ keepalived + haproxyͰԽ (Ϋϥελ֎ʹLBaaSͰLoadBalancerΛઃஔ͢ΔͷՄʣ
controller-manager scheduler DeploymentΛ͍ɺmasterෳͰՔಇ -
• Ϋϥελࣗମͷߏஙɾӡ༻ • Կͷίϯϙʔωϯτ͕ඞཁ͔ɺԿͷׂΛ͍࣋ͬͯΔͷ͔ • ֤ίϯϙʔωϯτͷΦϓγϣϯʹ͍ͭͯͷཧղʢΊͪΌଟ͍Ͱ͢ΑͶ…ʣ • Ϋϥελ্ͰιϑτΣΞΛ͏·͘ಈ͔͢ • controller-managerschedulerͳͲͷPodΛ࠷1ͭҡ͍࣋ͨ͠
• PodͷೖΕସ͑NodeͷDrain !51 Ϋϥελͷ্ɾԼ྆ํͷݟΛಉ࣌ʹಘΒΕͨ -
• ݱࡏAnsibleɿ1ϊʔυͣͭαʔϏεΞτ&ߋ৽&αʔϏεΠϯ • ࠷ॳImmutable Infrastructureͷߟ͑ํΛద༻Ͱ͖ͳ͍͔ߟ͑ͨ • ϕʔεΠϝʔδΛೖΕସ͑ͯߋ৽ • TerraformͰΖ͏ͱ͢Δͱ݁ߏେม •
masterϊʔυetcdΛͲ͏͢Δʁ • ࠓޙCustom ControllerΛͬͯཧ͢Δ͜ͱΛݕ౼͍ͯ͠Δ !52 %PDLFSLVCFMFUͷߋ৽ΛͲ͏͢Δ͔ -
• TerraformͰϊʔυͷ૿ݮՄೳ • ݮΒ࣌͢ʹdrain͢Δඞཁ͕͋Δ • KubernetesཧԼʹஔ͚ΔͱΑΓΑ͘Ͱ͖ͦ͏ • Kubernetes APIܦ༝ͰεέʔϧΠϯɾΞτ •
HPAͷΑ͏ͳΦʔτεέʔϧػߏͱΈ߹ΘͤΔ͜ͱͰ͖ͦ͏ • ͪ͜ΒCustom ControllerԽ͢Δ͜ͱΛݕ౼ !53 ϊʔυͷ૿ݮΛͲ͏͢Δ͔ -
• ൚༻తͳπʔϧΛͬͯSelf-Hosted KubernetesͷߏஙΛߦͬͨ • ͔ͳΓ͍͍ײ͡ʹಈ࡞͢ΔΫϥελʔΛ࡞Δ͜ͱ͕Ͱ͖ͨ • ͔͠͠ɺݶքײͨ͡ • ϊʔυͷೖΕସ͑ •
ϊʔυίϯϙʔωϯτͷߋ৽ • ࠓޙCustom ControllerΛ׆༻ͯ͠ΑΓޮతͳཧΛ࣮ݱ͍ͨ͠ !55 ·ͱΊ