Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Is Kubernetes On-premises Hardway?
Search
Kazuhiko Yamashita
November 05, 2021
Technology
2
540
Is Kubernetes On-premises Hardway?
CloudNativeDays Tokyo 2021にて登壇した資料です。
Kazuhiko Yamashita
November 05, 2021
Tweet
Share
More Decks by Kazuhiko Yamashita
See All by Kazuhiko Yamashita
実践ARMアーキテクチャ移行
pyama86
2
2.1k
リモートワーク時代の守護神 PHP開発者のためのセキュリティ強化術
pyama86
3
920
実践DevSecOps~クラウドネイティブとオンプレミスの間から~
pyama86
1
51
ペパボOpenTelemetry革命
pyama86
2
1.8k
Site Reliability Engineering for GMO
pyama86
10
1.2k
PHPアプリケーションのスケーラビリティと 信頼性を革新する nginx+ngx_mrubyとGoの融合
pyama86
2
420
ペパボOpenTelemetry革命
pyama86
0
860
ChatGPTの活用を体現し、 組織に浸透する技術
pyama86
1
1.1k
リリースフラグと プログレッシブデリバリーを活用した 安全なWEBアプリケーションデプロイ
pyama86
2
270
Other Decks in Technology
See All in Technology
Jetpack Compose Modifier 徹底解説 / Jetpack Compose Modifier
wiroha
0
170
PDF Viewer作成の今までとこれから
hunachi
0
290
より快適なエラーログ監視を目指して
leveragestech
4
1.4k
四国のあのイベントの〇〇システムを45日間で構築した話 / cloudohenro2024_tachibana
biatunky
0
330
Segment Anything Model 2
tenten0727
3
660
テスト”ケース”駆動開発 で手戻りをなくそう
ryohma0510
0
170
Oracle Base Database Service:サービス概要のご紹介
oracle4engineer
PRO
0
13k
The XZ Backdoor Story
fr0gger
0
3.6k
Agile in Automotive Industry, puzzles and lights.
hiranabe
3
1.2k
Developer Experienceを向上させる基盤づくりの取り組み事例集
coconala_engineer
0
140
Fediverse Discovery Providers overview
andypiper
0
160
疎通2024
sadnessojisan
5
1k
Featured
See All Featured
Six Lessons from altMBA
skipperchong
26
3.3k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
23
1.7k
KATA
mclloyd
27
13k
Scaling GitHub
holman
458
140k
Embracing the Ebb and Flow
colly
83
4.4k
Debugging Ruby Performance
tmm1
72
12k
GraphQLとの向き合い方2022年版
quramy
43
13k
Creatively Recalculating Your Daily Design Routine
revolveconf
215
12k
How to train your dragon (web standard)
notwaldorf
85
5.6k
Build your cross-platform service in a week with App Engine
jlugia
228
18k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
26
1.9k
Transcript
Is Kubernetes On-premises Hardway? ʙ݁ࠗɺͦΕରͰ͋Δʙ
ࢁԼ!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ UFOTOBQPODPN QZBNBGVO TUOTKQ
45/4 -JOVY/444FSWFS TUOTKQ
45/4
https://github.com/pyama86/github-replacer
ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
Is Kubernetes On-premises Hardway?
ϚωʔδυαʔϏεͷϝϦοτ • Control Plane/Data Planeͷཧ • όʔδϣϯΞοϓͷ༰қ͞ • Ϋϥυࣄۀऀ͕ఏڙ͢ΔͦͷଞͷϚωʔδυαʔϏεͱͷ࿈ܞͷ༰қ͞ •
ແݶεέʔϦϯά(If you have much money)
ΦϯϓϨϛεͷϝϦοτ • ͯ͢ΛΒͶͳΒͳ͍͕ނʹࣗ༝ • ਓ݅අΛআ͘ϥϯχϯάίετͷ҆͞
ࠓ͢͜ͱ • ϖύϘͷKubernetesΫϥελͷ֓ཁ • Hardwayͩͬͨ͜ͱ • ࠓޙΓ͍ͨ͜ͱ
KubernetesΫϥελ • OpenStack (Nyah) • Nyah Kubernetes Engine(NKE)
KubernetesΫϥελ نײ • ࡐ͝ͱʹΫϥελΛ͓ͯ͠Γɺ23Ϋϥελ(ൃද࣌) • ࡐʹΑͬͯNKE / GKE / EKSͰͷϋΠϒϦουΫϥυͰར༻
• AWS Direct ConnectͰઐ༻ઢར༻
KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτΣΞͷಋೖ
• όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
NKE • ΫϥελͷߏஙɺόʔδϣϯΞοϓ • ΫϥελཧϚχϑΣετͷద༻ • Data PlaneͷՃɺআ • AnsibleΛ༻͍ͨϓϩϏδϣχϯά
ΫϥελཧΛίʔυԽ͠CLIΠϯλʔϑΣʔεʹͨ͠ͷ
NKE ઃఆϑΝΠϧɺൿಗใετΞʹج͖ͮɺ ΫϥελΛߏஙɺӡ༻ VM VM VM NKE tenant- con fi
g.toml Hashicorp Vault conta iner conta iner conta iner
NKE • Golang • Hashicorp Vault • Consul • Packer
ओཁίϯϙʔωϯτ
Kubernetesͷόʔδϣϯཧ
Kubernetesͷόʔδϣϯཧ • NKEͷϒϥϯν͝ͱʹόʔδϣϯཧ • trunk: ։ൃ༻ϒϥϯν • 1.20,1.21 ϦϦʔεϒϥϯν
Kubernetesͷόʔδϣϯཧ trunk 1.20 1.21 Unit Test E2E Test Unit Test
E2E Test Unit Test E2E Test merge merge
Kubernetesͷόʔδϣϯཧ • CIΛར༻ͨ͠ςετΛύεͨ͠߹ɺ։ൃ༻Ϋϥελɺࣾπʔϧ༻Ϋ ϥελͷόʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ֤Ϋϥελͷཧऀ͕όʔδϣϯΞοϓίϚϯυΛ࣮ߦ • ΫϥελʹΑͬͯ2ܥ࣋ͭΑ͏ʹͯ͠ɺόʔδϣϯΞοϓ࡞ۀͳͲͷ μϯλΠϜΛආ͚ΔΛ͍ͯ͠Δ
Kubernetesͷόʔδϣϯཧ • Control Plane,Data PlaneͱʹPodΛ͍ग़ͭͭ͠ɺ ϩʔϦϯάΞοϓσʔτ • Control PlaneɺEtcdʹ͍ͭͯ1ೖΕସ͑͝ͱʹϔϧενΣοΫΛ ͍ΕͯμϯλΠϜΛආ͚͍ͯΔ
Kubernetesͷӡ༻ཧ
Kubernetesͷӡ༻ཧ • ࢹ • ηΩϡϦςΟࠪ • CI/CD • ϩάཧ
Kubernetesͷࢹ Prometheus Alert Manager Grafana mackerel-agent ࣌ܥྻσʔλͷอଘ ڞ௨ϧʔϧʹै͍ɺSlack௨ PrometheusͷσʔλͷϏδϡΞϥΠθʔγϣϯ Prometheus+AlertManagerͷࢹ
KubernetesͷηΩϡϦςΟࠪ • Wazuh • Falco • GateKeeper
Wazuh https://atmarkit.itmedia.co.jp/ait/articles/1902/18/news012.html OSͷઃఆࠪ ෆਖ਼ΞΫηεݕ ੬ऑੑࠪ
Falco ίϯςφͷৼΔ͍ࠪɾݕ
Gatekeeper Admission ControllerͰಈ࡞͢Δ ϚχϑΣετͷࠪͳͲ Ұॹʹͬͯ͘Δਓɺೖࣾͯ͘͠Εʙʙʙʙ
ࣗಈApply ࢹɺηΩϡϦςΟϙϦγʔҰ੪ tag cluster A cluster B cluster C apply
CI/CD • ςετɺίϯςφϏϧυɺ੬ऑੑεΩϟϯGithub ActionsͷSelf Hosted Runner্Ͱ࣮ߦ • ίϯςφΠϝʔδͷεΩϟϯΤϯδϯtrivyΛར༻ • CDArgoCD
+ argocd-image-updaterΛར༻
ϩάཧ Kafkaʹूͯ͠ɺ༻్ʹԠͯ͡SaaS
͜͜·Ͱͨ͜͠ͱ • NKEίϚϯυͷ։ൃʹΑͬͯΫϥελͷߏஙϝϯςφϯεΛࣗಈԽͯ͠ ͍Δ • ࢹηΩϡϦςΟࠪʹ͍ͭͯNKEͰϕʔεͱͳΔͷΛఏڙ • όʔδϣϯΞοϓʹ͍ͭͯE2EͰಈ࡞Λ୲อͭͭ͠ɺ։ൃ༻ΫϥελͰ ͕ͳ͍͜ͱΛ֬ೝͯ͠ɺద༻͍ͯ͠Δ
Hardwayͩͬͨ͜ͱ
1.12.7
[࠶ܝ]KubernetesΫϥελ ٕज़ج൫νʔϜ Embedded SRE • NKEίϚϯυͷ։ൃ • ϓϦηοτϚχϑΣετͷߋ৽ • Ϋϥελ্Ͱಈ͘ιϑτΣΞͷಋೖ
• όʔδϣϯΞοϓͳͲͷϝϯςφϯε ։ൃͱར༻ऀ͕ҟͳΔ
όʔδϣϯΞοϓͷಈػ͕͍͜ͱ͕͋Δ • Ϋϥελͷ༻్ • ୲ऀ͕ଟ • Kubernetesɺ͍͍ͩͨݹͯ͘ಈ͘ • όʔδϣϯΞοϓʹର͢Δ৺ཧোน
όʔδϣϯཧࣗಈԽ͍ͨ͠ NKE Manifests Cluster A NKE Manifests Cluster B NKE
Manifests Cluster C NKE Cluster A Cluster B Cluster C manifestͷఆٛʹج͍ͮͯࣗಈͰऩଋͯ͠΄͍͠
͋Δಥવͷ ”error: You must be logged in to the server
(Unauthorized)”
Կ͕ى͖͔ͨ kube-apiserver Service Account token ServiceAccountͷར༻͍ͯ͠ΔτʔΫϯ͕ࣦޮͯ͠ೝূΤϥʔ
ͳͥى͖͔ͨ • Kubernetes ͷ SAτʔΫϯ༗ޮظݶ͕Forever • ϖύϘͷKubernetesͷSAτʔΫϯͷ伴ࣗಈͰϩʔςʔγϣϯ͍ͯ͠Δ
Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺHashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ͍ग़͠ τʔΫϯͷݕূ
Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ
Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺHashicorpVaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ͍ग़͠ τʔΫϯͷݕূ
Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ Ӭٱอଘʂʂʂ
Կ͕ى͖͔ͨ kube-apiserver Service Account token ূ໌ॻɺ伴ͷߋ৽ɺHashicorp VaultͰࣗಈԽ kube-controller- manager τʔΫϯͷ͍ग़͠
τʔΫϯͷݕূ Hashicorp Vault Cert Key ূ໌ॻͱ伴ͷࣗಈ 伴͕ߋ৽͞ΕΔ͜ͱͰ ݕূ͕Ͱ͖ͳ͘ͳΔ
ରॲʂѹతఆରॲʂʂʂ ಈ͍͍ͯΔϙουಈ͖ଓ͚Δ͕ɺϦεέδϡʔϧ͕Ͱ͖ͳ͍ͷͰ ·ͣShellͰରॲ ͜ͷ͋ͱɺূ໌ॻͷঢ়گΛࢹͯ͠ஔ͖͑ΔϓϩηεΛಈ͔͍ͯ͠·͢
࠷ޙͷॴײ • ΦϯϓϨKubernetesΔͳΒཧιϑτΣΞΛ։ൃͨ͠΄͏͕౷߹తʹཧͰ͖ΔͷͰ ࠷ऴίετམͱͤΔͱࢥ͏ • Kubernetesͦͷͷͱͯྑ͘Ͱ͖͍ͯͯɺKubernetesࣗମͷԿ͔Λ౿Ή͜ͱͦΜͳʹͳ͍ • ࠓհ͍ͯ͠ͳ͍ωοτϫʔΫपΓͷΧʔωϧνϡʔχϯάͳͲɺඞཁͳέʔε͋ͬͨͷ Ͱɺͦ͏͍͏ྖҬΛݟΕΔਓ͕͍ͳ͍ͱݫ͍͠ͱࢥ͏ •
େମͷϢʔεέʔεVM + DockerͰࣄΓΔͷͰɺ΄ΜͱʹͦΕKubernetes͍Δͷʁͱ͍͏ έʔε݁ߏ͋Δͱࢥ͏ • ͜Ε·ͰͷιϑτΣΞʹՃ͑ͯɺKubernetesͱ͍͏ϨΠϠʔ͕ೖΔ͜ͱͰτϥϒϧγϡʔτ ཧେมʹͳΔ
͓͠·͍ ࠷৽ͷ࠾༻ใΛνΣοΫˠ !QC@SFDSVJU