Terraform管理下のマネージドリソースとk8sリソースを一元的にGitOpsするまでの試行錯誤

2023/2/27 CI/CD Conference 2023 前夜祭で登壇した資料です。
https://cloudnativedays.connpass.com/event/274402/

1. Terraform ؅ཧԼͷϚωʔδυϦιʔεͱ k8s ϦιʔεΛҰݩతʹ GitOps ͢Δ·Ͱͷ ࢼߦࡨޡ 2023/02/27 ιϑτόϯΫגࣜձࣾ Ϋϥ΢υΤϯδχΞϦϯάຊ෦ Cloud Native Days CI/CD Conference 2023 લ໷ࡇ ࡾվ໦ ༟໼
2. ࡾվ໦ ༟໼ (Mizorogi Yuya) 2 ࣗݾ঺հ ॴଐ ιϑτόϯΫגࣜձࣾ 
 ๏ਓࣄۀ౷ׅ Ϋϥ΢υΤϯδχΞϦϯάຊ෦ ɾ2021೥ʹத్ೖࣾ ɾ๏ਓ޲͚ͷΞϓϦέʔγϣϯج൫ͷ։ൃΛ୲౰ ɾझຯ͸ย෇͚ͱڝϓϩ 
 @_uya116
3. • ΞϓϦέʔγϣϯΤϯδχΞ͕ Kubernetes Λҙࣝ͠ͳͯ͘ྑ͍࢓૊Έ 3 ιϑτόϯΫͰ͸͜ΜͳαʔϏεΛ࡞͍ͬͯ·͢ ੡඼ϖʔδ͸ͪ͜Β
4. • Ϛωʔδυ Kubernetes ؀ڥʹࣄલఆٛࡁͷ Helm ύοέʔδΛద༻ 4 ιϑτόϯΫͰ͸͜ΜͳαʔϏεΛ࡞͍ͬͯ·͢ DEPLOYMENT STORAGE RDB VAULT MONITORING IAM ࣄલఆٛࡁύοέʔδ MESSAGING ͳͲ
5. • ϚωʔδυϦιʔεͱ Kubernetes Ϧιʔεͷ૒ํͷ؅ཧ͕ඞཁ ◦ ύϒϦοΫΫϥ΢υͷϚωʔδυ Kubernetes Λ࢖͏ͱͳ͓͞Β 5 എܠ
6. • IaC ܗࣜͱͦΕʹ൐͏ CD αʔϏε͕ҟͳΓύΠϓϥΠϯ͕෼͔Εͯ͠·͏ ◦ ͜ΕʹΑͬͯ؅ཧ͕൥ࡶʹͳΔ 6 ೋछྨͷϦιʔεΛ؅ཧ͢Δ͜ͱʹΑΔGitOpsͷ՝୊ Github Actions Jenkins AWS Code γϦʔζ Ϧιʔε CD αʔϏε Git Terraform CloudFormation Kubernetes ϚχϑΣετ ͳͲ ͳͲ ͳͲ IaC ܗࣜ ϚωʔδυϦιʔε Kubernetes Ϧιʔε
7. • IaC ܗࣜ or CD αʔϏεΛἧ͑Δ͜ͱʹΑΓҰݩԽͰ͖ͳ͍͔ݕ౼ 7 ݕ౼ Github Actions Jenkins AWS CodePipeline ͳͲ ͳͲ ͳͲ Ϧιʔε CD αʔϏε Git IaC ܗࣜ ϚωʔδυϦιʔε Kubernetes Ϧιʔε Terraform CloudFormation Kubernetes ϚχϑΣετ
8. • IaC ܗࣜΛἧ͑Δ৔߹ ◦ Terraform k8s provider ͔֤ϕϯμʔͷఏڙ͢Δ k8s controller ͕ީิ ◦ ύΠϓϥΠϯ͕؆ܿʹͳΔ͕ରԠϦιʔε͕ݶఆ͞Ε͍ͯΔ 8 ݕ౼ ʙIaC ܗࣜΛἧ͑Δʙ Github Actions Jenkins AWS CodePipeline ͳͲ ͳͲ ͳͲ Ϛωʔδυ Ϧιʔε ʹدͤΔ ᶃ Terraform k8s provider Kubernetes Ϧιʔε ʹدͤΔ ᶄ ConfigConnector / ASO / ACK controller ϚωʔδυϦιʔε Kubernetes Ϧιʔε IaC ܗࣜ Terraform CloudFormation Kubernetes ϚχϑΣετ
9. • CD αʔϏεΛἧ͑ͨ৔߹ ◦ ύΠϓϥΠϯ಺΍ద༻؀ڥͰίϚϯυΛ࣮ߦ͠ڧҾʹϦιʔεΛσϓϩΠ͢Δ ◦ ίϚϯυ࣮ߦͷͨΊࣗ༝౓͸ߴ͍͕ύΠϓϥΠϯ͕ෳࡶʹͳΔ 9 ݕ౼ ʙCD αʔϏεΛἧ͑Δʙ Github Actions Jenkins AWS CodePipeline ͳͲ ͳͲ Ϛωʔδυ Ϧιʔε ʹدͤΔ ᶅ k8s ίϯςΩετΛऔಘͯ͠ apply Kubernetes Ϧιʔε ʹدͤΔ ᶆ k8s ಺Ͱ Ϛωʔδυ༻ͷ IaC Λద༻ apply ϚωʔδυϦιʔε Kubernetes Ϧιʔε CD αʔϏε
10. 10 ݕ౼ ʙ֤ํ๏ͷൺֱʙ IaC ܗࣜΛἧ͑Δ CD αʔϏεΛἧ͑Δ Ϛωʔδυ Ϧιʔε ʹدͤΔ ᶃ Terraform k8s provider ᶅ k8s ίϯςΩετΛऔಘͯ͠ apply Kubernetes Ϧιʔε ʹدͤΔ ᶄ ConfigConnector / ASO / ACK ᶆ k8s ಺Ͱ Ϛωʔδυ༻ͷ IaC Λద༻ controller apply ෳࡶʹͳΔ͕ ࣗ༝౓͕ߴ͍ Ϧιʔε੍ݶ͕ ͋Δ͕؆ܿ ͲͪΒ͕ ϝΠϯ͔
11. • ฐαʔϏε͸ Kubernetes Ϧιʔεத৺ͷͨΊᶄᶆΛ࠾༻ ◦ ֤ϕϯμͷ k8s controller ʹରԠ͍ͯ͠ΔϚωʔδυϦιʔε → ᶄͰ࡞੒ ◦ ະରԠ΋͘͠͸ k8s controller Ͱͷಈ࡞͕ෆ҆ఆͳϦιʔε → ᶆͰ࡞੒ 11 ݕ౼݁Ռ IaC ܗࣜΛἧ͑Δ CD αʔϏεΛἧ͑Δ Kubernetes Ϧιʔε ʹدͤΔ ᶄ ConfigConnector / ASO / ACK ᶆ k8s ಺Ͱ Ϛωʔδυ༻ͷ IaC Λద༻ controller
12. • ݕ౼݁Ռʹैͬͯ GitOps ύΠϓϥΠϯΛߏங • Job Ͱ terraform ίϚϯυΛ࣮ߦ͢Δ͕࣮૷ͷෛՙ͕ߴ͍ ◦ Helm ࡟আ࣌ʹ Job Λ࣮ߦ͢Δ͕ terraform destroy ʹࣦഊ͢Δͱ Job ͕ࣦഊ͠ 
 Helm ͕ফͤͳ͘ͳΔ → ঢ়ଶʹԠͨ͡ίϚϯυͷ੍ޚ͕ඞཁ ◦ drift ൃੜ࣌ͷ੍ޚ ࢪࡦ ʙฐαʔϏεͷ GitOps ύΠϓϥΠϯʙ vender controller 12 ϕϯμͷ k8s controller ࣗ࡞ Job Ͱ terraform ࣮ߦ ※ ฐαʔϏεͷཱͯ෇্͚ Helm Ͱύοέʔδϯά͍ͯ͠Δ
13. • Terraform ϦιʔεΛ؅ཧ͢Δ Kubernetes Controller ◦ Flux ͱ࿈ܞͯ͠ Terraform ͷ GitOps Λ࣮ݱ͢Δ ◦ TF state ϑΝΠϧͷΫϥ΢υετϨʔδ؅ཧɺOIDC ࿈ܞͳͲඞཁͳػೳ͕ἧ͍ͬͯΔ 13 Weave GitOps Terraform Controller https://weaveworks.github.io/tf-controller/ Terraform Controller kind: Terraform ᶃݕ஌ ᶅ࡞੒ ᶄ TF ϑΝΠϧऔಘ
14. Weave GitOps Terraform Controller “ࣗ෼ͷϖʔε”Ͱ GitOps ͱ͍͏ίϯηϓτΛܝ͓͛ͯΓ 
 ϚχϑΣετʹ߹Θͤͨ Terraform apply / destroy ͷࣗಈద༻͚ͩͰͳ͘ 
 drift ͷݕग़ͷΈߦ͏͜ͱ΋Մೳ TF ϑΝΠϧͷ֨ೲݩɻflux ͷ GitRepository / OCIRepository Λࢦఆ TF ϑΝΠϧʹΘͨ͢؀ڥม਺ͷઃఆ terraform ίϚϯυΛ࣮ߦ͢Δ ServiceAccount Terraform CR Λ࡟আͨ͠ͱ͖ʹΫϥ΢υ্ͷϦιʔε΋ফ͔͢ backend ͷઃఆ
15. σϞ
16. • Weave GitOps Terraform Controller Λ༻͍Δ͜ͱͰҰݩ GitOps Λ࣮ݱ 16 ࢪࡦ ʙฐαʔϏεͷ GitOps ύΠϓϥΠϯʢAfterʣʙ vender controller ϕϯμͷ k8s controller ͰϦιʔε࡞੒ Terraform Controller Ͱ terraform ࣮ߦ terraform controller
17. • Flux v0.32.0 ͰରԠͨ͠ OCI ϦϙδτϦ ʹ TF ϑΝΠϧΛ֨ೲ͍ͯ͠Δ ◦ ͜ΕʹΑΓ Docker image, Helm Chart, TF ϑΝΠϧΛಉҰαʔϏεͰ؅ཧ͢Δ͜ͱ͕Մೳ ◦ ೝূํ๏΋ڞ௨ԽͰ͖Δ Weave GitOps Terraform Controller Ͱͷ޻෉ 17 AKS ฐαʔϏεͷ୲౰ൣғ OCI ϦϙδτϦ (Artifact Registry) GKE EKS
18. • Artifact Registry ʹ֤ k8s ͔ΒΞΫηε͢Δඞཁ͕͋Δ ◦ ظݶ͕௕͍ΫϨσϯγϟϧ৘ใ͸࣋ͪͨ͘ͳ͍ͨΊ OIDC ࿈ܞ͍ͨ͠ ◦ ͔͠͠ݱঢ় OCI ϦϙδτϦʹର͢Δ Flux source-controller Ͱ͸ GC ͱͷ OIDC ࿈ܞ͕ະରԠ 18 Weave GitOps Terraform Controller Ͱͷ޻෉ ✕ AWSͰݖݶҕ೚͞ΕͨτʔΫϯΛ༻͍ͯ GC ͱ࿈ܞ͍͕ͨ͠ɾɾɾ Flux
19. • OIDC ࿈ܞͨ͠ CronJob Ͱ imagePullSecret Λ࡞੒͢Δ ◦ imagePullSecret ͷߋ৽Λ OIDC ࿈ܞͨ͠ CronJob Ͱఆظతʹ࣮ࢪ͢Δ͜ͱͰ՝୊Λճආ ◦ ֤Ϛωʔδυ k8s ͔Β Artifact Registry ΁ͷΞΫηε͕Մೳͱͳͬͨ 19 Weave GitOps Terraform Controller Ͱͷ޻෉ ˕ Flux ✕ Flux Before After
20. • GitOps Λڞ௨Խ͢Δͱ࣍͸ςετ΋ڞ௨Խͨ͘͠ͳΔɾɾɾ 20 ςετʹ͍ͭͯ ςετର৅ vender controller terraform controller
21. • νʔϜϝϯόʔ͕ CI/CD ΧϯϑΝϨϯεຊฤͰ࿩͠·͢ʂ 21 ςετʹ͍ͭͯ
22. 22 ·ͱΊ 1. Weave GitOps Terraform Controller Λ༻͍Δ͜ͱͰ GitOps ͷҰݩԽΛ࣮ݱ 2. TF ϑΝΠϧ͸ OCI ϦϙδτϦͰ؅ཧ͠ imagePullSecret ൃߦʹΑΓΞΫηε 3. ςετͷڞ௨Խʹ͍ͭͯ͸ຊฤΛָ͓͠Έʹ vender controller terraform controller OCI ϦϙδτϦ