Kosko - 改用 JavaScript 來管理 Kubernetes YAML (Kubernetes Summit 2021)

GitHub: https://github.com/tommy351/kosko/
More info: https://kosko.dev/

Transcript

1. Kosko 改⽤ JavaScript 來管理 Kubernetes YAML Tommy Chen @ Kubernetes Summit 2021
2. About Me ⽬前在 Dcard 當 Architect，是個前後端 都在寫的雜⾷性⼯程師 Blog: https://zespia.tw GitHub: @tommy351 Twitter: @tommy351
3. 很久以前
4. 2016 年 6 ⽉ • Dcard 剛搬到 Kubernetes 上 • ⼤約有 20 個 components • 分成兩個 Git branch 對應 staging 和 production 環境 • 檔案結構⼤概像這樣 👉
5. 不到⼀年後
6. Git Branch 無法合併 • 有些 component 只會在其中 ⼀個環境佈署 • Staging 資料量不⾜所以不佈署 • Production 的 DB 佈署在 VM • 有些 config 在不同環境間的差 異很⼤ • Production 的 DB 是 cluster • Staging 會共⽤ DB Production Staging
7. 操作失誤 • 更新到錯的 Kubernetes cluster • 可能會造成 downtime • 改到錯的 Git branch • 因為 branch 無法合併，所以全靠 copy & paste • 沒有事前驗證 YAML 內容 • 即使有 PR review 依然很難避免錯誤發⽣
8. 難以重複使⽤ • YAML anchor 能夠重複使⽤，但無法跨檔案引⽤ • 只能透過 ConfigMap 或 Secret 來共⽤設定 • 只能以環境變數或 volume 的形式使⽤ • 但是設定以外的部分就無法重複使⽤了 • Sidecar • CronJob • 變數 (e.g., domain, image registry) • Component (e.g., PostgreSQL, Redis)
9. ⼜過了⼀年半
10. 現成⽅案 • Helm • Kustomize • Ksonnet (2018 年底)
11. Helm 👍 優點 • 有很多現成的 chart • 版本管理 • 透過指定不同的 values，即可 佈署到不同的環境 • 內建 Linter 和測試⼯具 👎 缺點 • ⽤ template 寫 YAML 很⿇煩 • ⽐較複雜，需要花時間上⼿ • Helm 佈署的⽅式不同，從 YAML 轉移到 Helm ⽐較困難
12. Source: ingress-nginx 3.34.0 · kubernetes/ingress-nginx (artifacthub.io) 縮排 😰
13. Kustomize 👍 優點 • 內建在 kubectl，不須額外安 裝其他⼯具 • 使⽤ YAML 學習成本低 • 可以 Patch 任意資源 • 利⽤ Overlay 可以佈署到不同 環境 👎 缺點 • 沒有內建驗證功能 • YAML 不易重複使⽤ • 有些情況不⽅便 Patch
14. apiVersion: apps/v1 kind: Deployment metadata: name: example spec: replicas: 1 selector: matchLabels: app: example template: metadata: labels: app: example spec: containers: - name: nginx image: nginx Base apiVersion: apps/v1 kind: Deployment metadata: name: example spec: replicas: 3 Patch apiVersion: apps/v1 kind: Deployment metadata: name: example spec: replicas: 3 selector: matchLabels: app: example template: metadata: labels: app: example spec: containers: - name: nginx image: nginx Result
15. Ksonnet 👍 優點 • Jsonnet ⽐較容易重複利⽤ • 內建驗證功能 • ⽀援多環境 👎 缺點 • Jsonnet 需要另外花時間學習， 且編輯器⽀援較差 • 概念⽐較複雜，需要花時間上 ⼿ • 已停⽌維護💀(2019/2)，後繼 專案：kubecfg, Tanka
16. Kosko • 使⽤ JavaScript • 容易上⼿ • ⽀援多環境 • 資料驗證 • 無痛轉移 https://kosko.dev
17. Why JavaScript? • 學習成本低 • Dcard 後端主要使⽤ Go 和 JavaScript • JavaScript ⽐起 Go 更適合⽤來寫 config • 容易重複利⽤ • 可以直接重複使⽤變數和函數 • 現成資源多 • 有很多現成的 library 可以直接⽤ • 編輯器整合極佳
18. https://youtu.be/5ip4uUUhtu4
19. import { Pod } from "kubernetes-models/v1/Pod"; API Version Kind 更多範例 import { Deployment } from "kubernetes- models/apps/v1/Deployment"; import { Certificate } from "@kubernetes- models/cert-manager/cert-manager.io/v1/Certificate";
20. Example: Pod import { Pod } from "kubernetes-models/v1/Pod"; const pod = new Pod({ metadata: { name: "busybox" }, spec: { containers: [ { name: "busybox", image: "busybox", command: ["sleep", "10000"] } ] } }); export default pod; apiVersion: v1 kind: Pod metadata: name: busybox spec: containers: - name: busybox image: busybox command: - sleep - '10000'
21. Example: Deployment + Service import { Deployment } from "kubernetes-models/apps/v1/Deployment"; import { Service } from "kubernetes-models/v1/Service"; const deployment = new Deployment({ metadata: { name: "nginx" }, spec: { // ... } }); const service = new Service({ metadata: { name: "nginx" }, spec: { // ... } }); export default [deployment, service]; --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx --- apiVersion: v1 kind: Service metadata: name: nginx
22. Example: Secret import { Secret } from "kubernetes-models/v1/Secret"; const secret = new Secret({ metadata: { name: "api-key" }, type: "Opaque", data: { secret: Buffer.from("confidential").toString("base64") } }); export default secret; apiVersion: v1 kind: Secret metadata: name: api-key type: Opaque data: secret: Y29uZmlkZW50aWFs
23. Example: YAML import { loadFile, loadUrl, loadString } from "@kosko/yaml"; const manifest = loadFile("manifest.yaml"); const certManager = loadUrl( "https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert- manager.yaml" ); const pod = loadString(` apiVersion: v1 kind: Pod metadata: name: my-pod `); export default [manifest, certManager, pod];
24. Example: Helm Chart import { loadChart } from "@kosko/helm"; const prometheus = loadChart({ chart: "prometheus", repo: "https://prometheus-community.github.io/helm-charts", version: "15.0.1", name: "prom-demo", namespace: "prom", values: { server: { ingress: { enabled: true } } } }); export default prometheus;
25. Example: Kustomize import { loadKustomize } from "@kosko/kustomize"; // Current folder const manifest = loadKustomize({ path: __dirname }); // GitHub repo const promOperator = loadKustomize({ path: "github.com/prometheus-operator/prometheus-operator" }) export default [manifest, promOperator];
26. Environment • 存放針對不同 Kubernetes cluster 設定的環境變數 • Global variables • 在多個 component 裡共享的環境變數 • 例如：image registry, domain name, namespace • Component variables • 僅在單⼀ component 裡使⽤的環境變數 • 例如：API key, database name, replicas
27. export default { namespace: "dev" }; Global vars export default { replicas: 1 }; Component vars import env from "@kosko/env"; // { namespace: "dev", replicas: 1 } const params = env.component("nginx"); const deployment = new Deployment({ metadata: { name: "nginx", namespace: params.namespace }, spec: { replicas: params.replicas, template: { spec: { containers: [{ name: "nginx", image: "nginx" }] } } } }); Component apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: dev spec: replicas: 1 template: spec: containers: - name: nginx image: nginx $ kosko generate --env dev
28. export default { namespace: "prod" }; Global vars export default { replicas: 15 }; Component vars import env from "@kosko/env"; // { namespace: "prod", replicas: 15 } const params = env.component("nginx"); const deployment = new Deployment({ metadata: { name: "nginx", namespace: params.namespace }, spec: { replicas: params.replicas, template: { spec: { containers: [{ name: "nginx", image: "nginx" }] } } } }); Component apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: prod spec: replicas: 15 template: spec: containers: - name: nginx image: nginx $ kosko generate --env prod
29. 資料驗證 • 使⽤ Kubernetes OpenAPI Spec 產⽣ TypeScript declaration 和 JSON schema • ⽀援 Kubernetes 內建資源和第三⽅ CRD • https://github.com/tommy351/kubernetes-models-ts
30. OpenAPI spec TypeScript declaration JSON schema
31. 使⽤經驗
32. 轉移到 Kosko • ⼤約花了⼀個⽉多 (2018/12~2019/1) • Kosko 內建 migrate 指令，能夠把 YAML 轉換成 JavaScript • 花了⼀個⽉把變數抽到 environment • ⽐較轉換前後的 YAML，只要相同的話就能保證相容 • 實際環境測試
33. Environment • 只把必要的變數放到 environment • 降低維護成本 • 避免 environment 過於冗⻑ • 改變資料夾結構 👉 • 根據 component 來區分更易讀 Group by type Group by component
34. 佈署 • CI 上只驗證內容是否正確 • 在本機佈署到 Kubernetes • ⽬前還是使⽤ kubectl apply • 寫了⼀個 script 來避免佈署到錯的 cluster • 重要 component 在佈署前會額外檢查版本狀態
35. 重複利⽤
36. Example: Selector const labels = { app: "demo" }; const deployment = new Deployment({ spec: { selector: { matchLabels: labels }, template: { metadata: { labels } } } }); const service = new Service({ spec: { selector: labels } });
37. Example: Port const httpPort = 80; const deployment = new Deployment({ spec: { template: { spec: { containers: [{ ports: [{ containerPort: httpPort }] }] } } } }); const service = new Service({ spec: { ports: [{ port: httpPort }] } });
38. Example: Environment Variables export function envVars(envs: Record<string, string>): IEnvVar[] { return Object.entries(envs).map(([name, value]) => { return { name, value }; }); } // Before [ { name: "LOG_LEVEL", value: "info" }, { name: "API_URL", value: "https://example.com" } ]; // After envVars({ LOG_LEVEL: "info", API_URL: "https://example.com" }); • 避免 name 重複 • 寫起來⽐較簡短
39. Example: Sidecar function withEnvoy(spec: IPodSpec): IPodSpec { return { ...spec, containers: [...spec.containers, { name: "envoy", image: "envoyproxy/envoy:v1.17.0" }] }; } const deployment = new Deployment({ spec: { selector: {}, template: { spec: withEnvoy({ containers: [{ name: "demo", image: "demo" }] }) } } }); apiVersion: apps/v1 kind: Deployment spec: selector: {} template: spec: containers: - name: demo image: demo - name: envoy image: envoyproxy/envoy:v1.17.0
40. Example: Component function createDatabase(name: string) { return [ new Deployment({ metadata: { name } }), new Service({ metadata: { name } }), new PersistentVolumeClaim({ metadata: { name } }) ]; } // components/post-api.js export default [ createDatabase("post-api-db"), new Deployment({ metadata: { name: "post-api" } }), new Service({ metadata: { name: "post-api" } }) ]; // components/user-api.js export default [ createDatabase("user-api-db"), new Deployment({ metadata: { name: "user-api" } }), new Service({ metadata: { name: "user-api" } }) ];
41. Thanks! Dcard 招募中 https://join.dcard.today Try Kosko! https://kosko.dev @tommy351