Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kosko - 改用 JavaScript 來管理 Kubernetes YAML (Kube...

Tommy Chen
December 22, 2021

Kosko - 改用 JavaScript 來管理 Kubernetes YAML (Kubernetes Summit 2021)

Tommy Chen

December 22, 2021
Tweet

More Decks by Tommy Chen

Other Decks in Technology

Transcript

  1. 2016 年 6 ⽉ • Dcard 剛搬到 Kubernetes 上 •

    ⼤約有 20 個 components • 分成兩個 Git branch 對應 staging 和 production 環境 • 檔案結構⼤概像這樣 👉
  2. Git Branch 無法合併 • 有些 component 只會在其中 ⼀個環境佈署 • Staging

    資料量不⾜所以不佈署 • Production 的 DB 佈署在 VM • 有些 config 在不同環境間的差 異很⼤ • Production 的 DB 是 cluster • Staging 會共⽤ DB Production Staging
  3. 操作失誤 • 更新到錯的 Kubernetes cluster • 可能會造成 downtime • 改到錯的

    Git branch • 因為 branch 無法合併,所以全靠 copy & paste • 沒有事前驗證 YAML 內容 • 即使有 PR review 依然很難避免錯誤發⽣
  4. 難以重複使⽤ • YAML anchor 能夠重複使⽤,但無法跨檔案引⽤ • 只能透過 ConfigMap 或 Secret

    來共⽤設定 • 只能以環境變數或 volume 的形式使⽤ • 但是設定以外的部分就無法重複使⽤了 • Sidecar • CronJob • 變數 (e.g., domain, image registry) • Component (e.g., PostgreSQL, Redis)
  5. Helm 👍 優點 • 有很多現成的 chart • 版本管理 • 透過指定不同的

    values,即可 佈署到不同的環境 • 內建 Linter 和測試⼯具 👎 缺點 • ⽤ template 寫 YAML 很⿇煩 • ⽐較複雜,需要花時間上⼿ • Helm 佈署的⽅式不同,從 YAML 轉移到 Helm ⽐較困難
  6. Kustomize 👍 優點 • 內建在 kubectl,不須額外安 裝其他⼯具 • 使⽤ YAML

    學習成本低 • 可以 Patch 任意資源 • 利⽤ Overlay 可以佈署到不同 環境 👎 缺點 • 沒有內建驗證功能 • YAML 不易重複使⽤ • 有些情況不⽅便 Patch
  7. apiVersion: apps/v1 kind: Deployment metadata: name: example spec: replicas: 1

    selector: matchLabels: app: example template: metadata: labels: app: example spec: containers: - name: nginx image: nginx Base apiVersion: apps/v1 kind: Deployment metadata: name: example spec: replicas: 3 Patch apiVersion: apps/v1 kind: Deployment metadata: name: example spec: replicas: 3 selector: matchLabels: app: example template: metadata: labels: app: example spec: containers: - name: nginx image: nginx Result
  8. Ksonnet 👍 優點 • Jsonnet ⽐較容易重複利⽤ • 內建驗證功能 • ⽀援多環境

    👎 缺點 • Jsonnet 需要另外花時間學習, 且編輯器⽀援較差 • 概念⽐較複雜,需要花時間上 ⼿ • 已停⽌維護💀(2019/2),後繼 專案:kubecfg, Tanka
  9. Why JavaScript? • 學習成本低 • Dcard 後端主要使⽤ Go 和 JavaScript

    • JavaScript ⽐起 Go 更適合⽤來寫 config • 容易重複利⽤ • 可以直接重複使⽤變數和函數 • 現成資源多 • 有很多現成的 library 可以直接⽤ • 編輯器整合極佳
  10. import { Pod } from "kubernetes-models/v1/Pod"; API Version Kind 更多範例

    import { Deployment } from "kubernetes- models/apps/v1/Deployment"; import { Certificate } from "@kubernetes- models/cert-manager/cert-manager.io/v1/Certificate";
  11. Example: Pod import { Pod } from "kubernetes-models/v1/Pod"; const pod

    = new Pod({ metadata: { name: "busybox" }, spec: { containers: [ { name: "busybox", image: "busybox", command: ["sleep", "10000"] } ] } }); export default pod; apiVersion: v1 kind: Pod metadata: name: busybox spec: containers: - name: busybox image: busybox command: - sleep - '10000'
  12. Example: Deployment + Service import { Deployment } from "kubernetes-models/apps/v1/Deployment";

    import { Service } from "kubernetes-models/v1/Service"; const deployment = new Deployment({ metadata: { name: "nginx" }, spec: { // ... } }); const service = new Service({ metadata: { name: "nginx" }, spec: { // ... } }); export default [deployment, service]; --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx --- apiVersion: v1 kind: Service metadata: name: nginx
  13. Example: Secret import { Secret } from "kubernetes-models/v1/Secret"; const secret

    = new Secret({ metadata: { name: "api-key" }, type: "Opaque", data: { secret: Buffer.from("confidential").toString("base64") } }); export default secret; apiVersion: v1 kind: Secret metadata: name: api-key type: Opaque data: secret: Y29uZmlkZW50aWFs
  14. Example: YAML import { loadFile, loadUrl, loadString } from "@kosko/yaml";

    const manifest = loadFile("manifest.yaml"); const certManager = loadUrl( "https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert- manager.yaml" ); const pod = loadString(` apiVersion: v1 kind: Pod metadata: name: my-pod `); export default [manifest, certManager, pod];
  15. Example: Helm Chart import { loadChart } from "@kosko/helm"; const

    prometheus = loadChart({ chart: "prometheus", repo: "https://prometheus-community.github.io/helm-charts", version: "15.0.1", name: "prom-demo", namespace: "prom", values: { server: { ingress: { enabled: true } } } }); export default prometheus;
  16. Example: Kustomize import { loadKustomize } from "@kosko/kustomize"; // Current

    folder const manifest = loadKustomize({ path: __dirname }); // GitHub repo const promOperator = loadKustomize({ path: "github.com/prometheus-operator/prometheus-operator" }) export default [manifest, promOperator];
  17. Environment • 存放針對不同 Kubernetes cluster 設定的環境變數 • Global variables •

    在多個 component 裡共享的環境變數 • 例如:image registry, domain name, namespace • Component variables • 僅在單⼀ component 裡使⽤的環境變數 • 例如:API key, database name, replicas
  18. export default { namespace: "dev" }; Global vars export default

    { replicas: 1 }; Component vars import env from "@kosko/env"; // { namespace: "dev", replicas: 1 } const params = env.component("nginx"); const deployment = new Deployment({ metadata: { name: "nginx", namespace: params.namespace }, spec: { replicas: params.replicas, template: { spec: { containers: [{ name: "nginx", image: "nginx" }] } } } }); Component apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: dev spec: replicas: 1 template: spec: containers: - name: nginx image: nginx $ kosko generate --env dev
  19. export default { namespace: "prod" }; Global vars export default

    { replicas: 15 }; Component vars import env from "@kosko/env"; // { namespace: "prod", replicas: 15 } const params = env.component("nginx"); const deployment = new Deployment({ metadata: { name: "nginx", namespace: params.namespace }, spec: { replicas: params.replicas, template: { spec: { containers: [{ name: "nginx", image: "nginx" }] } } } }); Component apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: prod spec: replicas: 15 template: spec: containers: - name: nginx image: nginx $ kosko generate --env prod
  20. 資料驗證 • 使⽤ Kubernetes OpenAPI Spec 產⽣ TypeScript declaration 和

    JSON schema • ⽀援 Kubernetes 內建資源和第三⽅ CRD • https://github.com/tommy351/kubernetes-models-ts
  21. 轉移到 Kosko • ⼤約花了⼀個⽉多 (2018/12~2019/1) • Kosko 內建 migrate 指令,能夠把

    YAML 轉換成 JavaScript • 花了⼀個⽉把變數抽到 environment • ⽐較轉換前後的 YAML,只要相同的話就能保證相容 • 實際環境測試
  22. Environment • 只把必要的變數放到 environment • 降低維護成本 • 避免 environment 過於冗⻑

    • 改變資料夾結構 👉 • 根據 component 來區分更易讀 Group by type Group by component
  23. 佈署 • CI 上只驗證內容是否正確 • 在本機佈署到 Kubernetes • ⽬前還是使⽤ kubectl

    apply • 寫了⼀個 script 來避免佈署到錯的 cluster • 重要 component 在佈署前會額外檢查版本狀態
  24. Example: Selector const labels = { app: "demo" }; const

    deployment = new Deployment({ spec: { selector: { matchLabels: labels }, template: { metadata: { labels } } } }); const service = new Service({ spec: { selector: labels } });
  25. Example: Port const httpPort = 80; const deployment = new

    Deployment({ spec: { template: { spec: { containers: [{ ports: [{ containerPort: httpPort }] }] } } } }); const service = new Service({ spec: { ports: [{ port: httpPort }] } });
  26. Example: Environment Variables export function envVars(envs: Record<string, string>): IEnvVar[] {

    return Object.entries(envs).map(([name, value]) => { return { name, value }; }); } // Before [ { name: "LOG_LEVEL", value: "info" }, { name: "API_URL", value: "https://example.com" } ]; // After envVars({ LOG_LEVEL: "info", API_URL: "https://example.com" }); • 避免 name 重複 • 寫起來⽐較簡短
  27. Example: Sidecar function withEnvoy(spec: IPodSpec): IPodSpec { return { ...spec,

    containers: [...spec.containers, { name: "envoy", image: "envoyproxy/envoy:v1.17.0" }] }; } const deployment = new Deployment({ spec: { selector: {}, template: { spec: withEnvoy({ containers: [{ name: "demo", image: "demo" }] }) } } }); apiVersion: apps/v1 kind: Deployment spec: selector: {} template: spec: containers: - name: demo image: demo - name: envoy image: envoyproxy/envoy:v1.17.0
  28. Example: Component function createDatabase(name: string) { return [ new Deployment({

    metadata: { name } }), new Service({ metadata: { name } }), new PersistentVolumeClaim({ metadata: { name } }) ]; } // components/post-api.js export default [ createDatabase("post-api-db"), new Deployment({ metadata: { name: "post-api" } }), new Service({ metadata: { name: "post-api" } }) ]; // components/user-api.js export default [ createDatabase("user-api-db"), new Deployment({ metadata: { name: "user-api" } }), new Service({ metadata: { name: "user-api" } }) ];