Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Honestbee does CI/CD on Kubernetes - Voxxed...

How Honestbee does CI/CD on Kubernetes - Voxxed Days Singapore 2018

recording of the talk here: https://youtu.be/YtqOoiF--Mw

Honestbee has been running Kubernetes in production since mid 2016. In this presentation we take a look at how Helm enabled us to manage our deployment configurations as code, and demonstrate how it can be used to power your continuous delivery (CI/CD) pipeline. We tackle the following examples and use cases:

- Continuous Delivery using Helm with Drone.io
- Secret management with VaultController
- Building and hosting our own chart repositories (and the iterations we did)
- Plugins to help manage values
- Chart best practices

Learn from our practical Kubernetes usage, what benefits Helm and Kubernetes provide us. What challenges you may face adopting these technologies in your company and best practices from our experience

vincentdesmet

June 01, 2018
Tweet

More Decks by vincentdesmet

Other Decks in Programming

Transcript

  1. $ whoami DevOps at Honestbee DevOps at Swatmobile @vincentdesmet /

    github.com/so0k Singapore-Kubernetes-User-Group Cloud-Native-Singapore
  2. Overview - Honestbee (~2 years): - Containers - Drone -

    Kubernetes - Helm - Vault - Swat (~3 weeks): - docker cloud + stacks → Kubernetes + Helm - Pluggable CD using Keel
  3. What are Containers? • Packages up software binaries & dependencies

    • Immutable & testable • Isolate software from each other • Portable across environments a self-contained process ref: kubernetes-comic
  4. Why Containers? image: ruby:2.1 services: - postgres stages: - Build

    - Test - Staging - Production ... source: GitLab CI Lightweight Reproducible builds
  5. Why Containers? $ make get-tags-prod getting apse1a tag: master-ebf3205b getting

    apse1b tag: master-ebf3205b $ make get-tags-staging getting apse1a tag: staging-98a2aa4f getting apse1b tag: staging-98a2aa4f Note: "latest" is not a version Versioned packages
  6. More than just packaging and Isolation - Scheduling - Resource

    Optimisation - Monitoring - Lifecycle and health - Auth{n,z} - Scaling - Discovery - … Source
  7. Version 1: sed kube-install: update-manifests kubectl create -f manifests/${SHORT_NAME}-deployment.tmp.yaml kube-update:

    update-manifests kubectl patch deployment ${SHORT_NAME} -p '{"spec":{"template":{"spec":{"containers":[{"name":"'"${SHORT_NAME}"'","image":"' "${IMAGE}"'"}]}}}}' update-manifests : @sed 's#\(image:\) .*#\1 $(IMAGE)#' manifests/${SHORT_NAME}-deployment.yaml > manifests/${SHORT_NAME}-deployment.tmp.yaml
  8. $ kubectl get deploy -l release=backend NAME DESIRED CURRENT UP-TO-DATE

    AVAILABLE backend-admin 16 16 16 16 backend-core 32 32 32 32 backend-core-bee 10 10 10 10 backend-karafka 1 1 1 1 backend-scheduler 1 1 1 1 backend-worker-all 8 8 8 8 backend-worker-critical 2 2 2 2 backend-worker-food-import 1 1 1 1 backend-worker-high 2 2 2 2 backend-worker-low 12 12 12 12 $ kubectl get hpa -l release=backend NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS backend-admin Deployment/backend-admin 9% / 70% 16 24 16 backend-core Deployment/backend-core 28% / 70% 32 48 32 backend-core-bee Deployment/backend-core-bee 20% / 80% 10 16 10 $ kubectl get scheduledscaler -l release=backend NAME AGE backend-admin 9d backend-core 9d backend-core-bee 9d
  9. apiVersion: apps/v1beta1 kind: Deployment metadata: labels: app: backend component: service-core

    name: backend-core spec: replicas: 32 ... template: ... spec: containers: - env: - name: NEW_RELIC_APP value: backend-core-apse1a;backend-core ... image: myregistry/ backend:master-ebf3205b volumeMounts: - mountPath: /app/config/application.yml name: backend-config subPath: application.yml volumes: - name: backend-config secret: items: - key: config apiVersion: apps/v1beta1 kind: Deployment metadata: labels: app: backend component: scheduler name: backend-scheduler spec: replicas: 1 ... template: ... spec: containers: - command: - bin/rake - resque:scheduler env: - name: NEW_RELIC_APP ... image: myregistry/ backend:master-ebf3205b volumeMounts: - mountPath: /app/config/application.yml name: backend-config subPath: application.yml volumes: - name: backend-config secret: items:
  10. Kubernetes Deployment challenges - Each application has multiple components -

    Every component has its own k8s resources How to: - Deploy, Manage, Edit and Update multiple k8s configurations - Deploy multiple k8s configurations as a single application - Parameterise and support multiple environments - Manage application releases: rollout, rollback, history, ... - ...
  11. Helm Package Manager for Kubernetes - Aims to provide Apt/Yum/Homebrew

    User Experience - Ensure collaboration - Shareable Packages Deployment Manager - Repeatable deployments - Manage multiple configurations - Update, Rollback and test application deployments
  12. Chart, Repositories, Releases - Chart: “Package”, “Bundle” - Repository: Package

    Repository - Release: Installed Chart (same chart can be installed multiple times)
  13. Helm Charts Render(Templates + Values) = Release Templates: - The

    Go Template language: {{ .foo | quote }} - Variables, simple control structures (looping, conditionals, ... ) - 50+ functions from Go/Sprig template libraries ...
  14. Easy to manage deployments $ helm install backend --values production.yaml

    $ helm history backend REVISION UPDATED STATUS CHART DESCRIPTION 110 Tue Apr 24 17:53 SUPERSEDED backend-2.0.7 Upgrade complete 111 Wed Apr 25 10:53 SUPERSEDED backend-2.0.7 Upgrade complete 112 Wed Apr 25 14:58 SUPERSEDED backend-2.0.7 Upgrade complete 113 Wed Apr 25 15:02 SUPERSEDED backend-2.0.7 Upgrade complete ... $ helm rollback backend 112
  15. Tip: Labels Crucial for: - Service Discovery - Graphing /

    Alerting - Troubleshooting Define standard set of labels include them for consistency ## _helpers.tpl {{- /* labels.standard prints the standard Helm labels. The standard labels are frequently used in metadata. */ -}} {{- define "labels.standard" }} app: {{ template "fullname" . }} chart: {{ template "chartref" . }} heritage: {{ .Release.Service | quote }} release: {{ .Release.Name | quote }} {{- end }} ## deploy.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: {{ template "backend.fullname" . }} labels: {{- include "labels.standard" . | indent 4 }} ...
  16. Tip: Powerful templating For example: iterate yaml maps using the

    range function {{- range $key, $value := .Values.map }} Key: $key Value: $value {{- end }} map: foo: "bar" baz: "qux" Key: foo Value: bar Key: baz Value: qux
  17. {{- range $api_name, $api := .Values.api.types }} {{- if gt

    $api.replicas 0.0 -}} apiVersion: apps/v1beta1 kind: Deployment metadata: name: {{ template "fullname" $ }}-{{ $api_name }} labels: {{ include "labels.standard" $ | indent 4 }} component: service-{{ $api_name }} spec: replicas: {{ $api.replicas }} ... --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: {{ template "fullname" $ }}-{{ $api_name }} labels: {{ include "labels.standard" $ | indent 4 }} component: {{ $api_name }}-service spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: {{ template "fullname" $ }}-{{ $api_name }} minReplicas: {{ $api.autoscaling.min }} maxReplicas: {{ $api.autoscaling.max }} targetCPUUtilizationPercentage: {{ $api.autoscaling.cpuTarget }} --- {{- with $api.autoscaling.steps }} apiVersion: "scaling.k8s.restdev.com/v1alpha1" kind: ScheduledScaler ... steps: {{ toYaml . | indent 2 }} {{- end }} --- {{ end -}} {{- end -}} api: types: admin: autoscaling: min: 4 max: 12 cpuTarget: 70 steps: - runat: '0 30 11 * * *' mode: range minReplicas: 8 # 11:30am: pre-lunch schedule maxReplicas: 12 ... resources: limits: cpu: 2 memory: 6Gi requests: cpu: 2 memory: 6Gi config: admin-panel-enabled: true url: foo.honestbee.com core-bee: autoscaling: min: 2 max: 16 cpuTarget: 80 steps: - runat: '0 00 08 * * *' mode: range minReplicas: 4 # 08:00am: day schedule maxReplicas: 16
  18. ## Tip: Use Helm Values to automate monitoring / alerts

    ## i.e. Terraform + https://coveo.github.io/gotemplate/ {{- $Values := yaml "production.yaml" }} {{- define "container_graphs" }} {{ $container := . }} graph { title = "Memory - Honestbee-{{ $container }}" viz = "timeseries" request { q = "avg:kubernetes.memory.requests { $helm_release, kube_container_name:{{ $container }} }" type = "line" ... } } {{- end }} resource "datadog_timeboard" "backend_production" { title = "Backend Timeboard (Terraform)" {{- range $api, $definition := $Values.api.types }} {{- template "container_graphs" $api }} {{- end }} template_variable { name = "helm_release" prefix = "helm_release" default = "backend-production" } } api: types: admin: autoscaling: min: 4 max: 12 cpuTarget: 70 steps: - runat: '0 30 11 * * *' mode: range minReplicas: 8 # 11:30am: pre-lunch schedule maxReplicas: 12 ... resources: limits: cpu: 2 memory: 6Gi requests: cpu: 2 memory: 6Gi config: admin-panel-enabled: true url: foo.honestbee.com core-bee: autoscaling: min: 2 max: 16 cpuTarget: 80 steps: - runat: '0 00 08 * * *' mode: range minReplicas: 4 # 08:00am: day schedule maxReplicas: 16
  19. $ gotemplate -P templates/timeboard.tf.template > timeboard.tf $ terraform apply datadog_timeboard.backend_production:

    Refreshing state... (ID: 102245) An execution plan has been generated and is shown Below. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: ~ datadog_timeboard.backend_production graph.0.request.0.q: "max:kubernetes.cpu..." => "avg:kubernetes.cpu..." Plan: 0 to add, 1 to change, 0 to destroy. Tip: Automate monitoring / alerts
  20. Tip: Helm Values Defines your application configuration per environment! -

    Should be source controlled or tracked - roboll/helmfile - skuid/helm-value-store - Should not contain secrets SECRETS
  21. Private Chart Repository Honestbee: Version 1 - Self-host: - Using

    s3 Bucket + VPC endpoint - CI/CD: - Package - Generate Index - Push to s3 articles/devops/2017-07/drone-helm- repository resource "aws_s3_bucket" "b" { bucket = "${var.bucket_name_prefix}.${var.domain_name}" policy = "${data.aws_iam_policy_document.s3-read.json}" website { index_document = "index.html" } cors_rule { allowed_headers = ["*"] allowed_methods = ["PUT", "POST"] allowed_origins = [ "https://${var.bucket_name_prefix}.${var.domain_name}", ] } } data "aws_iam_policy_document" "s3-read" { statement { sid = "Access-from-specific-VPC-only" effect = "Allow" actions = [ "s3:GetObject" ] resources = [ "arn:aws:s3:::${var.bucket_name_prefix}.${var.domain_name}/*", ] condition { test = "StringEquals" variable = "aws:sourceVpce" values = [ "${values(data.terraform_remote_state.kops-state.vpc_endpoint)}", ] } } }
  22. Private Chart Repository Honestbee Version 2: - Use ChartMuseum -

    CI/CD: - Package and push honestbee/drone-chartmuseum ChartMuseum is an open-source Helm Chart Repository written in Go (Golang), with support for cloud storage backends, including Google Cloud Storage and Amazon S3. Works as a valid Helm Chart Repository, and also provides an API for uploading new chart packages to storage etc.
  23. Secrets? Version 1 (k8s 1.7): Vault + VaultController - Hashicorp

    Vault - Supports k8s auth - roboll/kube-vault-controller - Add k8s custom resource: secretClaim kind: SecretClaim apiVersion: vaultproject.io/v1 metadata: name: {{ template "fullname" . }} labels: {{- include "labels.standard" . | indent 4 }} spec: type: Opaque path: "secret/{{ .Values.env }}/{{ .Release.Name }}" renew: 3900
  24. Swat May 7 → May 25 docker cloud + stack

    → kubernetes + helm routing-worker: autoredeploy: true deployment_strategy: every_node environment: - MASTER_URL=master - MASTER_SECRET=foo expose: - "3352" image: swat/routing-worker:staging links: - routing-master:master tags: - staging routing-master: autoredeploy: true environment: - VIRTUAL_HOST=http://bar.com,https://bar.com - ... services: routing-master: replicaCount: 1 image: repository: swat/routing-master tag: master pullPolicy: Always port: 3352 healthChecks: livenessPath: / readinessPath: / resources: limits: cpu: 500m memory: 500Mi ingress: enabled: true hosts: - "foo.com" environment: MASTER_URL: "master" services: routing-master: image: tag: staging resources: limits: cpu: 250m memory: 300Mi ingress: hosts: - "bar.com" default staging staging 1:1
  25. Swat May 7 → May 25 docker cloud + stack

    → kubernetes + helm - Automated haproxy config through VIRTUAL_HOST envvars - Automated Let's Encrypt Certificates - Manual DNS configuration - Challenges in Secret Management - zalando/kube-ingress-aws-controller - Automated AWS ALB configuration - Automated AWS Certificate Management - kubernetes/external-dns - Automated DNS configuration using AWS Route53 - mozilla/sops - Automated secret encrypt / decrypt using AWS KMS 1:1
  26. Swat: Secrets vNext (k8s 1.8+): Bank-vaults + vault-sidekick - banzaicloud/bank-vaults:

    - Automated unsealing - Automated Vault initialisation - UKHomeOffice/vault-sidekick - Vault audit by Service
  27. Swat - Pluggable CD May 25 → May 31 docker

    cloud + stack → helm + keel.sh - Automated deployment on image push - Git branch → Image tag (no sha) - Poll registry for image changes - Trigger Kubernetes rolling update using newer image digest through helm - Provides slack notifications (optional approval workflow): Re-use existing CI