) - CoreOS + Terraform! Issues: - Terraform 0.7 - Poor remote state support - Providers compiled within Terraform release - No or poor support for tls / templating / local providers ... - Hard to manage Etcd (lost cluster state a few times) - Hard to upgrade Kubernetes version (manual / slow)
… - Create cluster with join tokens - Manage TLS cert rotation - Secure (RBAC …) - But … status (q2 2017): - lacks support for HA - still alpha - kops: Kubernetes operations (upup) - Manage clusters… the k8s way - Self Hosting proposal - temp control-plane for bootstrap - core component of CoreOS/Tectonic
(orchestrated, self-updating clusters) - Enterprise Security (integrate with LDAP via dex) - Management Console for cluster maintenance and operations (etcd / control-plane) - Built-in Monitoring (prometheus for autoscaling vs heapster) - Open Cloud Services announced kubecon 2017 (i.e.: vault)
bundled Terraform patched by CoreOS - CoreOS added a lot of features to tf providers for Azure / AWS while building Tectonic (open source) - Tectonic vanilla mode (no Licenses, fully open source) - CoreOS + Ticketmaster engineers develop open source ALB Ingress controller - Terraform has grown a lot since 0.7 - Better state backend support - Better modules support - Better state management - Added support for Workspaces (environments) - ...
- System hosted kubelet Updated through node annotations - Periodic snapshotting for recovery - Depends on external DNS / LB - Note test coverage for version skew between control-plane components
you may consider) - Typhoon: Minimal and free Kubernetes distribution (Terraform) - Archon: Operator to manage Kubernetes clusters - bootkube-terraform: (part of typhoon) tf module to render bootkube assets with terraform - ...
create multiple clusters from same Tectonic install dir (lacked support for Terraform workspaces) - Takes time to study, understand and make changes (+ how to keep changes in line with upstream) - Some Licensed components were still bootstrapped in vanilla mode (required some clean up) - ...
- Installs packages / copies in assets - Sets up protokube (to manage etcd ~EBS volumes) - Sets up the OS for kubelet (renders /etc/kubernetes/manifests ) - kubelet requests PodCIDR from kube-controllermanager for CNI - kubelet configures and starts CRI with PodCIDR config - kubelet handles and reports workloads assigned by scheduler
kubernetes addons - Compile as separate binary: /go/src/k8s.io/kops# make channels - Apply addons from local or upstream channels ~/# channels apply channel -f beekeeper/addons.yaml NAME CURRENT UPDATE namespaces.honestbee.io 1.1.1 1.1.2 tiller.addons.k8s.io 2.7.0 2.7.2 Must specify --yes to update
cluster_group: a group of clusters. For example “staging clusters”, “svc clusters”, … A cluster_group represents shared infra (1 VPC, hosted stateful service endpoints (RDS / ElastiCache), … ) - utilities subnet - subnet used for Hosted Services such as RDS / ElastiCache / ...