Bare Metal k8s Cluster with CoreOS Matchbox

from our last k8s meetup on 22. March 2018

https://meetu.ps/e/DZP9q/mjfLb/a

Transcript

1. Bare Metal k8s Cluster with CoreOS Matchbox André Veelken - DevOps Engineer 22.03.2018
2. Cloud K8s vs. on-prem - majority of k8s clusters on AWS or other clouds - easy to set up, see e.g. kops tool - expensive - slow / noisy neighbors - easy scaling - ecosystem makes things easier
3. Cloud K8s vs. on-prem - fast & powerful - cheap in comparison - scaling? - DHCP? - OS images & config - Loadbalancing?
4. K8s on bare metal: choose your weapon kubeadm kubespray Ansible Puppet Matchbox (network booting)
5. Why our choice? - @chaosaffe, iPXE and
 packet.net - similar to datacenter - wanted: immutable 
 infrastructure on-prem - CoreOS Container Linux - without additional
 config management tools
6. What is Matchbox? github.com/coreos/matchbox - HTTP server - iPXE server - config templating engine (ignition) - static asset server - matches servers by labels (e.g. MAC address) 
 to Profiles with iPXE configs, 
 Container Linux configs
7. Advantages of Matchbox - YAML rendering for ignition and variable expansion - TLS auf gRPC - Validation - Hash signatures - tested, part of Tectonic
8. Cluster setup: Matchbox server - Matchbox server installation via package, Docker, rkt, … - secured via iptables
9. Cluster setup: Terraform (IAC tool) % terraform apply -auto-approve
10. Cluster setup: DNS - Terraform creates DNS records from 
 Terraform state at AWS Route53 for:
 
 - each control plane node - each worker node - api (kubectl endpoint) - all worker nodes
11. Cluster setup: OS image preparation - get-coreos script places images in /var/lib/matchbox/assets/coreos - structure: /var/lib/matchbox assets/ - free form, files, hashes groups/ - control plane, master groups ignition/ - config mgmt on Container Linux profiles/
12. Cluster setup: Matchbox profiles - defines OS image, boot params, ignition -
13. Cluster setup: bootkube - tool for launching self-hosted Kubernetes clusters - we create a k8s config on Container Linux with it - write IPs and MACs into
 bootkube-render.sh and execute it
14. Cluster setup: bootkube, rollout - bootkube renders assets into
 /var/lib/matchbox/assets - restart of control plane and worker nodes, they pull their assets
15. Cluster setup: bootkube, bootstrapping - on control plane node 1 - % systemctl start bootkube - % journalctl -f -u bootkube - certificates from bootkube in 
 /etc/kubernetes/secrets are needed
16. Cluster setup: done, woohoo! - use kubectl on a master to inspect cluster
17. Lessons learned - Matchbox server should be 
 at same location as cluster - use OEM version of Container Linux if needed, e.g. for packet - learned many lessons on k8s internals especially on control plane: etcd, manifests
18. - Typhoon k8s distro - uses Terraform for everything - unified workflow - write scripts for scale up & down
19. Sources - GitHub.com/coreos/matchbox - github.com/kubernetes- incubator/bootkube - typhoon.psdn.io/bare-metal/ - Artwork: github.com/ ashleymcnamara/gophers
20. - Kubernetes Community Slack #wg-onprem (former #sig-onprem) - packet.net Slack #k8s - in planning: k8s on baremetal informal gathering 
 at containerdays.io in June
21. Thank you! % terraform destroy -force