Kubernetes on prem - Cluster rollout with kubespray

Slides zum Vortrag

Transcript

1. Kubernetes on prem Cluster rollout with kubespray GEDOPLAN GmbH Dirk Weil
2. Dirk Weil Studied Informatik at RWTH Aachen Living in Bielefeld CEO of GEDOPLAN GmbH (www.gedoplan.de) JEE since 1999 Speaker and author 2 gedoplan.de Kubernetes on prem - Cluster rollout with kubespray
3. Objective Host a Kubernetes cluster on-prem Why? Test environments, playgrounds Learning Small, cheap (?) prod environments How? Bare Metal on Linux Hosts Load Balancer (MetalLB) Ingress Controller (Ingress-Nginx, cert-manager) Cluster Storage (Ceph) Kubernetes on prem - Cluster rollout with kubespray 3 gedoplan.de
4. Options kubeadm k3s … kubespray Kubernetes on prem - Cluster rollout with kubespray 4 gedoplan.de
5. Prearrangement Hosts with basic Linux installation e.g. Debian 11 ssh Server User with SSH key and sudo right Additional unused partition Kubernetes on prem - Cluster rollout with kubespray 5 gedoplan.de k8s-141 k8s-142 k8s-143
6. Prearrangement Workstation with Container environment (e. g. Docker Desktop) Alternative: Local Ansible installation https://www.ansible.com/ open source IT automation engine utilizes so-called playbooks (~Scripts) Kubernetes on prem - Cluster rollout with kubespray 6 gedoplan.de k8s-141 k8s-142 k8s-143
7. kubespray Ansible playbook for K8s rollout https://github.com/kubernetes-sigs/kubespray quay.io/kubespray/kubespray:v2.25.0 Topology configuration in so-called inventory Template in kubespray/inventory/sample Kubernetes on prem - Cluster rollout with kubespray 7 gedoplan.de Hosts and their duties Properties for various functionalities
8. kubespray Kubernetes on prem - Cluster rollout with kubespray 8 gedoplan.de all: hosts: k8s-141: ansible_host: k8s-141.gedoplan.intra k8s-142: ansible_host: k8s-142.gedoplan.intra k8s-143: ansible_host: k8s-143.gedoplan.intra children: kube_control_plane: hosts: k8s-141: kube_node: hosts: k8s-142: k8s-143: etcd: hosts: k8s-141: k8s-142: k8s-143: k8s_cluster: children: kube_control_plane: kube_node: calico_rr: hosts: {} hosts.yaml (.yml/.ini) Hosts Control Plane Worker Storage
9. kubespray Rollout K8s cluster Kubernetes on prem - Cluster rollout with kubespray 9 gedoplan.de /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ docker run --rm -it \ -v "$(pwd)"/inventory:/kubespray/inventory \ -v ~/.ssh/id_rsa:/root/.ssh/id_rsa \ quay.io/kubespray/kubespray:v2.25.0 \ ansible-playbook –u your_user_id -i inventory/hosts.yaml -b -K cluster.yml BECOME password: PLAY [Check Ansible version] ************************************************************* Thursday 23 May 2024 14:36:00 +0200 (0:00:00.041) 0:00:00.041 ********** TASK [Check 2.15.5 <= Ansible version <= 2.17.0] ***************************************** Ok: [localhost] => { “changed”: false, “msg”: “All assertions passed” } Thursday 23 May 2024 14:36:00 +0200 (0:00:00.089) 0:00:00.130 ********** … PLAY RECAP ******************************************************************************* k8s-141 : ok=736 changed=140 unreachable=0 failed=0 … k8s-142 : ok=578 changed=110 unreachable=0 failed=0 … k8s-143 : ok=578 changed=110 unreachable=0 failed=0 … localhost : ok=3 changed=0 unreachable=0 failed=0 … /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ docker run --rm -it \ -v "$(pwd)"/inventory:/kubespray/inventory \ -v ~/.ssh/id_rsa:/root/.ssh/id_rsa \ quay.io/kubespray/kubespray:v2.25.0 \ ansible-playbook –u your_user_id -i inventory/hosts.yaml -b -K cluster.yml BECOME password: PLAY [Check Ansible version] ************************************************************* Thursday 23 May 2024 14:36:00 +0200 (0:00:00.041) 0:00:00.041 ********** TASK [Check 2.15.5 <= Ansible version <= 2.17.0] ***************************************** Ok: [localhost] => { “changed”: false, “msg”: “All assertions passed” } Thursday 23 May 2024 14:36:00 +0200 (0:00:00.089) 0:00:00.130 ********** … PLAY RECAP ******************************************************************************* k8s-141 : ok=736 changed=140 unreachable=0 failed=0 … k8s-142 : ok=578 changed=110 unreachable=0 failed=0 … k8s-143 : ok=578 changed=110 unreachable=0 failed=0 … localhost : ok=3 changed=0 unreachable=0 failed=0 …
10. Load Balancer: MetalLB Open Source Load Balancer Implementation https://metallb.universe.tf/ Layer 2 Mode (ARP / NDP) Kubernetes on prem - Cluster rollout with kubespray 10 gedoplan.de k8s- 141 k8s- 142 k8s- 143 ARP Request „who has 192.168.10.140?“ ARP Response „me! I am 6c:c2:17:6d:32:b5“
11. Load Balancer: MetalLB kubespray group_vars: Kubernetes on prem - Cluster rollout with kubespray 11 gedoplan.de … metallb_enabled: true metallb_speaker_enabled: "{{ metallb_enabled }}" metallb_namespace: "metallb-system" metallb_protocol: "layer2" metallb_config: address_pools: primary: ip_range: - 192.168.10.140/32 auto_assign: true layer2: - primary … group_vars/k8s_cluster/addons.yml … kube_proxy_strict_arp: true … group_vars/k8s_cluster/k8s-cluster.yml Use strict ARP in kube-proxy (answer ARP requests for active service endpoints only) Activate MetalLB in layer 2 mode
12. Load Balancer: MetalLB Components include: Controller (Deployment, Service) IP (de)allocation to LoadBalancer services Speaker (DaemonSet) Respond to ARP requests (incl. leader election and failover) Address pool Level 2 (ARP) advertisement Kubernetes on prem - Cluster rollout with kubespray 12 gedoplan.de apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: primary namespace: metallb-system spec: addresses: - 192.168.10.140/32 autoAssign: true apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: primary namespace: metallb-system spec: ipAddressPools: - primary
13. Ingress Controller: Ingress-Nginx Multiplex (and loadbalance) web requests Hostname, path → background service Configuration by K8s ingresses Kubernetes on prem - Cluster rollout with kubespray 13 gedoplan.de Ingress-Nginx Service foo Service bar same IP
14. Ingress Controller: Ingress-Nginx kubespray group_vars: DaemonSet and Service bound to MetalLB allocated IP Kubernetes on prem - Cluster rollout with kubespray 14 gedoplan.de … ingress_nginx_enabled: true ingress_nginx_service_type: LoadBalancer ingress_nginx_namespace: "ingress-nginx" ingress_nginx_insecure_port: 80 ingress_nginx_secure_port: 443 ingress_nginx_class: nginx ingress_nginx_default: true … group_vars/k8s_cluster/addons.yml Activate Ingress-Nginx and let MetalLB assign IP
15. Certificate Management: cert-manager Auto-create TLS certificates Includes usage in ingresses Certificate issuers for various CAs Lets Encrypt Private CA … https://cert-manager.io Kubernetes on prem - Cluster rollout with kubespray 15 gedoplan.de
16. Certificate Management: cert-manager kubespray group_vars: Issuer for private CA Kubernetes on prem - Cluster rollout with kubespray 16 gedoplan.de … cert_manager_enabled: true cert_manager_namespace: "cert-manager" … group_vars/k8s_cluster/addons.yml Activate cert-manager apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: demo-internal spec: ca: secretName: demo-internal-ca apiVersion: v1 kind: Secret metadata: name: demo-internal-ca type: Opaque stringData: tls.crt: | -----BEGIN CERTIFICATE----- MIIDxTCCAq2gAwIBAgIBADANBgkqhkiG9w0 … -----END CERTIFICATE----- tls.key: | -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKc … -----END PRIVATE KEY----- Self-signed cert for CA
17. Certificate Management: cert-manager Ingress using cert-manager certificates: Kubernetes on prem - Cluster rollout with kubespray 17 gedoplan.de apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: whoami annotations: cert-manager.io/cluster-issuer: demo-internal spec: rules: - host: whoami.demo.gedoplan.demo http: paths: - pathType: Prefix path: "/" backend: service: name: whoami port: number: 80 tls: - hosts: - whoami.demo.gedoplan.demo secretName: whoami-demo-gedoplan-demo-tls cert-manager issuer Secret will be created by cert-manager
18. Certificate Management: cert-manager Let‘s Encrypt issuer Kubernetes on prem - Cluster rollout with kubespray 18 gedoplan.de apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@gedoplan.de privateKeySecretRef: name: letsencrypt solvers: - http01: ingress: ingressClassName: nginx apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: whoami annotations: cert-manager.io/cluster-issuer: letsencrypt … ACME http01 challenge (alternative: dns01) Select issuer in ingress
19. Cluster Storage: Ceph distributed storage system file, block and object storage Simple installation: Rook (https://rook.io) automates deployment and management of Ceph https://github.com/rook/rook.git Kubernetes on prem - Cluster rollout with kubespray 19 gedoplan.de Files used for demo
20. Cluster Storage: Ceph Deploy the Rook Operator Kubernetes on prem - Cluster rollout with kubespray 20 gedoplan.de /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ kubectl apply -k 1-operator namespace/rook-ceph created customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created … clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created configmap/rook-ceph-operator-config created deployment.apps/rook-ceph-operator created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE rook-ceph-operator-757cdc49bb-pswkm 1/1 Running 0 6m26s /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ kubectl apply -k 1-operator namespace/rook-ceph created customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created … clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created configmap/rook-ceph-operator-config created deployment.apps/rook-ceph-operator created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE rook-ceph-operator-757cdc49bb-pswkm 1/1 Running 0 6m26s wait for it
21. Cluster Storage: Ceph Create Ceph Cluster Devices must be empty (wipefs -a …) Ceph config directory must not exist (/var/lib/rook) Kubernetes on prem - Cluster rollout with kubespray 21 gedoplan.de /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ kubectl apply -k 2-cluster cephcluster.ceph.rook.io/rook-ceph created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE csi-cephfsplugin-6s4rx 2/2 Running 0 11m csi-cephfsplugin-provisioner-7f4d578c49-9rzv9 5/5 Running 0 11m … rook-ceph-osd-0-c544b7d7d-gh2wb 2/2 Running 0 9m58s rook-ceph-osd-1-649fdbf45c-87m5x 2/2 Running 0 9m57s rook-ceph-osd-prepare-k8s-142-qxgl4 0/1 Completed 0 9m30s rook-ceph-osd-prepare-k8s-143-5bglq 0/1 Completed 0 9m27s /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ kubectl apply -k 2-cluster cephcluster.ceph.rook.io/rook-ceph created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE csi-cephfsplugin-6s4rx 2/2 Running 0 11m csi-cephfsplugin-provisioner-7f4d578c49-9rzv9 5/5 Running 0 11m … rook-ceph-osd-0-c544b7d7d-gh2wb 2/2 Running 0 9m58s rook-ceph-osd-1-649fdbf45c-87m5x 2/2 Running 0 9m57s rook-ceph-osd-prepare-k8s-142-qxgl4 0/1 Completed 0 9m30s rook-ceph-osd-prepare-k8s-143-5bglq 0/1 Completed 0 9m27s wait for it … storage: useAllNodes: false useAllDevices: false nodes: - name: k8s-142 devices: - name: sda2 - name: k8s-143 devices: - name: sda3
22. Cluster Storage: Ceph Create Storage Class Block: block storage to be consumed by a pod (RWO) Shared Filesystem: filesystem to be shared by pods (RWX) Object: S3 compatible object store Kubernetes on prem - Cluster rollout with kubespray 22 gedoplan.de … annotations: "storageclass.kubernetes.io/is-default-class": "true" Demo: block storage facultative /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ kubectl apply -k 3-storageclass storageclass.storage.k8s.io/rook-ceph-block created cephblockpool.ceph.rook.io/replicapool created /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray ❯ kubectl apply -k 3-storageclass storageclass.storage.k8s.io/rook-ceph-block created cephblockpool.ceph.rook.io/replicapool created
23. Cluster Storage: Ceph Publish Ceph Dashboard Kubernetes on prem - Cluster rollout with kubespray 23 gedoplan.de apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: rook-ceph-mgr-dashboard namespace: rook-ceph annotations: cert-manager.io/cluster-issuer: demo-internal nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" nginx.ingress.kubernetes.io/server-snippet: | proxy_ssl_verify off; spec: rules: - host: ceph.demo.gedoplan.demo http: paths: - path: / pathType: Prefix backend: service: name: rook-ceph-mgr-dashboard port: name: https-dashboard tls: - hosts: - ceph.demo.gedoplan.demo secretName: ceph-demo-gedoplan-demo-tls tune to your needs
24. Cluster Storage: Ceph Consume storage Kubernetes on prem - Cluster rollout with kubespray 24 gedoplan.de apiVersion: apps/v1 kind: StatefulSet metadata: name: pg spec: serviceName: pg selector: matchLabels: name: pg volumeClaimTemplates: - metadata: name: pg-data spec: storageClassName: rook-ceph-block accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi template: … PVC using block storage (SC can be ommitted, if default)
25. More Demo project github.com/GEDOPLAN/k8s-on-prem-kubespray Slides speakerdeck.com/dirkweil gedoplan.de Trainings in Berlin, Bielefeld, Köln, inhouse Reviews, Coaching Development Teams Contact dirk.weil@gedoplan.de linkedin.com/in/dirk-weil-49940683 x.com/dirkweil bsky.app/profile/dirkweil.bsky.social 25 gedoplan.de Spring Boot vs. Quarkus