Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes on AWS - EKS Deep Dive - AWS Summit ...

Kubernetes on AWS - EKS Deep Dive - AWS Summit Berlin 2018

Elastic Container Services for Kubernetes (EKS). Presentation from the AWS Summit 2018 in Berlin.

Christoph Kassen

June 06, 2018
Tweet

More Decks by Christoph Kassen

Other Decks in Technology

Transcript

  1. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Tiffany Jernigan @tiffanyfayj Developer Advocate, AWS Christoph Kassen @christoph_k Solutions Architect, AWS kubernetes on aws
  2. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Containers Packaging Distribution Immutable infrastructure
  3. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  4. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Open source container management platform Helps you run containers at scale Gives you primitives for building modern applications © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is kubernetes?
  5. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J SCALE PERFORMANCE BREADTH A single extensible API
  6. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Vibrant and growing community of users and contributors © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  7. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J ON-PREMISES CLOUD © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes can be run anywhere!
  8. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Cloud-native applications MICROSERVICE TOOLING NATIVE APPLICATIONS
  9. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J “Run Kubernetes for me.” “Native AWS integrations” “An open source Kubernetes experience.”
  10. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. ELASTIC CONTAINER SERVICE FOR KUBERNETES GA yesterday 6/5! @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  11. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Amazon Container Services
  12. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J EKS is Kubernetes Certified
  13. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Open Source Kubernetes Community Kubernetes https://github.com/kubernetes/kuber netes CNI plugin https://github.com/aws/amazon-vpc- cni-k8s Heptio AWS Authenticator https://github.com/heptio/authentica tor Virtual Kubelet https://github.com/virtual- kubelet/virtual-kubelet/ SIG AWS https://github.com/kubernetes/comm unity/tree/master/sig-aws Cloud Provider Working Group https://github.com/kubernetes/comm unity/tree/master/wg-cloud-provider External-DNS https://github.com/kubernetes- incubator/external-dns CoreOS ALB Ingress https://github.com/coreos/alb- ingress-controller CODE REVIEWS FIXING BUGS IMPLEMENTING NEW FEATURES
  14. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J EKS - Customers Create EKS cluster Provision worker nodes Launch add-ons Launch workloads
  15. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J EKS - Kubernetes Control Plane Create cluster Create HA Control Plane IAM integration Certificate Management Setup LB
  16. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J mycluster.eks.amazonaws.com Availability Zone 1 Availability Zone 2 Availability Zone 3 Kubectl Workers
  17. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. EKS Architecture @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  18. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J EC2 Worker Nodes EKS Control Plane Customer VPC EKS VPC Network Load Balancer ENI API Access Kubectl Exec/Logs TLS Static IPs ENI Attachment Autoscaling Group EKS Architecture
  19. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. IAM Authentication
  20. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Kubectl 3) Authorizes AWS Identity with RBAC K8s API 1) Passes AWS Identity 2) Verifies AWS Identity 4) K8s action allowed/denied AWS Auth IAM Authentication + kubectl https://github.com/heptiolabs/kubernetes-aws-authenticator
  21. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. EKS Worker Nodes
  22. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J kubectl AWS Auth config map & RBAC Workers Role Role config map Worker provisioning
  23. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Metrics Nodes Node exporter Pod/Container Kube-state-metrics cAdvisor Application /metrics JMX Cluster-wide Aggregator Prometheus, Heapster Visualizer Grafana, Kibana, Dashboard Data Model InfluxDB, Graphite Alerting AlertManager, Kapacitor
  24. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Networking © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  25. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Native VPC networking with CNI plugin Pods have the same VPC address inside the pod as on the VPC Simple, secure networking Open source and on Github … { } https://github.com/aws/amazon-vpc-cni-k8s
  26. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Nginx Pod Java Pod ENI Veth IP: 172.16.1.147 Veth IP: 172.16.1.224 Nginx Pod Java Pod ENI Veth IP: 172.16.1.38 Veth IP: 172.16.1.24 ec2.associateaddress() VPC Subnet – 172.16.1.0/24 Instance 1 Instance 2 Primary Private IP: 172.16.1.118 Secondary IPs: 172.16.1.147, 172.16.1.224, … Primary Private IP: 172.16.1.15 Secondary IPs: 172.16.1.38, 172.16.1.24, … 172.16.0.0/16
  27. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J How do I configure network security with EKS? © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  28. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Kubernetes Network Policies enforce network security rules Calico is the leading implementation of the network policy API Open source, active development (>100 contributors) Commercial support available from Tigera https://www.projectcalico.org/
  29. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J STAGE SEPARATION “TENANT” SEPARATION FINE-GRAINED FIREWALLS COMPLIANCE Namespaces – without network policy, they are not network isolated Reduce attack surface within microservice-based applications Isolate dev, test, and prod E.g., PCI, HIPAA
  30. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J What version of Kubernetes does EKS support? 1.10.3 currently © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  31. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Kubernetes Autoscaling with Amazon EKS
  32. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Auto Scaling Two options AWS AutoScaling k8s Cluster Auto Scaler Cluster Autoscaler Reactive Aware of Pod / Cluster state Utilizes AWS AutoScaling AWS AutoScaling Scaling on CloudWatch Metrics Cluster Horizontal Pod Autoscaler Scales pods in response to k8s generated metrics (CPU) Pods
  33. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J ❯ helm search mysql > helm search mysql NAME CHART VERSION APP VERSION DESCRIPTION stable/mysql 0.6.0 5.7.14 Fast, reliable, sc stable/prometheus-mysql-exporter 0.1.0 v0.10.0 A Helm chart for p stable/percona 0.3.2 5.7.17 free, fully compat ... ❯ helm install install stable/mysql [displays README + information about deployment] ❯ helm list NAME REVISION UPDATED STATUS CHART NAMESPACE nobby-cow 1 Wed Jun 6 12:54:00 2018 DEPLOYED mysql-0.6.0 default Package manager that allows you to bundle up deployment resources and publish them https://github.com/kubernetes/helm
  34. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Hosting Helm repositories • Anywhere that serves HTTP can host a helm repo • Host private Helm Repo with Chartmuseum https://github.com/kubernetes-helm/chartmuseum • There’s also a handy plugin for S3! • This means IAM Role = auth for your repo • https://github.com/hypnoglow/helm-s3
  35. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Deploying Helm on EKS Helm 2.9+ works with EKS RBAC permissions required kubectl -n kube-system create serviceaccount tiller kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller helm init --service-account tiller
  36. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  37. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Know-how & Tools @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  38. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Load Balancing
  39. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Nginx Pods EC2 instances kube-proxy :32002 nginx-service :32001 Internet 10001:8080 10002:8080 10003:8080 Request to NGINX Pod {NLB}:443 NLB NLB Forwards to the node {node:32001} Service Type – LoadBalancer (NLB) k8s service ClusterIP receives request kube-proxy load balances to pods
  40. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Network Load Balancer apiVersion: v1 kind: Service metadata: name: nginx namespace: default labels: app: nginx annotations: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" spec: type: LoadBalancer externalTrafficPolicy: Local ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: app: nginx More options: • Draining • Logging • SSL Certs • Tagging • Security groups • Health checks https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws /aws.go
  41. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Nginx Pods EC2 instances kube-proxy :32001 nginx-service :32003 Internet Request to NGINX Pod {ALB}:443 ALB ALB Routes based on the path. /api /home 10002:8080 Webapp Pods 10002:8080 Installation: https://github.com/pahud/eks-alb-ingress Ingress Type – CoreOS ALB Ingress kube-proxy :32002 webapp-service :32004 Load Balances to pods Proxies request to the k8s service ClusterIP
  42. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J DNS
  43. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Automatic Route53 DNS creation for services apiVersion: v1 kind: Service metadata: name: nginx annotations: # Uses https://github.com/kubernetes-incubator/external-dns external-dns.alpha.kubernetes.io/hostname: nginx.highlyavailable.systems. spec: type: LoadBalancer ports: - port: 80 name: http targetPort: 80 selector: app: nginx
  44. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Automatic Route53 DNS creation for Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: nginx.highlyavailable.systems http: paths: - backend: serviceName: nginx servicePort: 80
  45. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Scheduling
  46. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Controlling scheduling Resource requirements Resource filters
  47. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Limit resource usage Container A Container B limit request 600m 600m limit request 800m 400m ⎲ ⎳ Pod CPU and memory resources
  48. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Resource Quotas apiVersion: v1 kind: Pod metadata: name: production spec: containers: - name: nginx-pod image: nginx resources: limits: memory: "800Mi" cpu: "800m" # 0.8 vCPU requests: memory: "600Mi" cpu: "400m“ # 0.4 vCPU Applied per Namespace apiVersion: v1 kind: ResourceQuota metadata: name: production spec: hard: requests.cpu: "1" requests.memory: 1Gi limits.cpu: "2" limits.memory: 2Gi ResourceQuota defined both, so Pod must define both Pod Resource Request
  49. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Controlling scheduling Resource requirements Constraints • Taints Node-level • Tolerations Pod-level Topology filters
  50. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Taints and Tolerations # Taint node $ kubectl taint nodes ip-10-0-32-12.us-west-2.compute.internal \ skynet=false:NoSchedule # Tolerations kind: Pod spec: tolerations: - key: skynet operator: Equal value: “false” effect: NoSchedule [...] Match taint to schedule onto tainted node
  51. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Controlling scheduling Resource requirements Constraints • Taints Node-level • Tolerations Pod-level Affinity/Anti-Affinity Topology filters
  52. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Affinity / Anti-Affinity • Control scheduling onto nodes ◦ Combine with Taints & Tolerations • Distribute Pods across cluster affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "beta.kubernetes.io/instance-type" operator: In values: [“r4.large",“r4.xlarge"]
  53. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Deployment Strategies
  54. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Rolling Update apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app labels: app: my-app spec: replicas: 10 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 # Numeric or percentage based value maxUnavailable: 0 [...]
  55. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Blue / Green Deployment apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app-blue labels: app: my-app spec: replicas: 3 template: metadata: labels: app: my-app version: blue [...] Blue apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app-green labels: app: my-app spec: replicas: 3 template: metadata: labels: app: my-app version: green [...] Green
  56. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Blue / Green Deployment Blue kind: Service metadata: name: my-app labels: app: my-app spec: type: LoadBalancer ports: - name: http port: 80 targetPort: http selector: app: my-app version: blue kind: Service metadata: name: my-app labels: app: my-app spec: type: NodePort ports: - name: http port: 80 targetPort: http selector: app: my-app version: green Green kubectl patch service my-app -p '{"spec":{"selector":{"version":"green"}}}'
  57. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Canary Deployment Production apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app-prod labels: app: my-app spec: replicas: 9 template: metadata: labels: app: my-app spec: containers: - name: my-app image: images/container:v1 [...] apiVersion: extensions/v1beta1 kind: Deployment metadata: name: my-app-canary labels: app: my-app spec: replicas: 1 template: metadata: labels: app: my-app spec: containers: - name: my-app image: images/container:v2 [...] More examples at https://container-solutions.com/kubernetes-deployment-strategies/ Canary
  58. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Network Policies
  59. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Network Policy kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: web-allow-prod spec: podSelector: matchLabels: app: web ingress: - from: - namespaceSelector: matchLabels: purpose: production Select affected Pods Define traffic that is allowed
  60. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Want to learn more?
  61. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Tooling and Ecosystem https://github.com/ramitsurana/awesome-kubernetes https://discuss.kubernetes.io/ http://slack.k8s.io/ TGIK Playlist: https://www.youtube.com/playlist?list=PLvmPtYZtoXOENHJiA Qc6HmV2jmuexKfrJ
  62. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J EKS – Getting started https://aws.amazon.com/eks https://aws.amazon.com/getting-started/projects/deploy- kubernetes-app-amazon-eks/ https://aws.amazon.com/blogs/aws/amazon-eks-now- generally-available/ https://aws.amazon.com/blogs/compute/ https://aws.amazon.com/blogs/opensource/category/comput e/amazon-elastic-container-service-for-kubernetes/ https://medium.com/containers-on-aws
  63. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Questions? @ C H R I S T O P H _ K @ T I F F A N Y F A Y J
  64. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. @ C H R I S T O P H _ K @ T I F F A N Y F A Y J Please complete the session survey in the summit mobile app.
  65. © 2018, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. https://aws.amazon.com/containers @christoph_k @tiffanyfayj Special thanks to: Paul Maddox, Abby Fuller, Nishi Davidson, Brandon Chavis, Arun Gupta, Chris Hein, Omar Lari, and many more... Thank You @ C H R I S T O P H _ K @ T I F F A N Y F A Y J