Introduction to Docker & Kubernetes @ JavaDays 2019

Transcript

1. Ondrej Sika Freelance DevOps Consultant & Lecturer ondrej@sika.io @ondrejsika Introduction to Docker & Kubernetes
2. About me My name is Ondrej Sika, I am an IT & DevOps consultant, architect and lecturer. I'm boosting effectivity & productivity of software development teams by using right tools and techniques which lead to faster development and reliable operation of software products. I help companies to set up whole DevOps pipeline using training, consulting and short term project work.
3. Agenda - DevOps - Docker - Kubernetes - Alternatives - Summary
4. DevOps
5. What is DevOps? DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market. Source: https://aws.amazon.com/devops/what-is-devops/
7. What does it mean? - "Agile Infrastructure" or "Agile Operations" - Rapid Delivery - Deliver changes automatically into production (staging, ...) - Reliability - People do mistakes, script don't. - Scaling - Easy scaling using Clouds, Kubernetes, Serverless, ... - Infrastructure as a Code - Treat your Infrastructure like a code (Terraform, ...) - Security - Security policy as a code
8. Rapid Delivery (CI/CD) - Continuous Integration - Integrate every - Continuous Delivery - Deliver changes automatically into production (staging, ...) - Tools for CI/CD: - Gitlab CI - Jenkins
9. Reliability - People make mistakes (especially under pressure), scripts don't - HA Infrastructure (no single point of failure) - Easy investigation using Git - Automatic rollbacks in case of fail after deployments
10. Infrastructure as a Code - Git Versioned - You can treat your infrastructure as a other code - merge requests, CI, ... - Automatic documentation - You can generate docs from the code - terraform graph -type=refresh | dot -Tsvg > infrastructure.svg - Simple Scaling - In infrastructure definition code - Auto scaling (Kubernetes, Auto Scaling Groups) - Reliable Upgrades - Review (merge requests) upgrades before applies - Rollbacks of infrastructure changes
11. Easy & Secure Scaling - Infrastructure as a Code - Scaling is easy and secure in Infrastructure as a Code - Terraform, Cloud Formation - Autoscaling - Applications in Kubernetes - Nodes of Clusters (AWS, Azure, …) - Auto Scaling Groups
13. Modern Open Source Tools for DevOps - SCM - Git - CI/CD - Gitlab CI / Jenkins - Container Engine - Docker - Orchestrator - Kubernetes, Swarm - Metrics & Monitoring - Prometheus - Logging - ELK, EFK - Provisioning - Ansible, Puppet - Infrastructure - Terraform
14. 12 Factor Apps - 12 rules how to write modern application - Rules are about: - Sustainable development & operation - Shipping your code (product) - Configuration - Scaling - Operations - Logs, Admin process, .. - Your Dev & Ops should read it - https://12factor.net/ Source: https://12factor.net/
15. Why Docker & Kubernetes?
16. Why Docker & Kubernetes? - Unify your environment - You need just Kubernetes Cluster (or machines with Docker) to run any application - Simple CI stack - Unified test, staging & production env - Solid role separation (but on shared codebase) - Devs: Dockerfile & Kubernetes manifest, ... - Ops: Kubernetes Clusters, Terraform manifests, ... - Bulk deployments & management - Treat your deployments like a cattle, not a pets - Deploy desired state - Declarative approach (instead of imperative)
17. Docker
18. What is Docker Docker is a set of platform-as-a-service products that use OS-level virtualization to deliver software in packages called containers. Source: https://en.wikipedia.org/wiki/Docker_(software)
19. What is a Container Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels. Source: https://en.wikipedia.org/wiki/Docker_(software)
20. Docker for Traditional Applications Make your application portable (able to deploy to new unified infrastructure) without touching code.
21. Docker for Traditional Applications - Be able to add your current application into DevOps pipeline - Be able to fast & easily deploy your current application to various unified environments (machines or clusters with Docker) - Make environment (libraries, dependencies, ...) as part of application (source code) - Deploy application with libraries & dependencies instead of installing dependencies on production servers. It's faster and more reliable approach. - Saves your productuction environment costs (resources) and minimize downtime
23. Docker for Microservices Docker is a simplest way how to build, ship & run microservices. In containers.
24. Docker for Microservices & DevOps - Simple integrations with various CI/CD tools - Fast, repeatable & cached builds - Simple application distribution throw Registry and Docker Trusted Registry - Be able to deploy several times per day - Defines simple interface for communication between containers and underlying layer (kubernetes or hardware)
25. Docker Quick Start
26. Install Docker Mac brew cask install docker Windows choco install docker-desktop Linux https://docs.docker.com/install/linux/docker-ce/debian/
27. System wide info docker version # print version docker info # system wide information docker system df # docker disk usage docker system prune # cleanup unused data
32. Run Docker Container Hello world docker run hello-world Simple web server docker run -p 80:80 ondrejsika/hellojavadays2019
35. Docker Image docker image ls # list all images docker image ls <image> # list all images docker image ls -q # quiet output, just IDs docker image rm <image> # remove image
40. Docker Run docker run [args..] <image> [<command>] # Eg.: docker run hello-world docker run debian cat /etc/os-release docker run ubuntu cat /etc/os-release docker run -ti debian
44. Common Docker Run Parameters --name <name> --rm - remove container after stop -d - run in detached mode -ti - map TTY a STDIN (for bash eg.) -e <variable>=<value> - set ENV variable
46. Work with Containers docker ps - list containers docker start <container> docker stop <container> docker restart <container> docker logs <container> - show STDOUT & STDERR docker rm <container> - remove container
49. Persistent Storage - Docker Volumes Volumes are persistent data storage for containers. Volumes can be shared between containers and data are written directly to host. docker run -ti -v my-volume:/data debian docker run -ti -v $(pwd)/my-data:/data debian
51. Port Forwarding Docker can forward specific port from container to host. docker run -p 80:80 ondrejsika/hellojavadays2019
53. Own Docker Images
54. Dockerfile Dockerfiles are used to produce docker images using reproducible builds. Dockerfiles defines each layer for Docker Image Overlay2 filesystem
55. .dockerignore Ignore unnecessary files for docker build process. Speed up the build. Same syntax as .gitignore
56. Build Docker Image docker build -t <image> <path> docker build -f <dockerfile> -t <image> <path>
57. Dockerfile FROM <image> - define base image RUN <command> - run command and save as layer COPY <local path> <image path> - copy file or directory to image ENV <variable> <value> - set ENV variable WORKDIR <path> - change working directory VOLUME <path> - define volume CMD <command> - executable which you want to start in container EXPOSE <port> - define port where container listen
58. Example Dockerfile FROM python:3.8-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . CMD ["python", "app.py"] EXPOSE 80
59. Build docker build -t ondrejsika/javadays2019-simple . docker push ondrejsika/javadays2019-simple
62. Multi-Stage Build
63. Multi-Stage Dockerfile FROM java-jdk:... as build RUN gradle assemble FROM java-jre:... COPY --from=build /build/demo.jar .
64. Example Multi-Stage Dockerfile FROM golang as build WORKDIR /build COPY app.go . ENV CGO_ENABLED=0 RUN go build -a -ldflags \ '-extldflags "-static"' app.go FROM scratch COPY --from=build /build/app . CMD ["/app"]
65. Build docker build -t ondrejsika/javadays2019-multi-stage . docker push ondrejsika/javadays2019-multi-stage
68. Docker BuildKit
69. Docker BuildKit Docker has new build tool called BuildKit which can speedup your builds. For example, it build multiple stages in parallel and more. You can also extend Dockerfile functionality for caches, mounts, … - https://docs.docker.com/develop/develop-images/build_enhancements/ - https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/ex perimental.md
70. BuildKit Dockerfile Example # syntax = docker/dockerfile:experimental FROM openjdk:jre RUN --mount=type=cache,target=/cache/.m2 \ --mount=type=cache,target=/cache/.gradle \ make
71. Docker Without Kubernetes If you run small application or just one server, you don't need Kubernetes. Take a look for: - Docker Compose - Docker Swarm
72. Demo Time
73. Kubernetes
74. What is Kubernetes? A Production-Grade Container Orchestration System
75. What is Kubernetes? Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. Source: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
76. What does Kubernetes do? Abstract away the underlying hardware - Remove concept of nodes - Manage your applications like cattle instead of like pets Deploy your desired state - You (admin) describe the desired state and kubernetes turn it into actual state
77. Kubernetes Users From small companies and startups to large enterprises
78. No vendor lock Kubernetes is no vendor lock to specific provider, you can run Kubernetes on: - AWS - GCP - DigitalOcean - Azure - OpenStack - or your private infrastructure
79. Why (and when) you should use Kubernetes - If you need HA - If you have to manage applications on many servers - If you don't want to care about servers (Kubernetes as a Service, IaaS) - If you want easily deploy your Dockerized applications (IaaS)
80. Which apps are suitable for Kubernetes? - Stateless workers - Batch processing - Web Servers - Mobile Backend Which not? - Databases - Persistent data storages
81. Core Concepts Pod - The basic and atomically schedulable building block of Kubernetes, which is a single instance of app. Pods are mortal. Deployment - Atomic update of Pods. Deployments contains Pod & ReplicaSet templates and keep running desired pods. Service - Provide immortal IP address or DNS name for some selected pods. Ingress - Provide external access to service using domain name. Storage, Configuration, Monitoring, ...
82. Kubernetes Cluster Components API Server - Stateless API server backed by distributed Etcd Controller Manager - ensure the actual state of the cluster equals the desired state Scheduler - Schedule creations of Pods on a Nodes Kubelet - Client for API Server, run Pods Kube Proxy - Forward traffic into cluster
84. Tools kubectl - Kubernetes client (for CLI) helm - Package manager for Kubernetes kubeadm - Tool for Kubernetes cluster setup (on VMs) minikube - Run Kubernetes locally for development kops - Create Kubernetes cluster in cloud
85. Kubernetes Cluster Components API Server - Stateless API server backed by distributed Etcd Controller Manager - ensure the actual state of the cluster equals the desired state Scheduler - Schedule creations of Pods on a Nodes Kubelet - Client for API Server, run Pods Kube Proxy - Forward traffic into cluster
86. Install Kubernetes Client Mac brew install kubernetes-cli Windows choco install kubernetes-cli Linux https://kubernetes.io/docs/tasks/tools/install-kubectl/
87. Install Helm Mac brew install kubernetes-helm Windows choco install kubernetes-helm Linux https://helm.sh/docs/install/
88. Setup Kubernetes Cluster - Manually using kubeadm - Using Ansible (Ansible use also kubeadm) - On the Cloud using kops (creates EC2 instances & setup cluster there) - Using Terraform or Cloud Formation
89. Create Kubernetes cluster using Terraform git clone git@github.com:ondrejsika/terraform-do-kubernetes-example.git cd terraform-do-kubernetes-example terraform init terraform apply -auto-approve terraform output kubeconfig > kubeconfig export KUBECONFIG=kubeconfig kubectl cluster-info kubectl get nodes
93. Kubernetes CLI - kubectl kubectl apply -f <file> kubectl get -f <file> kubectl get <resource> kubectl describe -f <file> kubectl delete -f <file>
94. Resources in Kubernetes
95. Resources in Kubernetes - Workload - Pods - Controllers - Deployments, StatefulSets, DaemonSers, Jobs, CronJobs - Service & Load Balancing - Services, Ingress - Storage - PersistentVolumes, PersistentVolumesClaims - Configuration - ConfigMaps, Secrets - RBAC - ServiceAccounts,Roles,RoleBindings
96. Pod - Minimal schedulable unit - Contains one (or more) containers running in one IPC & network namespace - Contains definition of Docker image, resource limits and other settings for containers - Pods are not used directly, we use controllers like Deployments, ... More: https://kubernetes.io/docs/concepts/workloads/pods/pod/
99. Deployment - Used to maintain some specific Pods up and running in N instances - Provide various deployment (upgrade) strategies - Allow us to rollback deployment More: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
102. StatefulSet - StatefulSet is the workload API object used to manage stateful applications. - Manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods. More: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
103. DaemonSet - A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. - As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Some typical uses of a DaemonSet are: - running a cluster storage daemon, such as glusterd, ceph, on each node. - running a logs collection daemon on every node, such as fluentd or logstash. More: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
104. Services - ClusterIP - Internal service to connect resources inside Kubernetes cluster - NodePort - Expose specific port on every node of cluster - Use ports from range 30000 - 32767 - LoadBalancer (cloud only) - Create new load balancer with new IP - Publish service on standart (defined) ports
109. Ingress - Ingress allows you expose services on domains and web paths - Easiest & cheapest way how to expose web services - Requires Ingress Controllers - Traefik - https://github.com/ondrejsika/kubernetes-ingress-traefik - Nginx + Cert Manager
111. Persistent Storage - EmptyDir - Simplest persistent storage - Chained to specific Pod (persistent only for that specific pod) - Stored on node - PersistentVolume (PV) - Storage which can be attached to pods - StorageClass (SC) - Dynamic provisioner of Persistent Volumes - PersistentVolumeClaim (PVC) - allow a user to consume abstract storage resources More: https://kubernetes.io/docs/concepts/storage/volumes/
116. ConfigMap & Secret - Store Configuration & Secrets for Pods & Kubernetes components
118. RBAC (Role Based Access Control) - ServiceAccount - User in Kubernetes - ClusterRole, Role - Define permissions in Kubernetes - ClusterRoleBinding, RoleBinding - Assigns Role to ServiceAccount
122. Helm
123. Helm - Package manager for Kubernetes helm repo add ondrejsika https://helm.oxs.cz helm install demo ondrejsika/one-image --set host=demo.k8s.sikademo.com NAME: demo LAST DEPLOYED: Tue Nov 12 11:46:09 2019 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: See: https://demo.k8s.sikademo.com
124. Demo Time
125. Summary
126. Summary - DevOps helps you with faster & reliable deployments - Docker helps you separate applications & unify your environment - Kubernetes remove concept of nodes and provide you one large pool of resources - Kubernetes deploy desired state - Docker & Kubernetes help you with microservice architecture - IaaS (Terraform) provide simple & reproducible infrastructure (even on private cloud)
127. Alternatives Docker - RKT - Containerd Kubernetes - Docker Swarm - OpenShit
128. Resources https://aws.amazon.com/devops/what-is-devops/ https://dev.to/ashokisaac/devops-in-3-sentences-17c4 https://devopsish.com/what-is-devops/ https://www.davidbegin.com/using-terraform-docs-to-automate-keeping-your-terraform-modules-doc umenting/ https://12factor.net/ https://www.youtube.com/watch?v=uMA7qqXIXBk https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ https://www.howtoforge.com/core-components-of-a-kubernetes-cluster/#the-kubeapiserver
129. sika.link/javadays2019
130. Thank you & Questions Ondrej Sika email: ondrej@sika.io www: https://ondrejsika.io twitter: @ondrejsika linkedin: /in/ondrejsika/ Slides: https://sika.link/javadays2019