Let's start Cloud Native with Rancher !!

#rancherjp #cndjp

Transcript

1. Let’s start Cloud Native with Rancher !! © 2019 cyberblack28 #rancherjp #cndjp
2. Profile Twitter : cyberblack28 Blog : https://cyberblack28.hatenablog.com/ Job Educational Solution Architect Developer Advocate/Technical Evangelist Infrastructure Engineer Frontend Engineer !! Music,Live,FES !! Take a picture !! https://www.flickr.com/photos/cyberblack ISBN-10: 4798155373 ISBN-13: 978-4798155371 I wrote the Chapter 6 Rancher 2.0 part. Release on March 15,2018 #rancherjp #kujiraya #deepcn
3. Company We’re hireling !! Docker Kubernetes Rancher Cloud Native
4. KubeCon + CloudNativeCon China 2018 Recap http://u0u0.net/PGHs Rancher Yes !!
5. KubeCon + CloudNativeCon NA 2018 Recap http://ur0.link/PkTG Vitess MySQL Operator http://ur0.link/PkUm
6. Agenda 1. Market trend of Kubernetes 2. Introduction of Kubernetes 3. Introduction of Rancher 4. Kubernetes & Rancher Handson 5. Rancher 2.2 Preview 6. Information
7. Market trend of Kubernetes
8. Market trend of Kubernetes Kubernetes<17I $(&H?
   CNCF. ,)"-.AEG/25 by Publickey 2018.3.8 Incubation Stage2 /,"02GJ
   ( /.2' D6;= 
   H <3?!-&% L
   97 >  CA   
   *) 2#!-&'+1$# 8B: 
   5F E
   CNCFTechnical Operating CommitteeTOC@IK4 A G/ 
   Kubernetes
   @70
   ;8D
   C4 B:=>!+,'%F9 
    63*#"-.
9. Market trend of Kubernetes AWSKubernetes
    
   Amazon EKS by Publickey 2018.6.7 • Amazon EKS(11" AWS re:Invent 2017 $ • "! ( &!)  "%! ()  '#*)
   201812
   GA
10. Market trend of Kubernetes Azure Kubernetes Service (AKS) GA – New regions, more features, increased productivity by Microsoft Azure Blog 2018.6.13 • Microsoft Build 2018 Azure Container Service Azure Kubernetes Service ./ • "
    61"
    $'"
    $'"2
    ( 2  ,&30#)*! 10  4+5% -
11. Market trend of Kubernetes    Kubernetes Google Cloud
    Google Kubernetes Engine on-PremGoogle Cloud Next ’18 by Publickey 2018.7.25 •  #"GKE(Google Kubernetes Engine)$ Kubernetes#" &
    • Google Cloud Console Kubernetes! • Google Cloud Platform MarketplaceKubernetes  % 
12. Market trend of Kubernetes Google
       Serverless containers Google Cloud Next ’18 by Publickey 2018.7.25 • Google Kubernetes Engine( )2
    $ &
    %
    +-  'GKE serverless add-on*, • '!
    
    $ #
    "'Kubernetes0/ +- KnativeGKE serverless add-on  .1*,
13. Market trend of Kubernetes GitHub Goes All in on Kubernetes by THENEWSTACK 2017.8.16 SRESite Reliability Engineering   
14. Introduction of Kubernetes
15. What’s kubernetes ? kubernetesGoogle14/"$ #.,/ "*Borg / (  52 OSS/"$ #.,/!-  B3'&" '&" '&" A CDk8skubernetes 8<@ 7=CNCF(Cloud Native Computing Foundation)?6  :8;0 >9 )+%"52   
16. What’s kubernetes ? • "
    
    " • !
    " 
     •
    
    " • 0*-1 • "
    #&43 • 
       • !
    "" • ! 2% • &%     )5 • $
    )5 • /6,"#)5 Kubernetes! '.+ $(
17. Kubernetes Architecture Master API UI CLI Node 3 Node 2 Node 1 Kubernetes
    F=:K MasterNode 1+(1&  • Master /#$;I> ?
    #"-01*1&AB
    /#$854G7D  • Node )0!.1 1%'LH3<  1%'/1$,LH
    Master@C> 1%'E9J62D  Node n Image Registry
18. Kubernetes Architecture API UI CLI API Server Scheduler Controller-Manager etcd Master 
      API Server kubernetesAPI B<69A
    REST,CRUD,FIFL5  Scheduler PodNode(,3 $ #.14   Controller-Manager 0$&?W !4-+4)
    Node[TNQMP
    !4'*2,1 "/4X
    1%$7O U@:   etcd 0$&=ECJZ (&$) V!4-+4)API Server>K 
    etcd;S ECMP 
    Y8H 9GRD  
19. Kubernetes Architecture Node 
      Kubelet NodeLG Pod&#%247"JM 
    YAML.5B  Container RuntimeRD  Pod9?
    P:
    NQ8  Container Runtime kubeletRD
    1%>C$7),;88 Kubernetes$7),37'0> OIE 
    Docker
    containerd
    rkt
    cri-o$7),37'0@<  Kube-proxy FNode=D 
    Service /%!+A Cluster IPHK  -(+6!Proxy *.5 +iptables@<  Master Kube-proxy Container Runtime Kubelet Node 1,2,3,n Pod Image Registry Pod Pod Pod Pod Pod Pod Pod Pod
20. Kubernetes Architecture Pod • 1 >3 ,!*21=6 • Pod/ ,."7 IP)5 • Pod/ ,.Node- ?< • Pod/ ,90 & (5 Container Volume Pod Container Container Pod
    #'%$ +981:4- ,(! IP
     ※;  Container 3:  
21. Kubernetes Architecture Pod Node2 Container Pod Container Pod Node   
    Node1 Node1 Container Pod Container  localhost IP
     Container Pod Node2 IP
    
22. Kubernetes Architecture Deployment & ReplicaSet Master Node 1,2,3,n Deployment .<%( replicas:3 Controllers • ReplicationController D RS Kube proxy Container Runtime Kubelet Pod1 Pod2 Pod3 Controller-Manager D RS Deployment ReplicaSet 
    DeploymentReplicaSet84
    A/
    CB?1   
    ReplicaSet79 
    Pod)&'52> ,
    .<%()&'5(replicas)0:   
    Pod- *'+ !&"#
    *( $!,  ReplicaSet
    .< 5Pod=;3 @6   Image Registry
23. Kubernetes Architecture Service Client Node Service Pod Pod Pod
      ClusterIP /"$7 Pod5C9>  -/.'IP(2" ExternalIP @6 NodeIP:Port ? '/+%4&)EG
    =89> NodePort :Node IP:Port ? '/+%4&)EG
    =89> LoadBalancer LB;D #"FBService
    L43(*/4 <B endpoint Pod Pod Node Node LoadBalancer Service/"$7=0"'Pod#" I1&4DNS<BAH , !' 
24. Kubernetes Architecture Ingress Ingress
    *( HTTP(S)  '(Service "$ ""+& ,% ")Kubernetes! Service#4(TCP/UDP over IP)
    Ingress#7(HTTP(S))   Service Pod Internet Pod Pod Ingress endpoint
25. Kubernetes Architecture Others PersistentVolumeClaim,PersistentVolume,StrageClass,Namespace,Label,Selector,ConfigMap,Secret% kubernetes  "!"  )&($ #'
26. Introduction of Rancher
27. What’s Rancher ? • ,"
    0&.)80#@? =B
    :6 0#>5&,! %* • '!$" 0# >5
    &-+0 &/
    !.+0@ ? 17=B49 • 32 Rancher Labshttp://rancher.com/ • &0%!(!< ;AC 
28. Difference between Rancher 1.6 and 2.1 
     
     • *14docker-compose-7 +, '!* • Network 
    ipsec,vxlan# • Storage 
    NFS,EBS# • Architecture 
    Docker" • Java Go5
    MySQL60 • $( )%,# • Architecture 
    kubernetes" • 100%Go5
    etcd60 •  )%,
    100% kubernetes • *14Helm-7 +,'!* • Network
    flannel,canal,Calico9
    kubernetes!& ,:.-73/ • Storage
    kubernetes2084  :.-73/
29. Rancher v2.1.0 • Rancher1$ Cattle Orchestrator Rancher2$ Kubernetes 
     • Windows Kubernetes #
     Rancher 2.1 !Windows Node Kubernetes " http://ur0.link/PjA3 Main outline • Latest - v2.1.5 - rancher/rancher:latest • Stable - v2.1.5 - rancher/rancher:stable Version Rancher Release - v2.1.0 http://urx3.nu/MJA0 Details
30. Feature of Rancher2.1 Kubernetes Clusters Create Import  
       Bare Metal Create,Import,Manage Kubernetes Clusters !! Manage
31. Feature of Rancher2.1 One Click Deployment ! !  #-$,
    (%! 
    !    Creating Private Catalogs ! ! )" '&  +*  
32. Feature of Rancher2.1 Major developer catalogs
33. Rancher2.1 Technical Architecture Cluster Controller Auth Proxy Rancher Server RKE Nodes Kubectl Kubernetes API Rancher UI CLI API Rancher Component Other Kubernetes Master Legend Rancher API Server etcd Cluster Agent 1 Cluster Agent 2 RKE Kubernetes Master GKE Nodes GKE Kubernetes Master Kubelet Kubelet 
      a x v x tiy Rd l o b e N H r c b e Re d c Dk AKC e d r b n b I Hus oC Gg eP e d v v S h
34. Rancher2.1 Deep Dive Documents https://www.slideshare.net/linecorp/lets-unbox-rancher-20-v200
35. Rancher Labs New Project “Rio” Kubernetes, Istio, Service Mesh like Docker Darren Shepherd Co-founder and Chief Architect at Rancher Labs, Inc. @ibuildthecloud https://github.com/rancher/rio https://www.slideshare.net/cyberblackvoom/whats-rio-112779732 Rio
    !!
36. Get started with Rancher 2.1 OS Hardware Software Ubuntu 16.04 (64-bit) Memory: 4GB • 1.12.6 • 1.13.1 • 17.03.2 Red Hat Enterprise Linux 7.5 (64-bit) RancherOS 1.3.0 (64-bit) Step 2: Start the server $ sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher Web Access!! https://<server_ip> Step 1: Prepare a Linux Host
37. Kubernetes & Rancher Handson
38. Kubernetes & Rancher Handson 1. Rancher Server 2. Kubernetes Cluster 3. weavescope   4. RancherUIkubectl  5. Guestbook
      rancher-server rancher-host Guestbook Guestbook Rancher Server weavescope Agent Server Qiita URL http://u0u1.net/PnyX
39. Kubernetes & Rancher Handson 1.Rancher Server 1.Instance2
      4/ rancher-server )'+ asia-northeast1,2 + asia-northeast1-b &+% vCPUx1 $"! Ubuntu 16.04 LTS !80GB #* HTTP"(# 87
    HTTPS"(# 87
    6.  -  GCP(Google Cloud Platform)30 #* ** 187/5    4/ rancher-host )'+ asia-northeast1,2 + asia-northeast1-b &+% vCPUx1 $"! Ubuntu 16.04 LTS !80GB #* HTTP"(# 87
    HTTPS"(# 87
    6.  -  1 2
40. Kubernetes & Rancher Handson 2.Docker 2:rancher-server,rancher-host $ curl https://releases.rancher.com/install-docker/17.03.2.sh | sh 3.Rancher2.0 Server (1:rancher-server ) $ sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher:v2.2.0-alpha3 4.  
    RancherUI rancher-server https://EXTERNAL-IP/ 5. 
41. 6.Helm
     
    Catalogs  Enable Global  Kubernetes & Rancher Handson
42. Kubernetes & Rancher Handson 1.
    Add Cluster 2.Kubernetes Cluster
43. Kubernetes & Rancher Handson 2.CUSTOM 
    Cluster Name 
    Next E C POF , , 2 E C POF SG E C POF G A K N D C D , , , , D C
44. Kubernetes & Rancher Handson 3.
    Show advanced options
45. Kubernetes & Rancher Handson 4. Node Role etcd Control Plane Node Address Public Address Internal Address 2 (rancher-host) IP Copy to Clipboard 2 (rancher-host) - Done 1 2 3 5 4
46. Kubernetes & Rancher Handson 
      
47. Kubernetes & Rancher Handson 
      
48. Kubernetes & Rancher Handson 1.Default
    Catalog Apps 3.weavescope    2.Launch 
49. Kubernetes & Rancher Handson 3.
    weave-scope
    View Details 
50. Kubernetes & Rancher Handson 4.
    Launch
51. Kubernetes & Rancher Handson 5. 
     Workloads
52. Kubernetes & Rancher Handson 6.
    Load Balancing
53. Kubernetes & Rancher Handson 7.
    Add Ingress
54. Kubernetes & Rancher Handson 9.Name 
    Namespaceweave-scope 
    Targetweave-scope- frontend-weave-scope 
    Port4040 
    Save  1 2 3 4 5
55. Kubernetes & Rancher Handson 10.
    URL
56. Kubernetes & Rancher Handson 11.weavescope
    
57. Kubernetes & Rancher Handson 1. 
    rancher-kubernetes-cluster 4.RancherUIkubectl   RancherUI kubectl   
    2.Launch kubectl 
58. Kubernetes & Rancher Handson 3.
     
59. Kubernetes & Rancher Handson 3.
    kubectl get nodes > kubectl get nodes NAME STATUS ROLES AGE VERSION rancher-host Ready controlplane,etcd,worker 39m v1.10.5 > kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} 4.
    kubectl version
60. Kubernetes & Rancher Handson 5.Guestbook 2 .1 Guestbook  yaml  
      https://raw.githubusercontent.com/kubernetes/examples/master/guestbook/all-in-one/guestbook-all-in-one.yaml 2 .1 > kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook/all-in-one/guestbook-all-in-one.yaml service "redis-master" created deployment "redis-master" created service "redis-slave" created deployment "redis-slave" created service "frontend" created deployment "frontend" created > kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE frontend ClusterIP 10.43.52.144 <none> 80/TCP 1m kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 47m redis-master ClusterIP 10.43.31.61 <none> 6379/TCP 1m redis-slave ClusterIP 10.43.55.29 <none> 6379/TCP 1m
61. Kubernetes & Rancher Handson 3.RancherUIIngress  default
    Add Ingress   2 1
62. Kubernetes & Rancher Handson 4.Name
    Targetfrontend
    Port80
    Save   1 2 3 4
63. Kubernetes & Rancher Handson 5.
    URL
64. Kubernetes & Rancher Handson 6.Guestbook
     
65. Kubernetes & Rancher Handson 7.weavescopeGuestbook 
     
66. Kubernetes & Rancher Handson 1guestbook-all-in-one.yaml apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis tier: backend role: master spec: ports: - port: 6379 targetPort: 6379 selector: app: redis tier: backend role: master --- apiVersion: apps/v1 # for k8s versions before 1.9.0 use apps/v1beta2 and before 1.8.0 use extensions/v1beta1 kind: Deployment metadata: name: redis-master spec: selector: matchLabels: app: redis role: master tier: backend replicas: 1 template: metadata: labels: app: redis role: master tier: backend spec: containers: - name: master image: k8s.gcr.io/redis:e2e # or just image: redis resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 6379 --- redis-master
67. Kubernetes & Rancher Handson 1guestbook-all-in-one.yaml redis-slave apiVersion: v1 kind: Service metadata: name: redis-slave labels: app: redis tier: backend role: slave spec: ports: - port: 6379 selector: app: redis tier: backend role: slave --- apiVersion: apps/v1 # for k8s versions before 1.9.0 use apps/v1beta2 and before 1.8.0 use extensions/v1beta1 kind: Deployment metadata: name: redis-slave spec: selector: matchLabels: app: redis role: slave tier: backend replicas: 2 template: metadata: labels: app: redis role: slave tier: backend spec: containers: - name: slave image: gcr.io/google_samples/gb-redisslave:v1 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns # If your cluster config does not include a dns service, then to # instead access an environment variable to find the master # service's host, comment out the 'value: dns' line above, and # uncomment the line below: # value: env ports: - containerPort: 6379 ---
68. Kubernetes & Rancher Handson 1guestbook-all-in-one.yaml frontend apiVersion: v1 kind: Service metadata: name: frontend labels: app: guestbook tier: frontend spec: # if your cluster supports it, uncomment the following to automatically create # an external load-balanced IP for the frontend service. # type: LoadBalancer ports: - port: 80 selector: app: guestbook tier: frontend --- apiVersion: apps/v1 # for k8s versions before 1.9.0 use apps/v1beta2 and before 1.8.0 use extensions/v1beta1 kind: Deployment metadata: name: frontend spec: selector: matchLabels: app: guestbook tier: frontend replicas: 3 template: metadata: labels: app: guestbook tier: frontend spec: containers: - name: php-redis image: gcr.io/google-samples/gb-frontend:v4 resources: requests: cpu: 100m memory: 100Mi env: - name: GET_HOSTS_FROM value: dns # If your cluster config does not include a dns service, then to # instead access environment variables to find service host # info, comment out the 'value: dns' line above, and uncomment the # line below: # value: env ports: - containerPort: 80
69. Kubernetes & Rancher Handson 2Guestbook  
70. Rancher 2.2 Preview
71. Rancher 2.2 Preview Rancher’s New Mluti-Tenant Prometheus Support
72. Rancher 2.2 Preview Prometheus
    GoogleBorgmon!5'-! 8PullKHTTP1,3#'HI &'*0KubernetesLBA9
    CN Service Discovery@E
    HI:J6? ;7    Grafana
    Prometheus>MHI+( 9<GI=)4 Web/2"%$2.= +(3 4 (!0FD  Rancher’s New Mluti-Tenant Prometheus Support
73. Rancher 2.2 Preview  
     PrometheusGrafana   Deploy from Catalog function
74. Rancher 2.2 Preview http://ur0.link/PjFf Rancher Official Blog
75. Rancher 2.2 Preview Rancher 2.2 simplifies and cuts this process down to just 2 steps
76. Rancher 2.2 Preview Rancher Official Blog http://ur0.link/PjHe
77. 1. 
    Tools-Monitoring Prometheus Rancher 2.2 Preview
78. 2.Prometheus  
    Save   1 2 Rancher 2.2 Preview
79. 3.
     Prometheus + Grafana
      Rancher 2.2 Preview
80. 4.
     
         
    Rancher 2.2 Preview
81. 5.Grafana   Grafana 
     Rancher 2.2 Preview
82. Nodes Cluster 6.
    Launch kubectl Rancher 2.2 Preview
83. 7.kubectl get pods --all-namespaces
     # Run kubectl commands inside here # e.g. kubectl get all > kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE cattle-prometheus exporter-kube-state-cluster-monitoring-779b96cf75-gg272 1/1 Running 0 20m cattle-prometheus exporter-node-cluster-monitoring-ljhch 1/1 Running 0 20m cattle-prometheus grafana-cluster-monitoring-5b4bc74756-hx7mx 3/3 Running 0 20m cattle-prometheus prometheus-cluster-monitoring-0 4/4 Running 1 20m cattle-prometheus prometheus-cluster-monitoring-nginx-5ff785dd4d-d6l8l 1/1 Running 3 20m cattle-prometheus prometheus-operator-system-monitor-77df6db6df-gdmmz 1/1 Running 0 20m cattle-system cattle-cluster-agent-77498f79c-5w78s 1/1 Running 0 30m cattle-system cattle-node-agent-s8gnz 1/1 Running 0 30m ingress-nginx default-http-backend-5bdd9fdd69-2ntmw 1/1 Running 0 30m ingress-nginx nginx-ingress-controller-xdwnk 1/1 Running 0 30m kube-system canal-wzw4r 3/3 Running 0 30m kube-system kube-dns-autoscaler-689f6f9756-zqn6f 1/1 Running 0 30m kube-system kube-dns-ddddcfcc8-xbpwn 3/3 Running 0 30m kube-system metrics-server-5444cf6dfc-g7xvp 1/1 Running 0 30m kube-system rke-ingress-controller-deploy-job-bshtx 0/1 Completed 0 30m kube-system rke-kubedns-addon-deploy-job-k9vxg 0/1 Completed 0 30m kube-system rke-metrics-addon-deploy-job-hmr7k 0/1 Completed 0 30m kube-system rke-network-plugin-deploy-job-tjqhn 0/1 Completed 0 31m Rancher 2.2 Preview
84. http://ur0.link/PjJn • December 2018 Online Meetup Introducing Rancher's New Multi Tenant Prometheus Support Webinar • Rancher v2.2.0-Alpha3 Multi-Tenant Prometheus Support
     Docs http://ur0.link/PjK6 Rancher 2.2 Preview
85. Information
86. Information Docker  $;&:   ISBN-10: 4844339621 ISBN-13: 978-4844339625 Docker,"
      439#0)<?  81765(. 2019/2/18+-;'* Windows
    $;&:   ISBN-10: 4295000957 ISBN-13: 978-4295000952 Windows   = / >%!"2/: 
87. Information c : ISBN-10: 4798153222 ISBN-13: 978-4798153223 L L L Docker D ――Linux c : ISBN-10: 4774176540 ISBN-13: 978-4774176543 LD e LD
88. Information kubernetes ?V@: &8$9*62; ISBN-10: 4873118409 ISBN-13: 978-4873118406 MU
    Kubernetes Up & Running<=QXXAG !
    EM#I  <=F HS "!  / D ?V@: [W@ ISBN-10: 4798155373 ISBN-13: 978-4798155371 Google,RedHat,IBM, $;,1-/,Rancher#>J(;.0TC &'+/ :)7;#O! !NL 48-/3%5#YR
    DZP BK 
89. Information The Kubernetes Book ,: Independently published ISBN-10: 1521823634 ISBN-13: 978-1521823637 kubernetes )$(+
    *% "'  Kubernetes in Action ,: Manning Pubns Co ISBN-10: 1617293725 ISBN-13: 978-1617293726 "'#-!  
     *&%  (+ 
90. Information Docker/Kubernetes ! +8,:5674, ISBN-10: 4297100339 ISBN-13: 978-4297100339 Docker/Kubernetes1:
    )!"%'&).-2/ (9;   )!".-#$(* 30       https://cyberblack28.hatenablog.com/
91. Information Kubernetes
     8L::7+5 ISBN-10: 4295004804 ISBN-13: 978-4295004806 C1M KubernetesFB  C2M KubernetesIH =KP C3M API3 kubectl C4M Workloads3 C5M Discovery & LB3 C6M Config & Storage3 C7M Cluster3 Metadata3 C8M 3G? $37 C9M ,4!" 7#& 2*4 C10M .7#&7 (% JD C11M ;< NO 137 C12M 13# C13M -'*$ Q@>9+7*$ C14M /' 37 C15M 7#&6 EA C16M CI/CDIH C17M -6) ServiceMesh C18M Kubernetes #!0 C19M Kubernetes 
92. Information  
    Kubernetes Azure
       :!:=<! ISBN-10: 4798157848 ISBN-13: 978-4798157849 1  ,1; 
    Kubernetes ,2; Kubernetes20/9 ,3; Kubernetes &  2  ,4; Kubernetes(+ ,5; 
    
    # ,6; 
     3  ,7;
    *-'7 ,8; .%)Availability ,9; 61)Scalability ,10; $3)Manageability ,11; 
    "4Security ,12; .58)Observability
93. Information Certified Kubernetes Administrator(CKA) ,/,. a C v kp l gh Y uc W ( kp l gh $$YZ W 1 no Ytcba Z rsN kp l gh 1 moei w U o l g 9CC 032EE      9CC FFF 6 4C: : 6 4 ) Certified Kubernetes Application Developer(CKAD)
94. Information Kubernetes&Docker KD100  https://www.mirantis.co.jp/training/ Kubernetes&Docker KD200 MIRANTIS kubernetes  
95. Information
     https://www.katacoda.com/ Web # !   Docker,Kubernetes,Istio,Prometheus%
    &"$  
96. Information
     https://labs.play-with-docker.com/ Play with Docker https://labs.play-with-k8s.com/ Play with Kubernetes
97. Information  Kubernetes https://thinkit.co.jp/author/13290 cndjp   https://speakerdeck.com/hhiroshell  
98. Information
           http://ur0.work/NQqi 
99. ! !