Oracle Cloud Hangout Cafe - Cloud Native流のストレージ管理

Oracle Cloud Hangout Cafe（おちゃかふぇ）のセッションスライドです。
1. Kubernetesの世界におけるストレージ管理
2. Rook/Cephから見るCloud Native Storage
3. デモ(Rook/Ceph)

(セッションの録画)
https://youtu.be/n95_om_bOiI

(イベントページ)
https://ochacafe.connpass.com/event/178405/

Transcript

1. Oracle Corporation Japan July 30th, 2020 Takuya Niita Oracle Cloud Hangout Cafe – Season 3 #1 Cloud Native
2. Copyright © 2020, Oracle and/or its affiliates. All rights reserved Oracle Java Oracle Corporation 2
3. 3 Copyright © 2020, Oracle and/or its affiliates. All rights reserved 3 2 1 Rook/Cephから見るCloud Native Storage Kubernetesの世界におけるストレージ管理 Agenda デモ(Rook/Ceph)
4. 4 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • (@takuya_0301) • • SIer • • Oracle 1 3 • Cloud Native 1 • Kubernetes 1 • 4 CKA(Certified Kubernetes Administrator) •
5. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Kubernetes Oracle Corporation Japan July 30th, 2020 Takuya Niita
6. 6 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Cloud Native Storage • • • • API Self-Managed • • RBAC • • Kubernetes – Cloud Native Storage – Kubernetes 参考: StorageOS Vision for Cloud Native Storage for Today’s Modern IT https://storageos.com/storageos-vision-cloud-native-storage-todays-modern
7. Software Defined Storage 7 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Software Defined Storage(SDS) • SNIA (2015 ) • / (Self-Managed) • (API) • - • - ( ) • - Software Defined Storage (SDS) API( / / ) API( ) #1 #2 #3 https://www.snia.org/sites/default/files/SNIA_Software_Defined_Storage_%20White_Paper_v1.pdf
8. 8 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • • API • • / • Kubernetes – – Kubernetes Kubernetes
9. 9 Copyright © 2020, Oracle and/or its affiliates. All rights reserved “ ” • • • • “ ” • • Volumes Kubernetes – –
10. 10 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Volumes • Volume Pod • NFS/iSCSI • Volume Pod Kubernetes – Volumes – Storage #A Pod Volume Kubernetes ”/data” ”/test-pd” hostPath manifest Storage #A Pod Volume Kubernetes
11. 11 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Volumes Cloud Native Storage • • Kubernetes • StorageClass • PersistentVolume • PersistentVolumeClaim Kubernetes – –
12. 12 Copyright © 2020, Oracle and/or its affiliates. All rights reserved StorageClass(SC) • • Provisioner - Local Volume - - (Ceph ) - (iSCSI/NFS) • Provisioner - Rook-Ceph Rook-Ceph API ( ) - Provisioner Kubernetes – StorageClass – AWS EBS kubernetes.io/aws-ebs Ceph RBD kubernetes.io/rbd Local kubernetes.io/no-provisioner
13. 13 Copyright © 2020, Oracle and/or its affiliates. All rights reserved oracle.com/oci • Oracle Container Engine for Kubernetes(OKE) • OCI Block Volume oracle.com/oci-fss • OCI File Storage • manifest apply OCI Provisioner :https://kubernetes.io/docs/concepts/storage/storage-classes/
14. 14 Copyright © 2020, Oracle and/or its affiliates. All rights reserved PersistentVolume(PV) • • • - ReadWriteOnce - ReadWriteMany - ReadOnlyMany • StorageClass - StorageClassName - StorageClass ”oci” Kubernetes – PersistentVolume – StorageClass
15. 15 Copyright © 2020, Oracle and/or its affiliates. All rights reserved PersistentVolumeClaim(PVC) • • • - Persistent Volume(PV) • - Persistent Volume(PV) Kubernetes – PersistentVolumeClaim – StorageClass
16. 16 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • SC PVC / • • PV/PVC • / Kubernetes – SC/PV/PVC – Storage 5Gi PVC 2Gi PV 2Gi SC Pod Volume Storage 5Gi PVC 2Gi SC Pod Volume PV 2Gi
17. 17 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Kubernetes Kubernetes ( ) Kubernetes ( ) 3 • in-tree plugin • FlexVolume • CSI(Container Storage Interface) Kubernetes
18. 18 Copyright © 2020, Oracle and/or its affiliates. All rights reserved in-tree plugin( ) • Kubernetes • https://github.com/kubernetes/kubernete s/tree/master/pkg/volume • iSCSI Plugin • Kubernetes • Kubernetes • • Kubernetes v1.14 CSI ( ) • Kubernetes v1.17 Kubernetes – in-tree plugin –
19. 19 Copyright © 2020, Oracle and/or its affiliates. All rights reserved FlexVolume • out-of-tree plugin(Kubernetes ) • - root - /usr/libexec/kubernetes/kubelet- plugins/volume/exec/ - (API) • kubelet Volume / / • Flex Volume in-tree plugin • Kubernetes Kubernetes – FlexVolume – kubelet Storage Driver / Flex Volume Driver
20. 20 Copyright © 2020, Oracle and/or its affiliates. All rights reserved CSI(Container Storage Interface) • Kubernetes Mesos Docker Cloud Foundry API • • out-of-tree plugin • Kubernetes v1.9 v1.13 GA • CSI • in-tree plugin CSI( ) Kubernetes – CSI – https://github.com/kubernetes/community/blob/master/contributors/design- proposals/storage/container-storage-interface.md#recommended-mechanism-for- deploying-csi-drivers-on-kubernetes • ( ) • Kubernetes
21. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Rook/Ceph Cloud Native Storage Oracle Corporation Japan July 30th, 2020 Takuya Niita
22. 22 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Rook • https://rook.io/ • Cloud Native Storage • Cloud Native Storage for Kubernetes • v1.3.8(2020/7 ) • CNCF • Incubating Stage(2020/7 ) • Ceph Cloud Native • v0.8 Ceph Cloud Native Storage – Rook –
23. 23 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • Kubernetes • Kubernetes Operator • Kubernetes • Operator • Prometheus • • Stable - Ceph - EdgeFS( Closed Source) • alpha - Cassandra/CockroachDB/NFS/YugabyteDB Rook
24. 24 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Ceph • https://ceph.io/ • 2003 Sage Weil UCSC • 2006 OSS • 2012 Inktank 2014 Red Hat • v15.2.4(2020/7 ) • C++ • CSI(Container Storage Interface) • cephalopod Ceph (cephalopod) • v15.x Octopus Cloud Native Storage – Ceph –
25. 25 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • RADOS(Reliable Autonomic Distributed Object Store) • • RADOS - S3 SWIFT ( S3 SWIFT ( )) • Ceph Worker Node • CSI(Container Storage Interface) • OpenStack • • • OSD(Object Storage Device) • • Ceph Ceph Contributions
26. 26 Copyright © 2020, Oracle and/or its affiliates. All rights reserved (RADOSGW/LIBRADOS) • LIBRADOS • RADOS • RADOSGW(Ceph Object Gateway) • AmazonS3/Swift (CephFS) • NFS HDFS Shared Storage (RBD) • Ceph :https://docs.ceph.com/docs/master/architecture/
27. 27 Copyright © 2020, Oracle and/or its affiliates. All rights reserved OSD(Object Storage Device) • ( ) • OSD ( ) • Primary OSD Primary OSD Secondary OSD - OSD Placement Group(PG) - PG OSD Primary OSD Secondary OSD - • OSD Placement Group(PG) CRUSH ( ) Ceph – OSD – Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 OSD#1 OSD#2 OSD#3 OSD#4 Ceph Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD
28. 28 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Pool • • Pool - (RADOSGW/CephFS/RBD) - Placement Group(PG) - Placement Group(PG) • OSD Pool • 1 OSD PG CRUSH • PG OSD Ceph – Pool/PG/CRUSH – 1( 2) 2( 3) PG1 OSD#1 OSD#3 Ceph Storage Cluster #1 Ceph Ceph Storage Cluster #2 PG2 OSD#4 OSD#2 OSD#5 OSD#1 OSD#2 OSD#3 OSD#4 CRUSH CephFS Write Write Ceph Storage Cluster # 3 OSD#5 Primary OSD RBD
29. 29 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • Ceph Ceph ( ) • • • (split brain ) 3 • • (Paxos) / Ceph – – Ceph Storage Cluster #1 Ceph Storage Cluster #2 Paxos Ceph OSD#1 OSD#2 OSD#3 OSD#4 Ceph
30. 30 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • v12.x • (Prometheus/Zabbix ) • Ceph Dashboard Ceph – – Ceph Storage Cluster #1 (Prometheus/Zabbix) Ceph OSD#1 OSD#2
31. 31 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Ceph – – 1( 2) PG1 PG2 2( 3) Ceph OSD#1 OSD#2 Ceph Storage Cluster #1 Ceph Storage Cluster #2 CRUSH Ceph Ceph Storage Cluster #3 CRUSH PG/OSD Primary OSD OSD#3 OSD#4 OSD#5 Paxos Paxos Primary OSD Secondary OSD
32. 32 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • OSD Ceph • OSD • OSD Ceph OSD • OSD OSD • OSD Ceph ( / )
33. 33 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Rook • Operator/(Agent) • Rook - Agent FlexVolume Ceph • MON Ceph • OSD’s Ceph OSD • MGR Ceph • MDS/RGW:RADOS • MDS CephFS RGW RADOSGW Rook-Ceph − – ” Many of the Ceph concepts like placement groups and crush maps are hidden so you don’t have to worry about them. Instead Rook creates a much simplified UX for admins that is in terms of physical resources, pools, volumes, filesystems, and buckets. ” • Ceph (PG CRUSH map ) • UX https://rook.io/docs/rook/v1.3/ceph-storage.html
34. 34 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Rook-Ceph CSI • csi-cephfsplugin • CephFS CSI • csi-rbdplugin • RBD CSI • csi-cephfsplugin-provisioner/csi-rbdplugin-provisioner • kube-apiserver CephFS RBD • Rook v1.1 CSI Driver • FlexVolume Driver deprecate Rook-Ceph – CSI(Container Storage Interface) – Rook-Ceph CSI RBD provisioner <rook operator namespace>.rbd.csi.ceph.com
35. 35 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Rook • Rook (rook-operator) rook-discover Ceph • OSD( ) Rook-Ceph CSI • CephFS RBD CSI provisioner Rook-Ceph − − kubelet Rook-Ceph CSI rook-operator rook-discover rook-discover rook-discover ceph-osd ceph-osd ceph-osd ceph- mon ceph- mon ceph- mon ceph-mgr provisioner csi-rbdplugin csi- cephfsplugin kubelet kubelet provisioner csi-rbdplugin csi- cephfsplugin provisioner csi-rbdplugin csi- cephfsplugin deployment daemonset
36. 36 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • Rook Operator • Rook-Ceph Ceph • api-server Rook Operator • CSI Application Attach/Detach Rook-Ceph – Kubernetes – kubelet Worker Node Ceph Rook- Ceph CSI api- server Application Rook Operator Master Node etcd Attach/Detach
37. 37 Copyright © 2020, Oracle and/or its affiliates. All rights reserved OSD on PVC • PV/PVC OSD Pod • PVC(Persistent Volume Claim) OSD • Rook 1.1 OSD on device • OSD • manifest • Kubernetes Rook-Ceph – Rook-Ceph OSD – PVC
38. 38 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Rook-Ceph • Kubernetes( ) Ceph • Ceph SDS(Software Defined Storage) • Ceph Rook Rook • Ceph stable Operator • Cassandra CockroachDB YugabyteDB Rook Rook/Ceph
39. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle Corporation Japan July 30th, 2020 Takuya Niita
40. 40 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Rook-Ceph • https://github.com/rook/rook/tree/master/cl uster/examples/kubernetes/ceph • 3 manifest • common.yaml • operator.yaml • cluster.yaml 1 – Rook-Ceph ( ) – Block Volume 100GB Ceph Oracle Container Engine for Kubernetes Block Volume 100GB Block Volume 100GB /
41. 41 Copyright © 2020, Oracle and/or its affiliates. All rights reserved common.yaml • Namespace • ”rook-ceph” • Custom Resource Definition(CRD) • cephclusters.ceph.rook.io • cephrbdmirrors.ceph.rook.io • cephfilesystems.ceph.rook.io • 14 CRD • RBAC • ClusterRoleBinding/ClusterRole/Role/Role Binding/ ServiceAccount 1 – Rook-Ceph (common.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/common.yaml ※ 2000 Namespace CRD
42. 42 Copyright © 2020, Oracle and/or its affiliates. All rights reserved operator.yaml • ConfigMap • Rook Operator • Deployment • Rook Operator - Ceph Rook Operator - rook-discover (FlexVolume )rook-agent 1 – Rook-Ceph (operator.yaml) – https://github.com/rook/rook/blob/master/clu ster/examples/kubernetes/ceph/operator.yaml ConfigMap Deployment
43. 43 Copyright © 2020, Oracle and/or its affiliates. All rights reserved cluster.yaml • CephCluster • OSD on device Ceph • Ceph - 1 – Rook-Ceph (cluster.yaml) – https://raw.githubusercontent.com/rook/rook /master/cluster/examples/kubernetes/ceph/ cluster.yaml CephCluster ( )
44. 44 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Ceph • • - • Grafana - Prometheus • • • • OSD 1 – Rook-Ceph (Ceph ) – Ceph Ceph
45. 45 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • Rook-Ceph • manifest • 10 • Ceph • 1 – Rook-Ceph (Wrap up) –
46. 46 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Oracle Container Engine for Kubernetes(OKE) • 1 IOPS • ( ) Standard VM + Block Volume • Standard VM with Rook-Ceph(Block Volume Ceph ) Worker Node(Compute Instance) • Standard VM • VM.Standard.2.8/OCPU :8 OCPU(16 vCPU)/ :120GB/ :8.2Gbps Rook-Ceph • Rook:v1.3.7( v1.3.8) • Ceph:v15.2.4( ) 2 – Rook-Ceph ( ) –
47. 47 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • Kubestone(benchmarking Operator that can evaluate the performance of Kubernetes installations) • https://kubestone.io/en/latest/ • fio Job • - Read/ Write ◦ iodepth=4/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth=64/ioengine=libaio/--bs=4k/--size=1G/--direct=1/--time_based/--runtime=180 ◦ iodepth 1 • Volume Size(Block Volume) • Job Pod /data PV 300Gi • Rook-Ceph OSD • 3 Woker Node 1OSD • ( ) 3 → 1 2 – Rook-Ceph ( ) –
48. 48 Copyright © 2020, Oracle and/or its affiliates. All rights reserved 2 – Rook-Ceph ( 1) – Block Volume Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 300Gi fio / Kubestone Job 3
49. 49 Copyright © 2020, Oracle and/or its affiliates. All rights reserved 2 – Rook-Ceph ( 2) – Block Volume Block Volume 100Gi Ceph (Ceph RBD) Oracle Container Engine for Kubernetes Persistent Volume 300Gi Job Pod 100Gi Block Volume 100Gi fio Kubestone Job 3 300Gi
50. 50 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • Read Rook-Ceph IOPS • - OSD - ( ) • Write Rook-Ceph IOPS • - (Rook-Ceph ) 2 – Rook-Ceph ( ) – iodepth=4 iodepth=64 1.49 1.00 0 0.5 1 1.5 2 Read Rook-Ceph Non Rook-Ceph 0.16 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph 1.21 1.00 0 0.5 1 1.5 Read Rook-Ceph Non Rook-Ceph 0.23 1.00 0 0.5 1 1.5 2 Write Rook-Ceph Non Rook-Ceph
51. 51 Copyright © 2020, Oracle and/or its affiliates. All rights reserved • Rook-Ceph Read • Write Ceph 2 – Rook-Ceph (Wrap up) –
52. Oracle Cloud Hangout Cafe 3 #1 – Cloud Native Oracle Corporation Japan July 30th, 2020 Takuya Niita
53. 53 Cloud Native Storage • Kubernetes • SC/PV/PVC • CSI Rook-Ceph • ( ) Rook • Ceph • Kubernetes Rook-Ceph Rook-Ceph • Ceph Kubernetes Ceph • Rook Operator Copyright © 2020, Oracle and/or its affiliates. All rights reserved
54. 54 Copyright © 2020, Oracle and/or its affiliates. All rights reserved Kuberentes (Storage/Volume) https://kubernetes.io/docs/concepts/storage/ Rook Advent Calendar 2019 https://qiita.com/advent-calendar/2019/rook Rook https://rook.io/docs/rook/v1.3/ Rook(GitHub) https://github.com/rook/rook Ceph https://docs.ceph.com/docs/master/ Ceph(GitHub) https://github.com/ceph & Special Thanks!!
55. 55 Kubernetes Meetup https://k8sjp.connpass.com/ Rook Meetup https://rook.connpass.com/ Copyright © 2020, Oracle and/or its affiliates. All rights reserved